Chapter 6. Manual index changes

This strategy consists in removing the existing index and then adding all entities back to the index using FullTextSession.purgeAll() and FullTextSession.index(), however there are some memory and efficiency constraints. For maximum efficiency Hibernate Search batches index operations and executes them at commit time. If you expect to index a lot of data you need to be careful about memory consumption since all documents are kept in a queue until the transaction commit. You can potentially face an OutOfMemoryException if you don’t empty the queue periodically: to do this you can use fullTextSession.flushToIndexes(). Every time fullTextSession.flushToIndexes() is called (or if the transaction is committed), the batch queue is processed applying all index changes. Be aware that, once flushed, the changes cannot be rolled back.

fullTextSession.setFlushMode(FlushMode.MANUAL);
fullTextSession.setCacheMode(CacheMode.IGNORE);
transaction = fullTextSession.beginTransaction();
//Scrollable results will avoid loading too many objects in memory
ScrollableResults results = fullTextSession.createCriteria( Email.class )
    .setFetchSize(BATCH_SIZE)
    .scroll(ScrollMode.FORWARD_ONLY);
int index = 0;
while(results.next()) {
    index++;
    fullTextSession.index(results.get(0)); //index each element
    if (index % BATCH_SIZE == 0) {
        fullTextSession.flushToIndexes(); //apply changes to indexes
        fullTextSession.clear(); //free memory since the queue is processed
    }
}
transaction.commit();

Try to use a batch size that guarantees that your application will not run out of memory: with a bigger batch size objects are fetched faster from database but more memory is needed.

Hibernate Search’s MassIndexer uses several parallel threads to rebuild the index; you can optionally select which entities need to be reloaded or have it reindex all entities. This approach is optimized for best performance but requires to set the application in maintenance mode: making queries to the index is not recommended when a MassIndexer is busy.

fullTextSession.createIndexer().startAndWait();

This will rebuild the index, deleting it and then reloading all entities from the database. Although it’s simple to use, some tweaking is recommended to speed up the process: there are several parameters configurable.

fullTextSession
 .createIndexer( User.class )
 .batchSizeToLoadObjects( 25 )
 .cacheMode( CacheMode.NORMAL )
 .threadsToLoadObjects( 12 )
 .idFetchSize( 150 )
 .transactionTimeout( 1800 )
 .progressMonitor( monitor ) //a MassIndexerProgressMonitor implementation
 .startAndWait();

This will rebuild the index of all User instances (and subtypes), and will create 12 parallel threads to load the User instances using batches of 25 objects per query; these same 12 threads will also need to process indexed embedded relations and custom FieldBridges or ClassBridges, to finally output a Lucene document. In this conversion process these threads are likely going to need to trigger lazy loading of additional attributes, so you will probably need a high number of threads working in parallel. When run in a JTA environment such as the WildFly application server, the mass indexer will use a timeout of 1800 seconds (= 30 minutes) for its transactions. Configure a timeout value which is long enough to load and index all entities of the type with the most instances, taking into account the configured batch size and number of threads to load objects. Note that these transactions are read-only, so choosing a substantially large value should pose no problem in general.

As of Hibernate Search 4.4.0, instead of indexing all the types in parallel, the MassIndexer is configured by default to index only one type in parallel. It prevents resource exhaustion especially database connections and usually does not slow down the indexing. You can however configure this behavior using MassIndexer.typesToIndexInParallel(int threadsToIndexObjects):

fullTextSession
 .createIndexer( User.class, Customer.class )
 .typesToIndexInParallel( 2 )
 .batchSizeToLoadObjects( 25 )
 .cacheMode( CacheMode.NORMAL )
 .threadsToLoadObjects( 5 )
 .idFetchSize( 150 )
 .progressMonitor( monitor ) //a MassIndexerProgressMonitor implementation
 .startAndWait();

Generally we suggest to leave cacheMode to CacheMode.IGNORE (the default), as in most reindexing situations the cache will be a useless additional overhead; it might be useful to enable some other CacheMode depending on your data: it could increase performance if the main entity is relating to enum-like data included in the index.

threads = typesToIndexInParallel * (threadsToLoadObjects + 1);
required JDBC connections = threads;

package org.myproject
import org.hibernate.search.spi.MassIndexerFactory

// ...

public class CustomIndexerFactory implements MassIndexerFactory {

  public void initialize(Properties properties) {
  }

  public MassIndexer createMassIndexer(...) {
    return new CustomIndexer();
  }

}

hibernate.search.massindexer.factoryclass = org.myproject.CustomIndexerFactory

Other parameters which affect indexing time and memory consumption are:

Previous versions also had a max_field_length but this was removed from Lucene, it’s possible to obtain a similar effect by using a LimitTokenCountAnalyzer.

All .indexwriter parameters are Lucene specific and Hibernate Search is just passing these parameters through - see Section 3.8.1, “Tuning indexing performance” for more details.

The MassIndexer uses a forward only scrollable result to iterate on the primary keys to be loaded, but MySQL’s JDBC driver will load all values in memory; to avoid this "optimization" set idFetchSize to Integer.MIN_VALUE.