Hibernate.orgCommunity Documentation

Chapter 3. Configuration

3.1. Directory configuration
3.2. Sharding indexes
3.3. Sharing indexes (two entities into the same directory)
3.4. Worker configuration
3.5. JMS Master/Slave configuration
3.5.1. Slave nodes
3.5.2. Master node
3.6. JGroups Master/Slave configuration
3.6.1. Slave nodes
3.6.2. Master node
3.6.3. JGroups channel configuration
3.7. Reader strategy configuration
3.8. Enabling Hibernate Search and automatic indexing
3.8.1. Enabling Hibernate Search
3.8.2. Automatic indexing
3.9. Tuning Lucene indexing performance
3.10. LockFactory configuration
3.11. Exception Handling Configuration

Apache Lucene has a notion of Directory to store the index files. The Directory implementation can be customized, but Lucene comes bundled with a file system (FSDirectoryProvider) and an in memory (RAMDirectoryProvider) implementation. DirectoryProviders are the Hibernate Search abstraction around a Lucene Directory and handle the configuration and the initialization of the underlying Lucene resources. Table 3.1, “List of built-in Directory Providers” shows the list of the directory providers bundled with Hibernate Search.

Table 3.1. List of built-in Directory Providers

ClassDescriptionProperties
org.hibernate.search.store.RAMDirectoryProviderMemory based directory, the directory will be uniquely identified (in the same deployment unit) by the @Indexed.index elementnone
org.hibernate.search.store.FSDirectoryProviderFile system based directory. The directory used will be <indexBase>/< indexName >

indexBase : Base directory

indexName: override @Indexed.index (useful for sharded indexes)

locking_strategy : optional, see Section 3.10, “LockFactory configuration”

org.hibernate.search.store.FSMasterDirectoryProvider

File system based directory. Like FSDirectoryProvider. It also copies the index to a source directory (aka copy directory) on a regular basis.

The recommended value for the refresh period is (at least) 50% higher that the time to copy the information (default 3600 seconds - 60 minutes).

Note that the copy is based on an incremental copy mechanism reducing the average copy time.

DirectoryProvider typically used on the master node in a JMS back end cluster.

The buffer_size_on_copy optimum depends on your operating system and available RAM; most people reported good results using values between 16 and 64MB.

indexBase: Base directory

indexName: override @Indexed.index (useful for sharded indexes)

sourceBase: Source (copy) base directory.

source: Source directory suffix (default to @Indexed.index). The actual source directory name being <sourceBase>/<source>

refresh: refresh period in second (the copy will take place every refresh seconds).

buffer_size_on_copy: The amount of MegaBytes to move in a single low level copy instruction; defaults to 16MB.

locking_strategy : optional, see Section 3.10, “LockFactory configuration”

org.hibernate.search.store.FSSlaveDirectoryProvider

File system based directory. Like FSDirectoryProvider, but retrieves a master version (source) on a regular basis. To avoid locking and inconsistent search results, 2 local copies are kept.

The recommended value for the refresh period is (at least) 50% higher that the time to copy the information (default 3600 seconds - 60 minutes).

Note that the copy is based on an incremental copy mechanism reducing the average copy time.

DirectoryProvider typically used on slave nodes using a JMS back end.

The buffer_size_on_copy optimum depends on your operating system and available RAM; most people reported good results using values between 16 and 64MB.

indexBase: Base directory

indexName: override @Indexed.index (useful for sharded indexes)

sourceBase: Source (copy) base directory.

source: Source directory suffix (default to @Indexed.index). The actual source directory name being <sourceBase>/<source>

refresh: refresh period in second (the copy will take place every refresh seconds).

buffer_size_on_copy: The amount of MegaBytes to move in a single low level copy instruction; defaults to 16MB.

locking_strategy : optional, see Section 3.10, “LockFactory configuration”


If the built-in directory providers do not fit your needs, you can write your own directory provider by implementing the org.hibernate.store.DirectoryProvider interface.

Each indexed entity is associated to a Lucene index (an index can be shared by several entities but this is not usually the case). You can configure the index through properties prefixed by hibernate.search.indexname . Default properties inherited to all indexes can be defined using the prefix hibernate.search.default.

To define the directory provider of a given index, you use the hibernate.search.indexname.directory_provider


applied on


will create a file system directory in /usr/lucene/indexes/Status where the Status entities will be indexed, and use an in memory directory named Rules where Rule entities will be indexed.

You can easily define common rules like the directory provider and base directory, and override those defaults later on on a per index basis.

Writing your own DirectoryProvider, you can utilize this configuration mechanism as well.

In some cases, it is necessary to split (shard) the indexing data of a given entity type into several Lucene indexes. This solution is not recommended unless there is a pressing need because by default, searches will be slower as all shards have to be opened for a single search. In other words don't do it until you have problems :)

For example, sharding may be desirable if:

Hibernate Search allows you to index a given entity type into several sub indexes. Data is sharded into the different sub indexes thanks to an IndexShardingStrategy. By default, no sharding strategy is enabled, unless the number of shards is configured. To configure the number of shards use the following property


This will use 5 different shards.

The default sharding strategy, when shards are set up, splits the data according to the hash value of the id string representation (generated by the Field Bridge). This ensures a fairly balanced sharding. You can replace the strategy by implementing IndexShardingStrategy and by setting the following property


Using a custom IndexShardingStrategy implementation, it's possible to define what shard a given entity is indexed to.

It also allows for optimizing searches by selecting which shard to run the query onto. By activating a filter (see Section 5.3.1, “Using filters in a sharded environment”), a sharding strategy can select a subset of the shards used to answer a query (IndexShardingStrategy.getDirectoryProvidersForQuery) and thus speed up the query execution.

Each shard has an independent directory provider configuration as described in Section 3.1, “Directory configuration”. The DirectoryProvider default name for the previous example are <indexName>.0 to <indexName>.4. In other words, each shard has the name of it's owning index followed by . (dot) and its index number.


This configuration uses the default id string hashing strategy and shards the Animal index into 5 subindexes. All subindexes are FSDirectoryProvider instances and the directory where each subindex is stored is as followed:

  • for subindex 0: /usr/lucene/indexes/Animal00 (shared indexBase but overridden indexName)

  • for subindex 1: /usr/lucene/indexes/Animal.1 (shared indexBase, default indexName)

  • for subindex 2: /usr/lucene/indexes/Animal.2 (shared indexBase, default indexName)

  • for subindex 3: /usr/lucene/shared/Animal03 (overridden indexBase, overridden indexName)

  • for subindex 4: /usr/lucene/indexes/Animal.4 (shared indexBase, default indexName)

It is technically possible to store the information of more than one entity into a single Lucene index. There are two ways to accomplish this:

It is possible to refine how Hibernate Search interacts with Lucene through the worker configuration. The work can be executed to the Lucene directory or sent to a JMS queue for later processing. When processed to the Lucene directory, the work can be processed synchronously or asynchronously to the transaction commit.

You can define the worker configuration using the following properties

Table 3.2. worker configuration

PropertyDescription
hibernate.search.worker.backendOut of the box support for the Apache Lucene back end and the JMS back end. Default to lucene. Supports also jms, blackhole, jgroupsMaster and jgroupsSlave.
hibernate.search.worker.executionSupports synchronous and asynchronous execution. Default to sync. Supports also async.
hibernate.search.worker.thread_pool.sizeDefines the number of threads in the pool. useful only for asynchronous execution. Default to 1.
hibernate.search.worker.buffer_queue.maxDefines the maximal number of work queue if the thread poll is starved. Useful only for asynchronous execution. Default to infinite. If the limit is reached, the work is done by the main thread.
hibernate.search.worker.jndi.*Defines the JNDI properties to initiate the InitialContext (if needed). JNDI is only used by the JMS back end.
hibernate.search.worker.jms.connection_factoryMandatory for the JMS back end. Defines the JNDI name to lookup the JMS connection factory from (/ConnectionFactory by default in JBoss AS)
hibernate.search.worker.jms.queueMandatory for the JMS back end. Defines the JNDI name to lookup the JMS queue from. The queue will be used to post work messages.
hibernate.search.worker.jgroups.clusterNameOptional for JGroups back end. Defines the name of JGroups channel.
hibernate.search.worker.jgroups.configurationFileOptional JGroups network stack configuration. Defines the name of a JGroups configuration file, which must exist on classpath.
hibernate.search.worker.jgroups.configurationXmlOptional JGroups network stack configuration. Defines a String representing JGroups configuration as XML.
hibernate.search.worker.jgroups.configurationStringOptional JGroups network stack configuration. Provides JGroups configuration in plain text.

This section describes in greater detail how to configure the Master / Slaves Hibernate Search architecture.

Every index update operation is taken from a JMS queue and executed. The master index is copied on a regular basis.


The refresh period should be higher that the expected time copy.

In addition to the Hibernate Search framework configuration, a Message Driven Bean should be written and set up to process the index works queue through JMS.


This example inherits from the abstract JMS controller class available in the Hibernate Search source code and implements a JavaEE 5 MDB. This implementation is given as an example and, while most likely be more complex, can be adjusted to make use of non Java EE Message Driven Beans. For more information about the getSession() and cleanSessionIfNeeded(), please check AbstractJMSHibernateSearchController's javadoc.

Describes how to configure JGroups Master/Slave back end. Configuration examples illustrated in JMS Master/Slave configuration section (Section 3.5, “JMS Master/Slave configuration”) also apply here, only a different backend needs to be set.

Optionally configuration for JGroups transport protocols (UDP, TCP) and channel name can be defined. It can be applied to both master and slave nodes. There are several ways to configure JGroups transport details. If it is not defined explicity, configuration found in the flush-udp.xml file is used.


Master and slave nodes communicate over JGroups channel that is identified by this same name. Name of the channel can be defined explicity, if not default HSearchCluster is used.


The different reader strategies are described in Reader strategy. Out of the box strategies are:

  • shared: share index readers across several queries. This strategy is the most efficient.

  • not-shared: create an index reader for each individual query

The default reader strategy is shared. This can be adjusted:

hibernate.search.reader.strategy = not-shared

Adding this property switches to the not-shared strategy.

Or if you have a custom reader strategy:

hibernate.search.reader.strategy = my.corp.myapp.CustomReaderProvider

where my.corp.myapp.CustomReaderProvider is the custom strategy implementation.

Hibernate Search is enabled out of the box when using Hibernate Annotations or Hibernate EntityManager. If, for some reason you need to disable it, set hibernate.search.autoregister_listeners to false. Note that there is no performance penalty when the listeners are enabled but no entities are annotated as indexed.

To enable Hibernate Search in Hibernate Core (ie. if you don't use Hibernate Annotations), add the FullTextIndexEventListener for the following six Hibernate events and also add it after the default DefaultFlushEventListener, as in the following example.


By default, every time an object is inserted, updated or deleted through Hibernate, Hibernate Search updates the according Lucene index. It is sometimes desirable to disable that features if either your index is read-only or if index updates are done in a batch way (see Section 6.3, “Rebuilding the whole Index”).

To disable event based indexing, set

hibernate.search.indexing_strategy manual

Note

In most case, the JMS backend provides the best of both world, a lightweight event based system keeps track of all changes in the system, and the heavyweight indexing process is done by a separate process or machine.

Hibernate Search allows you to tune the Lucene indexing performance by specifying a set of parameters which are passed through to underlying Lucene IndexWriter such as mergeFactor, maxMergeDocs and maxBufferedDocs. You can specify these parameters either as default values applying for all indexes, on a per index basis, or even per shard.

There are two sets of parameters allowing for different performance settings depending on the use case. During indexing operations triggered by database modifications, the parameters are grouped by the transaction keyword:

hibernate.search.[default|<indexname>].indexwriter.transaction.<parameter_name>

When indexing occurs via FullTextSession.index() or via a MassIndexer (see Section 6.3, “Rebuilding the whole Index”), the used properties are those grouped under the batch keyword:

hibernate.search.[default|<indexname>].indexwriter.batch.<parameter_name>

If no value is set for a .batch value in a specific shard configuration, Hibernate Search will look at the index section, then at the default section:

hibernate.search.Animals.2.indexwriter.transaction.max_merge_docs 10
hibernate.search.Animals.2.indexwriter.transaction.merge_factor 20
hibernate.search.default.indexwriter.batch.max_merge_docs 100

This configuration will result in these settings applied to the second shard of Animals index:

  • transaction.max_merge_docs = 10

  • batch.max_merge_docs = 100

  • transaction.merge_factor = 20

  • batch.merge_factor = Lucene default

All other values will use the defaults defined in Lucene.

The default for all values is to leave them at Lucene's own default, so the listed values in the following table actually depend on the version of Lucene you are using; values shown are relative to version 2.4. For more information about Lucene indexing performances, please refer to the Lucene documentation.

Warning

Previous versions had the batch parameters inherit from transaction properties. This needs now to be explicitly set.

Table 3.3. List of indexing performance and behavior properties

PropertyDescriptionDefault Value
hibernate.search.[default|<indexname>].exclusive_index_use

Set to true when no other process will need to write to the same index: this will enable Hibernate Search to work in exlusive mode on the index and improve performance in writing changes to the index.

false (releases locks as soon as possible)
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].max_buffered_delete_terms

Determines the minimal number of delete terms required before the buffered in-memory delete terms are applied and flushed. If there are documents buffered in memory at the time, they are merged and a new segment is created.

Disabled (flushes by RAM usage)
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].max_buffered_docs

Controls the amount of documents buffered in memory during indexing. The bigger the more RAM is consumed.

Disabled (flushes by RAM usage)
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].max_field_length

The maximum number of terms that will be indexed for a single field. This limits the amount of memory required for indexing so that very large data will not crash the indexing process by running out of memory. This setting refers to the number of running terms, not to the number of different terms.

This silently truncates large documents, excluding from the index all terms that occur further in the document. If you know your source documents are large, be sure to set this value high enough to accommodate the expected size. If you set it to Integer.MAX_VALUE, then the only limit is your memory, but you should anticipate an OutOfMemoryError.

If setting this value in batch differently than in transaction you may get different data (and results) in your index depending on the indexing mode.

10000
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].max_merge_docs

Defines the largest number of documents allowed in a segment. Larger values are best for batched indexing and speedier searches. Small values are best for transaction indexing.

Unlimited (Integer.MAX_VALUE)
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].merge_factor

Controls segment merge frequency and size.

Determines how often segment indexes are merged when insertion occurs. With smaller values, less RAM is used while indexing, and searches on unoptimized indexes are faster, but indexing speed is slower. With larger values, more RAM is used during indexing, and while searches on unoptimized indexes are slower, indexing is faster. Thus larger values (> 10) are best for batch index creation, and smaller values (< 10) for indexes that are interactively maintained. The value must no be lower than 2.

10
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].ram_buffer_size

Controls the amount of RAM in MB dedicated to document buffers. When used together max_buffered_docs a flush occurs for whichever event happens first.

Generally for faster indexing performance it's best to flush by RAM usage instead of document count and use as large a RAM buffer as you can.

16 MB
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].term_index_interval

Expert: Set the interval between indexed terms.

Large values cause less memory to be used by IndexReader, but slow random-access to terms. Small values cause more memory to be used by an IndexReader, and speed random-access to terms. See Lucene documentation for more details.

128
hibernate.search.[default|<indexname>].indexwriter.[transaction|batch].use_compound_fileThe advantage of using the compound file format is that less file descriptors are used. The disadvantage is that indexing takes more time and temporary disk space. You can set this parameter to false in an attempt to improve the indexing time, but you could run out of file descriptors if mergeFactor is also large.

Boolean parameter, use "true" or "false". The default value for this option is true.

true

Tip

When your architecture permits it, always set hibernate.search.default.exclusive_index_use=true as it greatly improves efficiency in index writing.

To tune the indexing speed it might be useful to time the object loading from database in isolation from the writes to the index. To achieve this set the blackhole as worker backend and start you indexing routines. This backend does not disable Hibernate Search: it will still generate the needed changesets to the index, but will discard them instead of flushing them to the index. As opposite to setting the hibernate.search.indexing_strategy to manual when using blackhole it will possibly load more data to rebuild the index from associated entities.

hibernate.search.worker.backend blackhole

The recommended approach is to focus first on optimizing the object loading, and then use the timings you achieve as a baseline to tune the indexing process.

The blackhole backend is not meant to be used in production, only as a tool to identify indexing bottlenecks.

Lucene Directories have default locking strategies which work well for most cases, but it's possible to specify for each index managed by Hibernate Search which LockingFactory you want to use.

Some of these locking strategies require a filesystem level lock and may be used even on RAM based indexes, but this is not recommended and of no practical use.

To select a locking factory, set the hibernate.search.<index>.locking_strategy option to one of simple, native, single or none, or set it to the fully qualified name of an implementation of org.hibernate.search.store.LockFactoryFactory; Implementing this interface you can provide a custom org.apache.lucene.store.LockFactory.


Configuration example:
hibernate.search.default.locking_strategy simple
hibernate.search.Animals.locking_strategy native
hibernate.search.Books.locking_strategy org.custom.components.MyLockingFactory

Hibernate Search allows you to configure how exceptions are handled during the indexing process. If no configuration is provided then exceptions are logged to the log output by default. It is possible to explicitly declare the exception logging mechanism as seen below:

hibernate.search.error_handler log

The default exception handling occurs for both synchronous and asynchronous indexing. Hibernate Search provides an easy mechanism to override the default error handling implementation.

In order to provide your own implementation you must implement the ErrorHandler interface, which provides handle ( ErrorContext context ) method. The ErrorContext provides a reference to the primary LuceneWork that failed, the underlying exception and any subsequent LuceneWork that could not be processed due to the primary exception.

public interface ErrorContext  {
   List<LuceneWork> getFailingOperations();
   LuceneWork getOperationAtFault();
   Throwable getThrowable();
   boolean hasErrors();
}

The following provides an example implementation of ErrorHandler:

public class CustomErrorHandler implements ErrorHandler {
   public void handle ( ErrorContext context ) {
      ...
      //publish error context to some internal error handling system
      ...
   }
}

To register this error handler with Hibernate Search you must declare the CustomErrorHandler fully qualified classname in the configuration properties:

hibernate.search.error_handler CustomerErrorHandler