Hibernate.orgCommunity Documentation

Chapter 4. Mapping entities to the index structure

4.1. Mapping an entity
4.1.1. Basic mapping
4.1.2. Mapping properties multiple times
4.1.3. Embedded and associated objects
4.1.4. Boost factor
4.1.5. Dynamic boost factor
4.1.6. Analyzer
4.2. Property/Field Bridge
4.2.1. Built-in bridges
4.2.2. Custom Bridge
4.3. Providing your own id
4.3.1. The ProvidedId annotation
4.4. Programmatic API
4.4.1. Mapping an entity as indexable
4.4.2. Adding DocumentId to indexed entity
4.4.3. Defining analyzers
4.4.4. Defining full text filter definitions
4.4.5. Defining fields for indexing
4.4.6. Programmatically defining embedded entities
4.4.7. Contained In definition
4.4.8. Date/Calendar Bridge
4.4.9. Defining bridges
4.4.10. Mapping class bridge
4.4.11. Mapping dynamic boost

All the metadata information needed to index entities is described through annotations. There is no need for xml mapping files. In fact there is currently no xml configuration option available (see HSEARCH-210). You can still use Hibernate mapping files for the basic Hibernate configuration, but the Hibernate Search specific configuration has to be expressed via annotations.

First, we must declare a persistent class as indexable. This is done by annotating the class with @Indexed (all entities not annotated with @Indexed will be ignored by the indexing process):


The index attribute tells Hibernate what the Lucene directory name is (usually a directory on your file system). It is recommended to define a base directory for all Lucene indexes using the hibernate.search.default.indexBase property in your configuration file. Alternatively you can specify a base directory per indexed entity by specifying hibernate.search.<index>.indexBase, where <index> is the fully qualified classname of the indexed entity. Each entity instance will be represented by a Lucene Document inside the given index (aka Directory).

For each property (or attribute) of your entity, you have the ability to describe how it will be indexed. The default (no annotation present) means that the property is ignored by the indexing process. @Field does declare a property as indexed. When indexing an element to a Lucene document you can specify how it is indexed:

  • name : describe under which name, the property should be stored in the Lucene Document. The default value is the property name (following the JavaBeans convention)

  • store : describe whether or not the property is stored in the Lucene index. You can store the value Store.YES (consuming more space in the index but allowing projection, see Section 5.1.2.5, “Projection” for more information), store it in a compressed way Store.COMPRESS (this does consume more CPU), or avoid any storage Store.NO (this is the default value). When a property is stored, you can retrieve its original value from the Lucene Document. This is not related to whether the element is indexed or not.

  • index: describe how the element is indexed and the type of information store. The different values are Index.NO (no indexing, ie cannot be found by a query), Index.TOKENIZED (use an analyzer to process the property), Index.UN_TOKENIZED (no analyzer pre-processing), Index.NO_NORMS (do not store the normalization data). The default value is TOKENIZED.

  • termVector: describes collections of term-frequency pairs. This attribute enables term vectors being stored during indexing so they are available within documents. The default value is TermVector.NO.

    The different values of this attribute are:

    ValueDefinition
    TermVector.YESStore the term vectors of each document. This produces two synchronized arrays, one contains document terms and the other contains the term's frequency.
    TermVector.NODo not store term vectors.
    TermVector.WITH_OFFSETSStore the term vector and token offset information. This is the same as TermVector.YES plus it contains the starting and ending offset position information for the terms.
    TermVector.WITH_POSITIONSStore the term vector and token position information. This is the same as TermVector.YES plus it contains the ordinal positions of each occurrence of a term in a document.
    TermVector.WITH_POSITION_OFFSETSStore the term vector, token position and offset information. This is a combination of the YES, WITH_OFFSETS and WITH_POSITIONS.

Whether or not you want to store the original data in the index depends on how you wish to use the index query result. For a regular Hibernate Search usage storing is not necessary. However you might want to store some fields to subsequently project them (see Section 5.1.2.5, “Projection” for more information).

Whether or not you want to tokenize a property depends on whether you wish to search the element as is, or by the words it contains. It make sense to tokenize a text field, but probably not a date field.

Note

Fields used for sorting must not be tokenized.

Finally, the id property of an entity is a special property used by Hibernate Search to ensure index unicity of a given entity. By design, an id has to be stored and must not be tokenized. To mark a property as index id, use the @DocumentId annotation. If you are using Hibernate Annotations and you have specified @Id you can omit @DocumentId. The chosen entity id will also be used as document id.


Example 4.2, “Adding @DocumentId ad @Field annotations to an indexed entity” define an index with three fields: id , Abstract and text . Note that by default the field name is decapitalized, following the JavaBean specification

Associated objects as well as embedded objects can be indexed as part of the root entity index. This is useful if you expect to search a given entity based on properties of associated objects. In the following example the aim is to return places where the associated city is Atlanta (In the Lucene query parser language, it would translate into address.city:Atlanta).


In this example, the place fields will be indexed in the Place index. The Place index documents will also contain the fields address.id, address.street, and address.city which you will be able to query. This is enabled by the @IndexedEmbedded annotation.

Be careful. Because the data is denormalized in the Lucene index when using the @IndexedEmbedded technique, Hibernate Search needs to be aware of any change in the Place object and any change in the Address object to keep the index up to date. To make sure the Place Lucene document is updated when it's Address changes, you need to mark the other side of the bidirectional relationship with @ContainedIn.

@ContainedIn is only useful on associations pointing to entities as opposed to embedded (collection of) objects.

Let's make our example a bit more complex:


Any @*ToMany, @*ToOne and @Embedded attribute can be annotated with @IndexedEmbedded. The attributes of the associated class will then be added to the main entity index. In the previous example, the index will contain the following fields

  • id

  • name

  • address.street

  • address.city

  • address.ownedBy_name

The default prefix is propertyName., following the traditional object navigation convention. You can override it using the prefix attribute as it is shown on the ownedBy property.

Note

The prefix cannot be set to the empty string.

The depth property is necessary when the object graph contains a cyclic dependency of classes (not instances). For example, if Owner points to Place. Hibernate Search will stop including Indexed embedded attributes after reaching the expected depth (or the object graph boundaries are reached). A class having a self reference is an example of cyclic dependency. In our example, because depth is set to 1, any @IndexedEmbedded attribute in Owner (if any) will be ignored.

Using @IndexedEmbedded for object associations allows you to express queries such as:

  • Return places where name contains JBoss and where address city is Atlanta. In Lucene query this would be

    +name:jboss +address.city:atlanta  
  • Return places where name contains JBoss and where owner's name contain Joe. In Lucene query this would be

    +name:jboss +address.orderBy_name:joe  

In a way it mimics the relational join operation in a more efficient way (at the cost of data duplication). Remember that, out of the box, Lucene indexes have no notion of association, the join operation is simply non-existent. It might help to keep the relational model normalized while benefiting from the full text index speed and feature richness.

Note

An associated object can itself (but does not have to) be @Indexed

When @IndexedEmbedded points to an entity, the association has to be directional and the other side has to be annotated @ContainedIn (as seen in the previous example). If not, Hibernate Search has no way to update the root index when the associated entity is updated (in our example, a Place index document has to be updated when the associated Address instance is updated).

Sometimes, the object type annotated by @IndexedEmbedded is not the object type targeted by Hibernate and Hibernate Search. This is especially the case when interfaces are used in lieu of their implementation. For this reason you can override the object type targeted by Hibernate Search using the targetElement parameter.


The @Boost annotation used in Section 4.1.4, “Boost factor” defines a static boost factor which is is independent of the state of of the indexed entity at runtime. However, there are usecases in which the boost factor may depends on the actual state of the entity. In this case you can use the @DynamicBoost annotation together with an accompanying custom BoostStrategy.


In
Example 4.8, “Dynamic boost examle” a dynamic boost is defined on class level specifying VIPBoostStrategy as implementation of the BoostStrategy interface to be used at indexing time. You can place the @DynamicBoost either at class or field level. Depending on the placement of the annotation either the whole entity is passed to the defineBoost method or just the annotated field/property value. It's up to you to cast the passed object to the correct type. In the example all indexed values of a VIP person would be double as important as the values of a normal person.

Note

The specified BoostStrategy implementation must define a public no-arg constructor.

Of course you can mix and match @Boost and @DynamicBoost annotations in your entity. All defined boost factors are cummulative as described in Section 4.1.4, “Boost factor”.

The default analyzer class used to index tokenized fields is configurable through the hibernate.search.analyzer property. The default value for this property is org.apache.lucene.analysis.standard.StandardAnalyzer.

You can also define the analyzer class per entity, property and even per @Field (useful when multiple fields are indexed from a single property).


In this example, EntityAnalyzer is used to index all tokenized properties (eg. name), except summary and body which are indexed with PropertyAnalyzer and FieldAnalyzer respectively.

Caution

Mixing different analyzers in the same entity is most of the time a bad practice. It makes query building more complex and results less predictable (for the novice), especially if you are using a QueryParser (which uses the same analyzer for the whole query). As a rule of thumb, for any given field the same analyzer should be used for indexing and querying.

Analyzers can become quite complex to deal with for which reason Hibernate Search introduces the notion of analyzer definitions. An analyzer definition can be reused by many @Analyzer declarations. An analyzer definition is composed of:

This separation of tasks - a list of char filters, and a tokenizer followed by a list of filters - allows for easy reuse of each individual component and let you build your customized analyzer in a very flexible way (just like Lego). Generally speaking the char filters do some pre-processing in the character input, then the Tokenizer starts the tokenizing process by turning the character input into tokens which are then further processed by the TokenFilters. Hibernate Search supports this infrastructure by utilizing the Solr analyzer framework. Make sure to add solr-core.jar and solr-solrj.jar to your classpath to use analyzer definitions. In case you also want to use the snowball stemmer also include the lucene-snowball.jar. Other Solr analyzers might depend on more libraries. For example, the PhoneticFilterFactory depends on commons-codec. Your distribution of Hibernate Search provides these dependencies in its lib directory.


A char filter is defined by its factory which is responsible for building the char filter and using the optional list of parameters. In our example, a mapping char filter is used, and will replace characters in the input based on the rules specified in the mapping file. A tokenizer is also defined by its factory. This example use the standard tokenizer. A filter is defined by its factory which is responsible for creating the filter instance using the optional parameters. In our example, the StopFilter filter is built reading the dedicated words property file and is expected to ignore case. The list of parameters is dependent on the tokenizer or filter factory.

Warning

Filters and char filters are applied in the order they are defined in the @AnalyzerDef annotation. Make sure to think twice about this order.

Once defined, an analyzer definition can be reused by an @Analyzer declaration using the definition name rather than declaring an implementation class.


Analyzer instances declared by @AnalyzerDef are available by their name in the SearchFactory.

Analyzer analyzer = fullTextSession.getSearchFactory().getAnalyzer("customanalyzer");

This is quite useful wen building queries. Fields in queries should be analyzed with the same analyzer used to index the field so that they speak a common "language": the same tokens are reused between the query and the indexing process. This rule has some exceptions but is true most of the time. Respect it unless you know what you are doing.

Solr and Lucene come with a lot of useful default char filters, tokenizers and filters. You can find a complete list of char filter factories, tokenizer factories and filter factories at http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters. Let check a few of them.




We recommend to check all the implementations of org.apache.solr.analysis.TokenizerFactory and org.apache.solr.analysis.TokenFilterFactory in your IDE to see the implementations available.

So far all the introduced ways to specify an analyzer were static. However, there are use cases where it is useful to select an analyzer depending on the current state of the entity to be indexed, for example in multilingual applications. For an BlogEntry class for example the analyzer could depend on the language property of the entry. Depending on this property the correct language specific stemmer should be chosen to index the actual text.

To enable this dynamic analyzer selection Hibernate Search introduces the AnalyzerDiscriminator annotation. The following example demonstrates the usage of this annotation:


The prerequisite for using @AnalyzerDiscriminator is that all analyzers which are going to be used are predefined via @AnalyzerDef definitions. If this is the case one can place the @AnalyzerDiscriminator annotation either on the class or on a specific property of the entity for which to dynamically select an analyzer. Via the impl parameter of the AnalyzerDiscriminator you specify a concrete implementation of the Discriminator interface. It is up to you to provide an implementation for this interface. The only method you have to implement is getAnalyzerDefinitionName() which gets called for each field added to the Lucene document. The entity which is getting indexed is also passed to the interface method. The value parameter is only set if the AnalyzerDiscriminator is placed on property level instead of class level. In this case the value represents the current value of this property.

An implemention of the Discriminator interface has to return the name of an existing analyzer definition if the analyzer should be set dynamically or null if the default analyzer should not be overridden. The given example assumes that the language parameter is either 'de' or 'en' which matches the specified names in the @AnalyzerDefs.

During indexing time, Hibernate Search is using analyzers under the hood for you. In some situations, retrieving analyzers can be handy. If your domain model makes use of multiple analyzers (maybe to benefit from stemming, use phonetic approximation and so on), you need to make sure to use the same analyzers when you build your query.

You can retrieve the scoped analyzer for a given entity used at indexing time by Hibernate Search. A scoped analyzer is an analyzer which applies the right analyzers depending on the field indexed: multiple analyzers can be defined on a given entity each one working on an individual field, a scoped analyzer unify all these analyzers into a context-aware analyzer. While the theory seems a bit complex, using the right analyzer in a query is very easy.


In the example above, the song title is indexed in two fields: the standard analyzer is used in the field title and a stemming analyzer is used in the field title_stemmed. By using the analyzer provided by the search factory, the query uses the appropriate analyzer depending on the field targeted.

If your query targets more that one query and you wish to use your standard analyzer, make sure to describe it using an analyzer definition. You can retrieve analyzers by their definition name using searchFactory.getAnalyzer(String).

In Lucene all index fields have to be represented as Strings. For this reason all entity properties annotated with @Field have to be indexed in a String form. For most of your properties, Hibernate Search does the translation job for you thanks to a built-in set of bridges. In some cases, though you need a more fine grain control over the translation process.

Hibernate Search comes bundled with a set of built-in bridges between a Java property type and its full text representation.

null

null elements are not indexed. Lucene does not support null elements and this does not make much sense either.

java.lang.String

String are indexed as is

short, Short, integer, Integer, long, Long, float, Float, double, Double, BigInteger, BigDecimal

Numbers are converted in their String representation. Note that numbers cannot be compared by Lucene (ie used in ranged queries) out of the box: they have to be padded

java.util.Date

Dates are stored as yyyyMMddHHmmssSSS in GMT time (200611072203012 for Nov 7th of 2006 4:03PM and 12ms EST). You shouldn't really bother with the internal format. What is important is that when using a DateRange Query, you should know that the dates have to be expressed in GMT time.

Usually, storing the date up to the millisecond is not necessary. @DateBridge defines the appropriate resolution you are willing to store in the index ( @DateBridge(resolution=Resolution.DAY) ). The date pattern will then be truncated accordingly.

@Entity 
@Indexed
public class Meeting {
    @Field(index=Index.UN_TOKENIZED)
    @DateBridge(resolution=Resolution.MINUTE)
    private Date date;
    ...                 
java.net.URI, java.net.URL

URI and URL are converted to their string representation

java.lang.Class

Class are converted to their fully qualified class name. The thread context classloader is used when the class is rehydrated

Sometimes, the built-in bridges of Hibernate Search do not cover some of your property types, or the String representation used by the bridge does not meet your requirements. The following paragraphs describe several solutions to this problem.

The simplest custom solution is to give Hibernate Search an implementation of your expected Object to String bridge. To do so you need to implements the org.hibernate.search.bridge.StringBridge interface. All implementations have to be thread-safe as they are used concurrently.


Then any property or field can use this bridge thanks to the @FieldBridge annotation

@FieldBridge(impl = PaddedIntegerBridge.class)
private Integer length;                

Parameters can be passed to the Bridge implementation making it more flexible. The Bridge implementation implements a ParameterizedBridge interface, and the parameters are passed through the @FieldBridge annotation.


The ParameterizedBridge interface can be implemented by StringBridge, TwoWayStringBridge, FieldBridge implementations.

All implementations have to be thread-safe, but the parameters are set during initialization and no special care is required at this stage.

If you expect to use your bridge implementation on an id property (ie annotated with @DocumentId ), you need to use a slightly extended version of StringBridge named TwoWayStringBridge. Hibernate Search needs to read the string representation of the identifier and generate the object out of it. There is no difference in the way the @FieldBridge annotation is used.


It is critically important for the two-way process to be idempotent (ie object = stringToObject( objectToString( object ) ) ).

Some use cases require more than a simple object to string translation when mapping a property to a Lucene index. To give you the greatest possible flexibility you can also implement a bridge as a FieldBridge. This interface gives you a property value and let you map it the way you want in your Lucene Document. The interface is very similar in its concept to the Hibernate UserTypes.

You can for example store a given property in two different document fields:


In the previous example the fields where not added directly to Document but we where delegating this task to the LuceneOptions helper; this will apply the options you have selected on @Field, like Store or TermVector options, or apply the choosen @Boost value. It is especially useful to encapsulate the complexity of COMPRESS implementations so it's recommended to delegate to LuceneOptions to add fields to the Document, but nothing stops you from editing the Document directly and ignore the LuceneOptions in case you need to.

Tip

Classes like LuceneOptions are created to shield your application from changes in Lucene API and simplify your code. Use them if you can, but if you need more flexibility you're not required to.

It is sometimes useful to combine more than one property of a given entity and index this combination in a specific way into the Lucene index. The @ClassBridge and @ClassBridge annotations can be defined at the class level (as opposed to the property level). In this case the custom field bridge implementation receives the entity instance as the value parameter instead of a particular property. Though not shown in this example, @ClassBridge supports the termVector attribute discussed in section Section 4.1.1, “Basic mapping”.


In this example, the particular CatFieldsClassBridge is applied to the department instance, the field bridge then concatenate both branch and network and index the concatenation.

You can provide your own id for Hibernate Search if you are extending the internals. You will have to generate a unique value so it can be given to Lucene to be indexed. This will have to be given to Hibernate Search when you create an org.hibernate.search.Work object - the document id is required in the constructor.

Although the recommended approach for mapping indexed entities is to use annotations, it is sometimes more convenient to use a different approach:

While it has been a popular demand in the past, the Hibernate team never found the idea of an XML alternative to annotations appealing due to it's heavy duplication, lack of code refactoring safety, because it did not cover all the use case spectrum and because we are in the 21st century :)

Th idea of a programmatic API was much more appealing and has now become a reality. You can programmatically and safely define your mapping using a programmatic API: you define entities and fields as indexable by using mapping classes which effectively mirror the annotation concepts in Hibernate Search. Note that fan(s) of XML approach can design their own schema and use the programmatic API to create the mapping while parsing the XML stream.

In order to use the programmatic model you must first construct a SearchMapping object. This object is passed to Hibernate Search via a property set to the Configuration object. The property key is hibernate.search.model_mapping or it's type-safe representation Environment.MODEL_MAPPING.

SearchMapping mapping = new SearchMapping();
[...]
configuration.setProperty( Environment.MODEL_MAPPING, mapping );

//or in JPA
SearchMapping mapping = new SearchMapping();
[...]
Map<String,String> properties = new HashMap<String,String)(1);
properties.put( Environment.MODEL_MAPPING, mapping );
EntityManagerFactory emf = Persistence.createEntityManagerFactory( "userPU", properties );

The SearchMapping is the root object which contains all the necessary indexable entities and fields. From there, the SearchMapping object exposes a fluent (and thus intuitive) API to express your mappings: it contextually exposes the relevant mapping options in a type-safe way, just let your IDE autocompletion feature guide you through.

Today, the programmatic API cannot be used on a class annotated with Hibernate Search annotations, chose one approach or the other. Also note that the same default values apply in annotations and the programmatic API. For example, the @Field.name is defaulted to the property name and does not have to be set.

Each core concept of the programmatic API has a corresponding example to depict how the same definition would look using annotation. Therefore seeing an annotation example of the programmatic approach should give you a clear picture of what Hibernate Search will build with the marked entities and associated properties.

Analyzers can be programmatically defined using the analyzerDef(String analyzerDef, Class<? extends TokenizerFactory> tokenizerFactory) method. This method also enables you to define filters for the analyzer definition. Each filter that you define can optionally take in parameters as seen in the following example :



The programmatic API provides easy mechanism for defining full text filter definitions which is available via @FullTextFilterDef and @FullTextFilterDefs. Note that contrary to the annotation equivalent, full text filter definitions are a global construct and are not tied to an entity. The next example depicts the creation of full text filter definition using the fullTextFilterDef method.



When defining fields for indexing using the programmatic API, call field() on the property(String propertyName, ElementType elementType) method. From field() you can specify the name, index, store, bridge and analyzer definitions.



In this section you will see how to programmatically define entities to be embedded into the indexed entity similar to using the @IndexEmbedded model. In order to define this you must mark the property as indexEmbedded. The is the option to add a prefix to the embedded entity definition and this can be done by calling prefix as seen in the example below: