Chapter 5. Querying

Let’s start with the most basic use case - searching for a specific word:

Query luceneQuery = mythQB.keyword().onField("history").matching("storm").createQuery();

keyword() means that you are trying to find a specific word. onField() specifies in which Lucene field to look. matching() tells what to look for. And finally createQuery() creates the Lucene query object. A lot is going on with this line of code.

We make the example a little more advanced now and have a look at how to search a field that uses ngram analyzers. ngram analyzers index succession of ngrams of your words which helps to recover from user typos. For example the 3-grams of the word hibernate are hib, ibe, ber, rna, nat, ate.

@AnalyzerDef(name = "ngram",
  tokenizer = @TokenizerDef(factory = StandardTokenizerFactory.class ),
  filters = {
    @TokenFilterDef(factory = StandardFilterFactory.class),
    @TokenFilterDef(factory = LowerCaseFilterFactory.class),
    @TokenFilterDef(factory = StopFilterFactory.class),
    @TokenFilterDef(factory = NGramFilterFactory.class,
      params = {
        @Parameter(name = "minGramSize", value = "3"),
        @Parameter(name = "maxGramSize", value = "3") } )
  }
)
@Entity
@Indexed
public class Myth {
  @Field(analyzer=@Analyzer(definition="ngram")
  public String getName() { return name; }
  public String setName(String name) { this.name = name; }
  private String name;

  ...
}

Query luceneQuery = mythQb.keyword().onField("name").matching("Sisiphus")
   .createQuery();

The matching word "Sisiphus" will be lower-cased and then split into 3-grams: sis, isi, sip, phu, hus. Each of these n-gram will be part of the query. We will then be able to find the Sysiphus myth (with a y). All that is transparently done for you.

To search for multiple possible words in the same field, simply add them all in the matching clause.

//search document with storm or lightning in their history
Query luceneQuery =
    mythQB.keyword().onField("history").matching("storm lightning").createQuery();

To search the same word on multiple fields, use the onFields method.

Query luceneQuery = mythQB
    .keyword()
    .onFields("history","description","name")
    .matching("storm")
    .createQuery();

Sometimes, one field should be treated differently from another field even if searching the same term, you can use the andField() method for that.

Query luceneQuery = mythQB.keyword()
    .onField("history")
    .andField("name")
      .boostedTo(5)
    .andField("description")
    .matching("storm")
    .createQuery();

In the previous example, only field name is boosted to 5.

To execute a fuzzy query (based on the Levenshtein distance algorithm), start like a keyword query and add the fuzzy flag.

Query luceneQuery = mythQB
    .keyword()
      .fuzzy()
        .withThreshold(.8f)
        .withPrefixLength(1)
    .onField("history")
    .matching("starm")
    .createQuery();

threshold is the limit above which two terms are considering matching. It’s a decimal between 0 and 1 and defaults to 0.5. prefixLength is the length of the prefix ignored by the "fuzzyness": while it defaults to 0, a non zero value is recommended for indexes containing a huge amount of distinct terms.

You can also execute wildcard queries (queries where some of parts of the word are unknown). The character ? represents a single character and * represents any character sequence. Note that for performance purposes, it is recommended that the query does not start with either ? or *.

Query luceneQuery = mythQB
    .keyword()
      .wildcard()
    .onField("history")
    .matching("sto*")
    .createQuery();

So far we have been looking for words or sets of words, you can also search exact or approximate sentences. Use phrase() to do so.

Query luceneQuery = mythQB
    .phrase()
    .onField("history")
    .sentence("Thou shalt not kill")
    .createQuery();

You can search approximate sentences by adding a slop factor. The slop factor represents the number of other words permitted in the sentence: this works like a within or near operator

Query luceneQuery = mythQB
    .phrase()
      .withSlop(3)
    .onField("history")
    .sentence("Thou kill")
    .createQuery();

After looking at all these query examples for searching for to a given word, it is time to introduce range queries (on numbers, dates, strings etc). A range query searches for a value in between given boundaries (included or not) or for a value below or above a given boundary (included or not).

//look for 0 <= starred < 3
Query luceneQuery = mythQB
    .range()
    .onField("starred")
    .from(0).to(3).excludeLimit()
    .createQuery();

//look for myths strictly BC
Date beforeChrist = ...;
Query luceneQuery = mythQB
    .range()
    .onField("creationDate")
    .below(beforeChrist).excludeLimit()
    .createQuery();

This set of queries has its own chapter, check out Chapter 9, Spatial.

Have you ever looked at an article or document and thought: "I want to find more like this"? Have you ever appreciated an e-commerce website that gives you similar articles to the one you are exploring?

More Like This queries are achieving just that. You feed it an entity (or its identifier) and Hibernate Search returns the list of entities that are similar.

The query DSL API should be self explaining. Let’s look at some usage examples.

QueryBuilder qb = fullTextSession.getSearchFactory()
        .buildQueryBuilder()
        .forEntity( Coffee.class )
        .get();

Query mltQuery = qb
        .moreLikeThis()
            .comparingAllFields()
            .toEntityWithId( coffeeId )
            .createQuery();
List<Object[]> results = (List<Object[]>) fullTextSession
        .createFullTextQuery( mltQuery, Coffee.class )
        .setProjection( ProjectionConstants.THIS, ProjectionConstants.SCORE )
        .list();

This first example takes the id of an Coffee entity and finds the matching coffees across all fields. To be fair, this is not across all fields. To be included in the More Like This query, fields need to store term vectors or the actual field value. Id fields (of the root entity as well as embedded entities) and numeric fields are excluded. The latter exclusion might change in future versions.

Looking at the Coffee class, the following fields are considered: name as it is stored, description as it stores the term vector. id and internalDescription are excluded.

@Entity @Indexed
public class Coffee {

    @Id @GeneratedValue
    public Integer getId() { return id; }

    @Field(termVector = TermVector.NO, store = Store.YES)
    public String getName() { return name; }

    @Field(termVector = TermVector.YES)
    public String getSummary() { return summary; }

    @Column(length = 2000)
    @Field(termVector = TermVector.YES)
    public String getDescription() { return description; }

    public int getIntensity() { return intensity; }

    // Not stored nor term vector, i.e. cannot be used for More Like This
    @Field
    public String getInternalDescription() { return internalDescription; }

    // ...
}

In the example above we used projection to retrieve the relative score of each element. We might use the score to only display the results for which the score is high enough.

Often, you are only interested in a few key fields to find similar entities. Plus some fields are more important than others and should be boosted.

Query mltQuery = qb
        .moreLikeThis()
            .comparingField("summary").boostedTo(10f)
            .andField("description")
            .toEntityWithId( coffeeId )
            .createQuery();

In this example, we look for similar entities by summary and description. But similar summaries are more important than similar descriptions. This is a critical tool to make More Like This meaningful for your data set.

Instead of providing the entity id, you can pass the full entity object. If the entity contains the identifier, we will use it to find the term vectors or field values. This means that we will compare the entity state as stored in the Lucene index. If the identifier cannot be retrieved (for example if the entity has not been persisted yet), we will look at each of the entity properties to find the most meaningful terms. The latter is slower and won’t give the best results - avoid it if possible.

Here is how you pass the entity instance you want to compare with:

Coffee coffee = ...; //managed entity from somewhere

Query mltQuery = qb
        .moreLikeThis()
            .comparingField("summary").boostedTo(10f)
            .andField("description")
            .toEntity( coffee )
            .createQuery();

Query mltQuery = qb
        .moreLikeThis()
            .excludeEntityUsedForComparison()
            .comparingField("summary").boostedTo(10f)
            .andField("description")
            .toEntity( coffee )
            .createQuery();

You can ask Hibernate Search to give a higher score to the very similar entities and downgrade the score of mildly similar entities. We do that by boosting each meaningful terms by their individual overall score. Start with a boost factor of 1 and adjust from there.

Query mltQuery = qb
        .moreLikeThis()
            .favorSignificantTermsWithFactor(1f)
            .comparingField("summary").boostedTo(10f)
            .andField("description")
            .toEntity( coffee )
            .createQuery();

Remember, more like this is a very subjective meaning and will vary depending on your data and the rules of your domain. With the various options offered, Hibernate Search arms you with the tools to adjust this weapon. Make sure to continuously test the results against your data set.

You can combine queries to create more complex queries. The following aggregation operators are available:

These aggregations have a similar effect as the classic boolean operators AND, OR and NOT, but have different names to emphasise that they will have an impact on scoring.

For example the SHOULD operator between two queries will have an effect similar to the boolean OR: if either of the two combined queries matches the entry, the entry will be included in the match; though the entries which match both queries will have an higher score than those which only match one of them.

The sub-queries can be any Lucene query including a boolean query itself.

Query combinedQuery = querybuilder
    .bool()
        .must( queryA )
        .must( queryB )
    .createQuery();

Query combinedQuery = querybuilder
    .bool()
        .should( queryA )
        .should( queryB )
    .createQuery();

Query combinedQuery = querybuilder
    .bool()
        .must( queryA )
        .must( queryB ).not()
    .createQuery();

Let’s look at a few more practical examples; note how the querybuilder usage can be nested and how 'should', 'must', and 'not' can be combined in many ways:

//look for popular modern myths that are not urban
Date twentiethCentury = ...;
Query luceneQuery = mythQB
    .bool()
      .must( mythQB.keyword().onField("description").matching("urban").createQuery() )
        .not()
      .must( mythQB.range().onField("starred").above(4).createQuery() )
      .must( mythQB
        .range()
        .onField("creationDate")
        .above(twentiethCentury)
        .createQuery() )
    .createQuery();

//look for popular myths that are preferably urban
Query luceneQuery = mythQB
    .bool()
      .should( mythQB.keyword().onField("description").matching("urban").createQuery() )
      .must( mythQB.range().onField("starred").above(4).createQuery() )
    .createQuery();

//look for all myths except religious ones
Query luceneQuery = mythQB
    .all()
      .except( monthQb
        .keyword()
        .onField( "description_stem" )
        .matching( "religion" )
        .createQuery()
      )
    .createQuery();

We already have seen several query options in the previous example, but lets summarize again the options for query types and fields:

Let’s check out an example using some of these options

Query luceneQuery = mythQB
    .bool()
      .should( mythQB.keyword().onField("description").matching("urban").createQuery() )
      .should( mythQB
        .keyword()
        .onField("name")
          .boostedTo(3)
          .ignoreAnalyzer()
        .matching("urban").createQuery() )
      .must( mythQB
        .range()
          .boostedTo(5).withConstantScore()
        .onField("starred").above(4).createQuery() )
    .createQuery();

As you can see, the Hibernate Search query DSL is an easy to use and easy to read query API and by accepting and producing Lucene queries, you can easily incorporate query types not (yet) supported by the DSL. Please give us feedback!

Once the Lucene query is built, it needs to be wrapped into an Hibernate Query. If not specified otherwise, the query will be executed against all indexed entities, potentially returning all types of indexed classes.

FullTextSession fullTextSession = Search.getFullTextSession( session );
org.hibernate.Query fullTextQuery = fullTextSession.createFullTextQuery( luceneQuery );

It is advised, from a performance point of view, to restrict the returned types:

fullTextQuery = fullTextSession
    .createFullTextQuery(luceneQuery, Customer.class);

// or

fullTextQuery = fullTextSession
    .createFullTextQuery(luceneQuery, Item.class, Actor.class);

In Example 5.9, “Filtering the search result by entity type” the first example returns only matching Customer instances, the second returns matching Actor and Item instances. The type restriction is fully polymorphic which means that if there are two indexed subclasses Salesman and Customer of the baseclass Person, it is possible to just specify Person.class in order to filter on result types.

Out of performance reasons it is recommended to restrict the number of returned objects per query. In fact is a very common use case anyway that the user navigates from one page to an other. The way to define pagination is exactly the way you would define pagination in a plain HQL or Criteria query.

org.hibernate.Query fullTextQuery =
    fullTextSession.createFullTextQuery(luceneQuery, Customer.class);
fullTextQuery.setFirstResult(15); //start from the 15th element
fullTextQuery.setMaxResults(10); //return 10 elements

Apache Lucene provides a very flexible and powerful way to sort results. While the default sorting (by relevance) is appropriate most of the time, it can be interesting to sort by one or several other properties.

In order to do so, you will have to build a Lucene Sort object, which can be done by using either the Hibernate Search sort DSL or the native Lucene SortField class.

If you use the Hibernate Search sort DSL, it will look like this:

QueryBuilder builder = fullTextSession.getSearchFactory()
    .buildQueryBuilder().forEntity(Book.class).get();
Query luceneQuery = /* ... */;
FullTextQuery query = s.createFullTextQuery( luceneQuery, Book.class );
Sort sort = builder
  .sort()
    .byField("author").desc() // Descending order
    .andByField("title") // Default order (ascending)
  .createSort();
query.setSort(sort);
List results = query.list();

Alternatively, you may build your sort using the Lucene SortField class directly. Then you will have to always specify the sort field type manually.

FullTextQuery query = s.createFullTextQuery( luceneQuery, Book.class );
org.apache.lucene.search.Sort sort = new Sort(
    new SortField("author", SortField.Type.STRING),
    new SortField("title", SortField.Type.STRING));
query.setSort(sort);
List results = query.list();

QueryBuilder builder = fullTextSession.getSearchFactory()
    .buildQueryBuilder().forEntity(Book.class).get();
Query luceneQuery = /* ... */;
FullTextQuery query = s.createFullTextQuery( luceneQuery, Book.class );
Sort sort = builder
  .sort()
    .byField("author")
      .onMissingValue().sortFirst()
    .andByField("title")
  .createSort();
query.setSort(sort);
List results = query.list();

QueryBuilder builder = fullTextSession.getSearchFactory()
    .buildQueryBuilder().forEntity(Book.class).get();
Query luceneQuery = /* ... */;
FullTextQuery query = s.createFullTextQuery( luceneQuery, Book.class );
Sort sort = builder
  .sort()
    .byField("authorId")
      .onMissingValue().use(42L) // Assume the ID is 42 when it's missing
    .andByField("title")
  .createSort();
query.setSort(sort);
List results = query.list();

QueryBuilder builder = fullTextSession.getSearchFactory()
    .buildQueryBuilder().forEntity(Book.class).get();
Query luceneQuery = /* ... */;
FullTextQuery query = s.createFullTextQuery( luceneQuery, Book.class );
SortField complexSortField = /* ... */;
Sort sort = builder
  .sort()
    .byField("author")
    .andByNative(complexSortField)
  .createSort();
query.setSort(sort);
List results = query.list();

When you restrict the return types to one class, Hibernate Search loads the objects using a single query. It also respects the static fetching strategy defined in your domain model.

It is often useful, however, to refine the fetching strategy for a specific use case.

Criteria criteria =
    s.createCriteria(Book.class).setFetchMode("authors", FetchMode.JOIN);
s.createFullTextQuery(luceneQuery).setCriteriaQuery(criteria);

In this example, the query will return all Books matching the luceneQuery. The authors collection will be loaded from the same query using an SQL outer join.

When defining a criteria query, it is not necessary to restrict the returned entity types when creating the Hibernate Search query from the full text session: the type is guessed from the criteria query itself.

For some use cases, returning the domain object (including its associations) is overkill. Only a small subset of the properties is necessary. Hibernate Search allows you to return a subset of properties:

org.hibernate.search.FullTextQuery query =
    s.createFullTextQuery(luceneQuery, Book.class);
query.setProjection("id", "summary", "body", "mainAuthor.name");
List results = query.list();
Object[] firstResult = (Object[]) results.get(0);
Integer id = firstResult[0];
String summary = firstResult[1];
String body = firstResult[2];
String authorName = firstResult[3];

Hibernate Search extracts the properties from the Lucene index and convert them back to their object representation, returning a list of Object[]. Projections avoid a potential database round trip (useful if the query response time is critical). However, it also has several constraints:

Projection is also useful for another kind of use case. Lucene can provide metadata information about the results. By using some special projection constants, the projection mechanism can retrieve this metadata:

org.hibernate.search.FullTextQuery query =
    s.createFullTextQuery(luceneQuery, Book.class);
query.setProjection(
    FullTextQuery.SCORE,
    FullTextQuery.THIS,
    "mainAuthor.name" );
List results = query.list();
Object[] firstResult = (Object[]) results.get(0);
float score = firstResult[0];
Book book = firstResult[1];
String authorName = firstResult[2];

You can mix and match regular fields and projection constants. Here is the list of the available constants:

By default, Hibernate Search uses the most appropriate strategy to initialize entities matching your full text query. It executes one (or several) queries to retrieve the required entities. This is the best approach to minimize database round trips in a scenario where none / few of the retrieved entities are present in the persistence context (ie the session) or the second level cache.

If most of your entities are present in the second level cache, you can force Hibernate Search to look into the cache before retrieving an object from the database.

FullTextQuery query = session.createFullTextQuery(luceneQuery, User.class);
query.initializeObjectWith(
    ObjectLookupMethod.SECOND_LEVEL_CACHE,
    DatabaseRetrievalMethod.QUERY
);

ObjectLookupMethod defines the strategy used to check if an object is easily accessible (without database round trip). Other options are:

You can also customize how objects are loaded from the database (if not found before). Use DatabaseRetrievalMethod for that:

The defaults for both methods, the object lookup as well as the database retrieval can also be configured via configuration properties. This way you don’t have to specify your preferred methods on each query creation. The property names are hibernate.search.query.object_lookup_method and hibernate.search.query.database_retrieval_method respectively. As value use the name of the method (upper- or lowercase). For example:

hibernate.search.query.object_lookup_method = second_level_cache
hibernate.search.query.database_retrieval_method = query

You can limit the time a query takes in Hibernate Search in two ways:

Query luceneQuery = ...;
FullTextQuery query = fullTextSession.createFullTextQuery(luceneQuery, User.class);

//define the timeout in seconds
query.setTimeout(5);

//alternatively, define the timeout in any given time unit
query.setTimeout(450, TimeUnit.MILLISECONDS);

try {
    query.list();
}
catch (org.hibernate.QueryTimeoutException e) {
    //do something, too slow
}

Query luceneQuery = ...;
FullTextQuery query = fullTextEM.createFullTextQuery(luceneQuery, User.class);

//define the timeout in milliseconds
query.setHint( "javax.persistence.query.timeout", 450 );

try {
    query.getResultList();
}
catch (javax.persistence.QueryTimeoutException e) {
    //do something, too slow
}

Query luceneQuery = ...;
FullTextQuery query = fullTextSession.createFullTextQuery(luceneQuery, User.class);

//define the timeout in seconds
query.limitExecutionTimeTo(500, TimeUnit.MILLISECONDS);
List results = query.list();

Query luceneQuery = ...;
FullTextQuery query = fullTextSession.createFullTextQuery(luceneQuery, User.class);

//define the timeout in seconds
query.limitExecutionTimeTo(500, TimeUnit.MILLISECONDS);
List results = query.list();

if ( query.hasPartialResults() ) {
    displayWarningToUser();
}

In the case of a discrete faceting request, you start with giving the request a unique name. This name will later be used to retrieve the facet values (see Section 5.4.4, “Interpreting a Facet result”). Then you need to specify on which index field you want to categorize on and which faceting options to apply. An example for a discrete faceting request can be seen in Example 5.34, “Creating a discrete faceting request”:

QueryBuilder builder = fullTextSession.getSearchFactory()
    .buildQueryBuilder().forEntity(Cd.class).get();

FacetingRequest labelFacetingRequest = builder.facet()
    .name("labelFacetRequest")
    .onField("label")
    .discrete()
    .orderedBy(FacetSortOrder.COUNT_DESC)
    .includeZeroCounts(false)
    .maxFacetCount(3)
    .createFacetingRequest();

When executing this faceting request a Facet instance will be created for each discrete value for the indexed field label. The Facet instance will record the actual field value including how often this particular field value occurs within the original query results. Parameters orderedBy, includeZeroCounts and maxFacetCount are optional and can be applied on any faceting request. Parameter orderedBy allows to specify in which order the created facets will be returned. The default is FacetSortOrder.COUNT_DESC, but you can also sort on the field value. Parameter includeZeroCount determines whether facets with a count of 0 will be included in the result (by default they are not) and maxFacetCount allows to limit the maximum amount of facets returned.

@Fields({
  @Field(name="price"),
  @Field(name="price_facet",
         analyze=Analyze.NO,
         bridge=@FieldBridge(impl = IntegerBridge.class))
})
private int price;

The creation of a range faceting request is similar. We also start with a name for the request and the field to facet on. Then we have to specify ranges for the field values. A range faceting request can be seen in Example 5.35, “Creating a range faceting request”. There, three different price ranges are specified. below and above can only be specified once, but you can specify as many from - to ranges as you want. For each range boundary you can also specify via excludeLimit whether it is included into the range or not.

QueryBuilder builder = fullTextSession.getSearchFactory()
    .buildQueryBuilder()
        .forEntity(Cd.class)
            .get();
FacetingRequest priceFacetingRequest = builder.facet()
    .name("priceFaceting")
    .onField("price_facet")
    .range()
    .below(1000)
    .from(1001).to(1500)
    .above(1500).excludeLimit()
    .createFacetingRequest();