Hibernate.orgCommunity Documentation

Chapter 9. Infinispan

9.1. Configure Infinispan
9.1.1. Adding Infinispan dependencies
9.1.2. Infinispan specific configuration properties
9.1.3. Cache names used by Hibernate OGM
9.2. Manage data size
9.3. Clustering: store data on multiple Infinispan nodes
9.4. Storage principles
9.4.1. Properties and built-in types
9.4.2. Identifiers
9.4.3. Entities
9.4.4. Associations
9.5. Transactions
9.6. Storing a Lucene index in Infinispan

Infinispan is an open source in-memory data grid focusing on high performance. As a data grid, you can deploy it on multiple servers - referred to as nodes - and connect to it as if it were a single storage engine: it will cleverly distribute both the computation effort and the data storage.

It is trivial to setup on a single node, in your local JVM, so you can easily try Hibernate OGM. But Infinispan really shines in multiple node deployments: you will need to configure some networking details but nothing changes in terms of application behaviour, while performance and data size can scale linearly.

From all its features we will only describe those relevant to Hibernate OGM; for a complete description of all its capabilities and configuration options, refer to the Infinispan project documentation at infinispan.org.

You configure Hibernate OGM and Infinispan in two steps basically:

The advanced configuration details of an Infinispan Cache are defined in an Infinispan specific XML configuration file; the Hibernate OGM properties are simple and usually just point to this external resource.

To use the default configuration provided by Hibernate OGM - which is a good starting point for new users - you don’t have to set any property.

Depending on the cache mapping approach, Hibernate OGM will either:

The preferred strategy is CACHE_PER_TABLE as it offers both more fine grained configuration options and the ability to work on specific entities in a more simple fashion.

In the following paragraphs, we will explain which aspects of Infinispan you’re likely to want to reconfigure from their defaults. All attributes and elements from Infinispan which we don’t mention are safe to ignore. Refer to the Infinispan User Guide for the guru level performance tuning and customizations.

An Infinispan configuration file is an XML file complying with the Infinispan schema; the basic structure is shown in the following example:


There are global settings that can be set before the cache_container section. These settings will affect the whole instance; mainly of interest for Hibernate OGM users is the jgroups element in which we will set JGroups configuration overrides.

Inside the cache-container section are defined explicit named caches and their configurations as well as the default cache (named DEFAULT here) if we want to affect all named caches. This is where we will likely want to configure clustering modes, eviction policies and CacheStores.

In its default configuration Infinispan stores all data in the heap of the JVM; in this barebone mode it is conceptually not very different than using a HashMap: the size of the data should fit in the heap of your VM, and stopping/killing/crashing your application will get all data lost with no way to recover it.

To store data permanently (out of the JVM memory) a CacheStore should be enabled. The infinispan-core.jar includes a simple implementation able to store data in simple binary files, on any read/write mounted filesystem; this is an easy starting point, but the real stuff is to be found in the additional modules found in the Infinispan distribution. Here you can find many more implementations to store your data in anything from JDBC connected relational databases, other NoSQL engines, to cloud storage services or other Infinispan clusters. Finally, implementing a custom CacheStore is quite easy.

To limit the memory consumption of the precious heap space, you can activate a passivation or an eviction policy; again there are several strategies to play with, for now let’s just consider you’ll likely need one to avoid running out of memory when storing too many entries in the bounded JVM memory space; of course you don’t need to choose one while experimenting with limited data sizes: enabling such a strategy doesn’t have any other impact in the functionality of your Hibernate OGM application (other than performance: entries stored in the Infinispan in-memory space is accessed much quicker than from any CacheStore).

A CacheStore can be configured as write-through, committing all changes to the CacheStore before returning (and in the same transaction) or as write-behind. A write-behind configuration is normally not encouraged in storage engines, as a failure of the node implies some data might be lost without receiving any notification about it, but this problem is mitigated in Infinispan because of its capability to combine CacheStore write-behind with a synchronous replication to other Infinispan nodes.


In this example we enabled both eviction and a CacheStore (the persistence element). LIRS is one of the choices we have for eviction strategies. Here it is configured to keep (approximately) 2000 entries in live memory and evict the remaining as a memory usage control strategy.

The CacheStore is enabling passivation, which means that the entries which are evicted are stored on the filesystem.

Warning

You could configure an eviction strategy while not configuring a passivating CacheStore! That is a valid configuration for Infinispan but will have the evictor permanently remove entries. Hibernate OGM will break in such a configuration.

The best thing about Infinispan is that all nodes are treated equally and it requires almost no beforehand capacity planning: to add more nodes to the cluster you just have to start new JVMs, on the same or different physical servers, having your same Infinispan configuration and your same application.

Infinispan supports several clustering cache modes; each mode provides the same API and functionality but with different performance, scalability and availability options:

To use the replication or distribution cache modes Infinispan will use JGroups to discover and connect to the other nodes.

In the default configuration, JGroups will attempt to autodetect peer nodes using a multicast socket; this works out of the box in the most network environments but will require some extra configuration in cloud environments (which often block multicast packets) or in case of strict firewalls. See the JGroups reference documentation, specifically look for Discovery Protocols to customize the detection of peer nodes.

Nowadays, the JVM defaults to use IPv6 network stack; this will work fine with JGroups, but only if you configured IPv6 correctly. It is often useful to force the JVM to use IPv4.

It is also useful to let JGroups know which networking interface you want to use; especially if you have multiple interfaces it might not guess correctly.


Note

You don’t need to use IPv4: JGroups is compatible with IPv6 provided you have routing properly configured and valid addresses assigned.

The jgroups.bind_addr needs to match a placeholder name in your JGroups configuration in case you don’t use the default one.

The default configuration uses distribution as cache mode and uses the jgroups-tcp.xml configuration for JGroups, which is contained in the Infinispan jar as the default configuration for Infinispan users. Let’s see how to reconfigure this:

Example 9.4. Reconfiguring cache mode and override JGroups configuration


<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:7.0 http://www.infinispan.org/schemas/infinispan-config-7.0.xsd"
    xmlns="urn:infinispan:config:7.0">

    <jgroups>
        <stack-file name="custom-stack" path="my-jgroups-conf.xml" />
    </jgroups>

    <cache-container name="HibernateOGM" default-cache="DEFAULT">
        <transport stack="custom-stack" />

        <!-- *************************************** -->
        <!--     Default cache used as template      -->
        <!-- *************************************** -->
        <distrubuted-cache name="DEFAULT" mode="SYNC">
            <locking striping="false" acquire-timeout="10000"
                concurrency-level="500" write-skew="false" />
            <transaction mode="NON_DURABLE_XA"
                transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" />
            <state-transfer enabled="true" timeout="480000"
                await-initial-transfer="true" />
        </distributed-cache>

        <!-- Override the cache mode: -->
        <replicated-cache name="User" mode="SYNC">
            <locking striping="false" acquire-timeout="10000"
                concurrency-level="500" write-skew="false" />
            <transaction mode="NON_DURABLE_XA"
                transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" />
            <state-transfer enabled="true" timeout="480000"
                await-initial-transfer="true" />
        </replicated-cache>

        <distributed-cache name="Order" mode="SYNC">
            <locking striping="false" acquire-timeout="10000"
                concurrency-level="500" write-skew="false" />
            <transaction mode="NON_DURABLE_XA"
                transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" />
            <state-transfer enabled="true" timeout="480000"
                await-initial-transfer="true" />
        </distributed-cache>

        <distributed-cache name="associations_User_Order" mode="SYNC">
            <locking striping="false" acquire-timeout="10000"
                concurrency-level="500" write-skew="false" />
            <transaction mode="NON_DURABLE_XA"
                transaction-manager-lookup="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" />
            <state-transfer enabled="true" timeout="480000"
                await-initial-transfer="true" />
        </distributed-cache>

    </cache-container>

</infinispan>

In the example above we specify a custom JGroups configuration file and set the cache mode for the default cache to distribution; this is going to be inherited by the Order and the associations_User_Order caches. But for User we have chosen (for the sake of this example) to use replication.

Now that you have clustering configured, start the service on multiple nodes. Each node will need the same configuration and jars.

Tip

We have just shown how to override the clustering mode and the networking stack for the sake of completeness, but you don’t have to!

Start with the default configuration and see if that fits you. You can fine tune these setting when you are closer to going in production.

To describe things simply, each entity is stored under a single key. The value itself is a map containing the columns / values pair.

Each association from one entity instance to (a set of) another is stored under a single key. The value contains the navigational information to the (set of) entity.

Entity identifiers are used to build the key in which the entity is stored in the cache.

The key is comprised of the following information:

In CACHE_PER_TABLE, the table name is inferred from the cache name. In CACHE_PER_KIND, the table name is necessary to identify the entity in the generic cache.



Since Infinispan has not native sequence nor identity column support, these are simulated using the table strategy, however their default values vary. We highly recommend you explicitly use a TABLE strategy if you want to generate a monotonic identifier.

But if you can, use a pure in-memory and scalable strategy like a UUID generator.


As you can see, in CACHE_PER_TABLE, the key does not contain the id source table name. It is inferred by the cache name hosting that key.



Entities are stored in the cache named after the entity name when using the CACHE_PER_TABLE strategy. In the CACHE_PER_KIND strategy, entities are stored in a single cache named ENTITIES.

The key is comprised of the following information:

In CACHE_PER_TABLE, the table name is inferred from the cache name. In CACHE_PER_KIND, the table name is necessary to identify the entity in the generic cache.

The entry value is an instance of org.infinispan.atomic.FineGrainedMap which contains all the entity properties - or to be specific columns. Each column name and value is stored as a key / value pair in the map. We use this specialized map as Infinispan is able to transport changes in a much more efficient way.


As you can see, the table name is not part of the key for CACHE_PER_TYPE. In the rest of this section we will no longer show the CACHE_PER_KIND strategy.




Here, we see that the collection of elements is stored in a separate cache and entry. The association key is made of:

  • the foreign key column names pointing to the owner of this association
  • the foreign key column values pointing to the owner of this association
  • the association table name in the CACHE_PER_KIND approach where all associations share the same cache

The association entry is a map containing the representation of each entry in the collection. The keys of that map are made of:

  • the names of the columns uniquely identifying that specific collection entry (e.g. for a Set this is all of the columns)
  • the values of the columns uniquely identifying that specific collection entry

The value attack to that collection entry key is a Map containing the key value pairs column name / column value.


Here we used an indexed collection and to identify the entry in the collection, only the owning entity id and the index value is enough.

Associations between entities are mapped like (collection of) embeddables except that the target entity is represented by its identifier(s).













Infinispan supports transactions and integrates with any standard JTA TransactionManager; this is a great advantage for JPA users as it allows to experience a similar behaviour to the one we are used to when we work with RDBMS databases.

If you’re having Hibernate OGM start and manage Infinispan, you can skip this as it will inject the same TransactionManager instance which you already have set up in the Hibernate / JPA configuration.

If you are providing an already started Infinispan CacheManager instance by using the JNDI lookup approach, then you have to make sure the CacheManager is using the same TransactionManager as Hibernate:


Infinispan supports different transaction modes like PESSIMISTIC and OPTIMISTIC, supports XA recovery and provides many more configuration options; see the Infinispan User Guide for more advanced configuration options.

Hibernate Search, which can be used for advanced query capabilities (see Chapter 7, Query your entities), needs some place to store the indexes for its embedded Apache Lucene engine.

A common place to store these indexes is the filesystem which is the default for Hibernate Search; however if your goal is to scale your NoSQL engine on multiple nodes you need to share this index. Network sharing file systems are a possibility but we don’t recommended that. Often the best option is to store the index in whatever NoSQL database you are using (or a different dedicated one).

Tip

You might find this section useful even if you don’t intend to store your data in Infinispan.

The Infinispan project provides an adaptor to plug into Apache Lucene, so that it writes the indexes in Infinispan and searches data in it. Since Infinispan can be used as an application cache to other NoSQL storage engines by using a CacheStore (see Section 9.2, “Manage data size”) you can use this adaptor to store the Lucene indexes in any NoSQL store supported by Infinispan:

  • Cassandra
  • Filesystem (but locked correctly at the Infinispan level)
  • MongoDB
  • HBase
  • JDBC databases
  • JDBM
  • BDBJE
  • A secondary (independent) Infinispan grid
  • Any Cloud storage service supported by JClouds

How to configure it? Here is a simple cheat sheet to get you started with this type of setup:

  • Add org.hibernate:hibernate-search-infinispan:5.1.0.Final to your dependencies
  • set these configuration properties:

    • hibernate.search.default.directory_provider = infinispan
    • hibernate.search.default.exclusive_index_use = false
    • hibernate.search.infinispan.configuration_resourcename = [infinispan configuration filename]

The referenced Infinispan configuration should define a CacheStore to load/store the index in the NoSQL engine of choice. It should also define three cache names:


This configuration is not going to scale well on write operations: to do that you should read about the master/slave and sharding options in Hibernate Search. The complete explanation and configuration options can be found in the Hibernate Search Reference Guide

Some NoSQL support storage of Lucene indexes directly, in which case you might skip the Infinispan Lucene integration by implementing a custom DirectoryProvider for Hibernate Search. You’re very welcome to share the code and have it merged in Hibernate Search for others to use, inspect, improve and maintain.