TreeCache: a Tree Structured Replicated Transactional Cache

This and its companion documents describe JBossCache's TreeCache, a tree-structured replicated transactional cache, and the TreeCacheAop, an "object-oriented" cache that is AOP-enabled subclass of TreeCache, allowing for Plain Old Java Objects (POJOs) to be inserted and replicated transactionally between nodes in a cluster. The TreeCache is configurable, i.e. aspects of the system such as replication (async, sync, or none), transaction isolation levels, eviction policy, and transactional manager are all configurable.

TreeCache can also be used independently, but TreeCacheAop requires both TreeCache and the JBossAOP standalone subsystems.

The structure of the cache is A tree with nodes. Each node has a name and zero or more children. A node can only have 1 parent; there is currently no support for graphs. A node can be reached by navigating from the root recursively through children, until the node is found. It can also be accessed by giving a fully qualified name (FQN), which consists of the concatenation of all node names from the root to this particular node.

A TreeCache can have multiple roots, allowing for a number of different trees to be present in the cache. Note that a one level tree is essentially a hashtable. Each node in the tree has a hashmap of keys and values. For a replicated cache, all keys and values have to be serializable. Serializability is not a requirement for TreeCacheAop, where reflection and AOP is used to replicate any type.

A TreeCache can be either local or replicated. Local trees exist only inside the VM in which they are created, whereas replicated trees propagate any changes to all other replicated trees in the same cluster. A cluster may span different hosts on a network or even just different JVMs on a single host.

The first version of JBossCache was a hashmap. However, the decision was taken to go with a tree structured cache because (a) it is more flexible and efficient and (b) a tree can always be reduced to a hashmap, thereby offering both possibilities. The efficiency argument was driven by concerns over replication overhead, and was that a value itself can be a rather sophisticated object, with aggregation pointing to other objects, or an object containing many fields. A small change in the object would therefore trigger the entire object (possibly the transitive closure over the object graph) to be serialized and propagated to the other nodes in the cluster. With a tree, only the modified nodes in the tree need to be serialized. This is not necessarily a concern for TreeCache, but is a vital requirement for TreeCacheAop (as we will see in the TreeCacheAop documentation).

When a change is made to the TreeCache, and that change is done in the context of a transaction, then we wait with replication until the TX commits successfully. All modifications are kept in a list associated with the transaction for the caller. When the TX commits, we replicate the changes. Otherwise, (on a rollback) we simply undo the changes and release the locks, resulting in no replication traffic. For example, if a caller makes 100 modifications and then rolls back the TX, we will not replicate anything, resulting in no network traffic.

If a caller has no TX associated with it (and isolation level is not NONE), we will replicate right after each modification, e.g. in the above case we would send 100 messages, plus an additional message for the rollback. In this sense, no transaction can be think of autocommit is on in JDBC terminology where each operation is committed automatically.

There is an API for plugging in different transaction managers: all it requires is to get the TX associated with the caller’s thread. There is a dummy TransactionManager implementation, and one for JBoss.

Finally, we use pessimistic locking of the cache by default (optimistic locking is on the todo list). As mentioned previously, we can configure the local locking policy corresponding to the JDBC style transaction isolation level, i.e., SERIALIZABLE, REPEATABLE, READ_COMMITTED, READ_UNCOMMITTED and NONE. More on the transaction isolation level will be discussed later. Note that the cluster-wide isolation level is by default read-uncommitted because we don’t acquire a cluster-wide lock on touching an object for which we don’t yet have a lock (this would result in too much overhead for messaging)^[1] .

Figure 1. Schematic TreeCache architecture

The architecture is shown above. The example shows 2 Java VMs, each has created an instance of TreeCache. These VMs can be located on the same machine, or on 2 different machines. The setup of the underlying group communication subsystem is done using JGroups (http://www.jgroups.org).

Any modification (see API below) in one cache will be replicated to the other cache^[2] and vice versa. Depending on the TX settings, this will be done either after each modification or at the end of a TX (at commit time). When a new cache is created, it can optionally acquire the contents from one of the existing caches.

Here's some sample code before we dive into the API itself:

TreeCache tree = new TreeCache();
tree.setClusterName("demo-cluster");
tree.setClusterProperties("default.xml");
tree.setCacheMode(TreeCache.REPL_SYNC);
tree.createService(); // not necessary, but is same as MBean lifecycle
tree.startService(); // kick start tree cache
tree.put("/a/b/c", "name", "Ben");
tree.put("/a/b/c/d", "uid", new Integer(322649));
Integer tmp=(Integer)tree.get("/a/b/c/d", "uid");
tree.remove("/a/b");
tree.stopService();
tree.destroyService(); // not necessary, but is same as MBean lifecycle

The sample code first creates a TreeCache instance and then configures it. There is another constructor which accepts a number of configuration options. However, the TreeCache can be configured entirely from an XML file (shown later), so we don't recommend manual configuration as shown in the sample.

The cluster name, properties of the underlying JGroups stack, and cache mode (synchronous replication) are configured first (a list of configuration options is shown later). Then we start the TreeCache. If replication is enabled, this will make the TreeCache join the cluster, and (optionally) acquire initial state from an existing node.

Then we add 2 items into the cache: the first element creates a node "a" with a child node "b" that has a child node "c". (TreeCache by default creates intermediary nodes that don't exist). The key "name" is then inserted into the "/a/b/c" node, with a value of "Ben".

The other element will create just the subnode "d" of "c" because "/a/b/c" already exists. It binds the integer 322649 under key "uid".

The resulting tree looks like this:

Figure 2. Sample Tree Nodes

The TreeCache has 4 nodes "a", "b", "c" and "d". Nodes "/a/b/c" has values "name" associated with "Ben" in its hashmap, and node "/a/b/c/d" has values "uid" and 322649.

Each node can be retrieved by its absolute name (e.g. "/a/b/c") or by navigating from parent to children (e.g. navigate from "a" to "b", then from "b" to "c").

The next method in the example gets the value associated with key="uid" in node "/a/b/c/d", which is the integer 322649.

The remove() method then removes node "/a/b" and all subnodes recursively from the cache. In this case, nodes "/a/b/c/d", "/a/b/c" and "/a/b" will be removed, leaving only "/a".

Finally, the TreeCache is stopped. This will cause it to leave the cluster, and every node in the cluster will be notified. Note that TreeCache can be stopped and started again. When it is stopped, all contents will be deleted. And when it is restarted, if it joins a cache group, the state will be replicated initially. So potentially you can recreate the contents.

In the sample, replication was enabled, which caused the 2 put() and the 1 remove() methods to replicated their changes to all nodes in the cluster. The get() method was executed on the local cache only.

Keys into the cache can be either strings separated by slashes ('/'), e.g. "/a/b/c", or they can be fully qualified names Fqns . An Fqn is essentially a list of Objects that need to implement hashCode() and equals(). All strings are actually transformed into Fqns internally. Fqns are more efficient than strings, for example:

String n1 = "/300/322649";
Fqn n2 = new Fqn(new Object{new Integer(300), new Integer(322649)});

In this example, we want to access a node that has information for employee with id=322649 in department with id=300. The string version needs 2 hashmap lookups on Strings, whereas the Fqn version needs to hashmap lookups on Integer. In a large hashtable, the hashCode() method for String may have collisions, leading to actual string comparisons. Also, clients of the cache may already have identifiers for their objects in Object form, and don't want to transform between Object and Strings, preventing unnecessary copying.

Note that the modification methods are put() and remove(). The only get method is get().

There are 2 put() methods^[3] : put(Fqn node, Object key, Object key) and put(Fqn node, Hashmap values). The former takes the node name, creates it if it doesn't yet exist, and put the key and value into the node's hashmap, returning the previous value. The latter takes a hashmap of keys and values and adds them to the node's hashmap, overwriting existing keys and values. Content that is not in the new hashmap remains in the node's hashmap.

There are 3 remove() methods: remove(Fqn node, Object key), remove(Fqn node) and removeData(Fqn node). The first removes the given key from the node. The second removes the entire node and all subnodes, and the third removes all elements from the given node's hashmap.

The get methods are: get(Fqn node) and get(Fqn node, Object key). The former returns a Node^[4] object, allowing for direct navigation, the latter returns the value for the given key for a node.

Also, the TreeCache has a number of getters and setters. Since the API may change at any time, we recommend the Javadoc for up-to-date information.

The TreeCache can be configured to be either local or replicated .

Local caches don't join a cluster and don't replicate changes to other nodes in a cluster. Therefore their elements don't need to be serializable^[5] . On the other hand, replicated caches replicate all changes to the other TreeCaches (node) in the cluster. Replication can either happen after each modification (no transactions), or at the end of a transaction (commit time).

Replication can be synchronous or asynchronous . Use of either one of the options is application dependent. Synchronous replication blocks the caller (e.g. on a put()) until the modifications have been replicated successfully to all nodes in a cluster. Asynchronous replication performs replication in the background (the put() returns immediately). TreeCache also offers a replication queue, where modifications are replicated periodically (i.e. interval-based), or when the queue size exceeds a number of elements, or a combination thereof).

Asynchronous replication is faster (no caller blocking), because synchronous replication requires acknowledgments from all nodes in a cluster that they received and applied the modification successfully (round-trip time). However, when a synchronous replication returns successfully, the caller knows for sure that all modifications have been applied at all nodes, whereas this may or may not be the case with asynchronous replication. With asynchronous replication, errors are simply written to a log.

TreeCache currently uses pessimistic locking to prevent concurrent access to the same data. Locking is not exposed directly to user. Instead, a transaction isolation level which provides different locking behavior is configurable.

Locking internally is done on a node-level, so for example when we want to access "/a/b/c", a lock will be acquired for nodes "a", "b" and "c". When the same transaction wants to access "/a/b/c/d", since we already hold locks for "a", "b" and "c", we only need to acquire a lock for "d".

Lock owners are either transactions (call comes in on a transaction) or threads (no transaction associated with incoming transaction). Regardless, a local transaction or a thread is internally transformed into an instance of GlobalTransaction, which is used as a globally unique ID for modifications across a cluster. E.g. when we run a two-phase commit protocol (see below) across the cluster, the GlobalTransaction uniquely identifies the unit of work across a cluster.

TreeCache supports transaction isolation level in the manner of the JDBC model. A user can configure per instance-wide isolation level (as those defined in java.sql.Connection), i.e., NONE, READ_UNCOMMITTED, READ_COMMITTED, REPEATABLE_READ, and SERIALIZABLE. The mapping is summarized here.

Locks can be read or write locks. Write locks serialize read and write access, whereas read-only locks only serialize read access. When a write lock is held, no other write or read locks can be acquired. When a read lock is held, others can acquire read locks. However, to acquire write locks, one has to wait until all read locks have been released. When scheduled concurrently, write locks always have precedence over read locks. Note that (if enabled) read locks can be upgraded to write locks.

Using read-write locks helps in the following scenario: consider a tree with entries "/a/b/n1" and "/a/b/n2". With write-locks, when Tx1 accesses "/a/b/n1", Tx2 cannot access "/a/b/n2" until Tx1 has completed and released its locks. However, with read-write locks this is possible, because Tx1 acquires read-locks for "/a/b" and a read-write lock for "/a/b/n1". Tx2 is then able to acquire read-locks for "/a/b" as well, plus a read-write lock for "/a/b/n2". This allows for more concurrency in accessing the cache.

A TreeCache can be configured to use transactions to bundle units of work, which can then be replicated as one unit. Alternatively, if transaction support is disabled, it is equivalent to setting AutoCommit to on where modifications are potentially^[6] replicated after every change (if replication is enabled).

What TreeCache needs to do on every incoming call (e.g. put()) is (a) get the transaction associated with the thread and (b) register (if not already done) with the transaction manager to be notified when a transaction commits or is rolled back. In order to do this, the cache has to be configured with an instance of TransactionManagerLookup which returns a javax.transaction.TransactionManager.

There are currently 2 implementations of TransactionManagerLookup available: DummyTransactionManagerLookup and JBossTransactionManagerLookup. The former is a dummy implementation of a TransactionManager which can be used for standalone TreeCache applications (running outside an appserver). This is just for demo purposes (e.g. the standalone demo runs with this one). The latter is to be used when TreeCache is used inside JBoss, and its implementation looks like this:

public class JBossTransactionManagerLookup implements TransactionManagerLookup {

    public JBossTransactionManagerLookup() {;}

    public TransactionManager getTransactionManager() throws Exception {
       Object tmp=new InitialContext().lookup("java:/TransactionManager");
       return (TransactionManager)tmp;
    }
}

The implementation looks up the JBoss TransactionManager from the JNDI and returns it.

When a call comes in, the TreeCache gets the current transaction and records the modification under the transaction as key. (If there is no transaction, the modification is applied immediately and possibly replicated). So over the lifetime of the transaction all modifications will be recorded and associated with the transaction. Also, the TreeCache registers with the transaction to be notified of transaction committed or aborted when it first encounters the transaction.

When a transaction rolls back, we undo the changes in the cache and release all locks.

When the transaction commits, we initiate a two-phase commit protocol^[7] : in the first phase, a PREPARE containing all modifications for the current transaction is sent to all nodes in the cluster. Each node acquires all necessary locks and applies the changes, and then sends back a success message. If a node in a cluster cannot acquire all locks, or fails otherwise, it sends back a failure message.

The coordinator of the two-phase commit protocol waits for all responses (or a timeout, whichever occurs first). If one of the nodes in the cluster responds with FAIL (or we hit the timeout), then a rollback phase is initiated: a ROLLBACK message is sent to all nodes in the cluster. On reception of the ROLLBACK message, every node undoes the changes for the given transaction, and releases all locks held for the transaction.

If all responses are OK, a COMMIT message is sent to all nodes in the cluster. On reception of a COMMIT message, each node applies the changes for the given transaction and releases all locks associated with the transaction.

When we referred to 'transaction', we actually mean a global representation of a local transaction, which uniquely identifies a transaction across a cluster.

Properties prop = new Properties();
prop.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.cache.transaction.DummyContextFactory");
User Transaction tx=(UserTransaction)new InitialContext(prop).lookup("UserTransaction");
TreeCache tree = new TreeCache();
PropertyConfigurator config = new PropertyConfigurator();
config.configure(tree, "META-INF/replSync-service.xml");
tree.createService(); // not necessary
tree.startService(); // kick start tree cache

try {
   tx.begin();
   tree.put("/classes/cs-101", "description", "the basics");
   tree.put("/classes/cs-101", "teacher", "Ben");
   tx.commit();
}
catch(Throwable ex) {
   try { tx.rollback(); } catch(Throwable t) {}
}

UserTransaction tx = new DummyUserTransaction(DummyTransactionManager.getInstance());

Eviction policy specifies the behavior of a node residing inside the cache, e.g., life time and maximum numbers allowed.

public interface TreeCacheListener {

   /**
    * Called when a node is created
    * @param fqn
    */
   void nodeCreated(Fqn fqn);

   /**
    * Called when a node is removed.
    * @param fqn
    */
   void nodeRemoved(Fqn fqn);

   /**
    * Called when a node is loaded into memory via the CacheLoader. This is not the same
    * as {@link #nodeCreated(Fqn)}.
    */
   void nodeLoaded(Fqn fqn);

   /**
    * Called when a node is evicted (not the same as remove()).
    * @param fqn
    */
   void nodeEvicted(Fqn fqn);

   /**
    * Called when a node is modified, e.g., one (key, value) pair
    * in the internal map storage has been modified.
    * @param fqn
    */
   void nodeModified(Fqn fqn);

   /**
    * Called when a node is visisted, i.e., get().
    * @param fqn
    */
   void nodeVisited(Fqn fqn);

   /**
    * Called when the cache is started.
    * @param cache
    */
   void cacheStarted(TreeCache cache);

   /**
    * Called when the cache is stopped.
    * @param cache
    */
   void cacheStopped(TreeCache cache);

   void viewChange(View new_view);  // might be MergeView after merging
}

 public interface ExtendedTreeCacheListener
{
   /**
    * Called when a node is about to be evicted or has been evicted from the 
    * in-memory cache.
    * Note: Currently TreeCacheListener has {@link TreeCacheListener#nodeEvicted(Fqn)} 
    * which will be merged with method in release 1.3.
    * 
    * @param fqn
    * @param pre
    * @see TreeCacheListener#nodeEvicted(Fqn)
    */
   void nodeEvict(Fqn fqn, boolean pre);
   
   /**
    * Called when a node is about to be removed or has been removed from the 
    * in-memory cache.
    * Note: Currently TreeCacheListener has {@link TreeCacheListener#nodeRemoved(Fqn)} 
    * which will be merged with this method in release 1.3.
    * 
    * @param fqn
    * @param pre
    * @param isLocal
    * @see TreeCacheListener#nodeRemoved(Fqn)
    */
   void nodeRemove(Fqn fqn, boolean pre, boolean isLocal);

   /**
    * Called when a node is about to be modified or has been modified.  
    * Note: Currently TreeCacheListener has {@link TreeCacheListener#nodeModified(Fqn)} 
    * which will be merged with this method in release 1.3.
    * 
    * @param fqn
    * @param pre
    * @param isLocal
    * @see TreeCacheListener#nodeModified(Fqn)
    */ 
   void nodeModify(Fqn fqn, boolean pre, boolean isLocal);
   
   /**
    * Called when a node is to be or has been activated into memory via the 
    * CacheLoader that was evicted earlier.
    * 
    * @param fqn
    * @param pre
    */
   void nodeActivate(Fqn fqn, boolean pre);
    
   /**
    * Called when a node is to be or has been written to the backend store via the 
    * cache loader due to a node eviction by the eviction policy provider
    *  
    * @param fqn
    * @param pre
    */
   void nodePassivate(Fqn fqn, boolean pre);
}
      
      

   /**
    * Called by eviction policy provider. Note that eviction is done only in local mode,
    * that is, it doesn't replicate the node removal. 
    * @param fqn Will remove everything associated with this fqn.
    * @throws Exception
    */
   public void evict(Fqn fqn) throws Exception;

   /**
    * Like <code>get</code> method but without triggering a node visit event. This is used
    * to prevent refresh of the cache data in the eviction policy.
    * @param fqn
    * @param key
    * @return
    * @throws LockingException
    * @throws TimeoutException
    */
   public Object peek(Fqn fqn, Object key) throws LockingException, TimeoutException;

JBossCache can use a CacheLoader to access a backend datastore. If JBossCache is configured with a CacheLoader, then the following properties are provided:

Currently, the CacheLoader looks similar to the TreeCache interface. In the future, it will actually implement the same interface. The goal is to be able to form hierarchical cache topologies, where one cache can delegate to another, which in turn may delegate to yet another cache.

The CacheLoader interface is defined in org.jboss.cache.loader.CacheLoader as follows (edited for brevity):

public interface CacheLoader extends Service {

   /**
    * Sets the configuration. Will be called before {@link #create()} and {@link #start()}
    * @param props A set of properties specific to a given CacheLoader
    */
   void setConfig(Properties props);

   void setCache(TreeCache c);


   /**
    * Returns a list of children names, all names are <em>relative</em>. Returns null if the parent node is not found.
    * The returned set must not be modified, e.g. use Collections.unmodifiableSet(s) to return the result
    * @param fqn The FQN of the parent
    * @return Set<String>. A list of children. Returns null if no children nodes are present, or the parent is
    * not present
    */
   Set getChildrenNames(Fqn fqn) throws Exception;


   /**
    * Returns the value for a given key. Returns null if the node doesn't exist, or the value is not bound
    */
   Object get(Fqn name, Object key) throws Exception;


   /**
    * Returns all keys and values from the persistent store, given a fully qualified name
    * @param name
    * @return Map<Object,Object> of keys and values for the given node. Returns null if the node was not found
    * @throws Exception
    */
   Map get(Fqn name) throws Exception;


   /**
    * Checks whether the CacheLoader has a node with Fqn
    * @return True if node exists, false otherwise
    */
   boolean exists(Fqn name) throws Exception;


   /**
    * Inserts key and value into the attributes hashmap of the given node. If the node does not exist, all
    * parent nodes from the root down are created automatically
    */
   void put(Fqn name, Object key, Object value) throws Exception;

   /**
    * Inserts all elements of attributes into the attributes hashmap of the given node, overwriting existing
    * attributes, but not clearing the existing hashmap before insertion (making it a union of existing and
    * new attributes)
    * If the node does not exist, all parent nodes from the root down are created automatically
    * @param name The fully qualified name of the node
    * @param attributes A Map of attributes. Can be null
    */
   void put(Fqn name, Map attributes) throws Exception;

   /**
    * Inserts all modifications to the backend store. Overwrite whatever is already in
    * the datastore.
    * @param modifications A List<Modification> of modifications
    * @throws Exception
    */
   void put(List modifications) throws Exception;

   /** Removes the given key and value from the attributes of the given node. No-op if node doesn't exist */
   void remove(Fqn name, Object key) throws Exception;

   /**
    * Removes the given node. If the node is the root of a subtree, this will recursively remove all subnodes,
    * depth-first
    */
   void remove(Fqn name) throws Exception;

   /** Removes all attributes from a given node, but doesn't delete the node itself */
   void removeData(Fqn name) throws Exception;


   /**
    * Prepare the modifications. For example, for a DB-based CacheLoader:
    * <ol>
    * <li>Create a local (JDBC) transaction
    * <li>Associate the local transaction with <code>tx</code> (tx is the key)
    * <li>Execute the coresponding SQL statements against the DB (statements derived from modifications)
    * </ol>
    * For non-transactional CacheLoader (e.g. file-based), this could be a null operation
    * @param tx            The transaction, just used as a hashmap key
    * @param modifications List<Modification>, a list of all modifications within the given transaction
    * @param one_phase     Persist immediately and (for example) commit the local JDBC transaction as well. When true,
    *                      we won't get a {@link #commit(Object)} or {@link #rollback(Object)} method call later
    */
   void prepare(Object tx, List modifications, boolean one_phase) throws Exception;

   /**
    * Commit the transaction. A DB-based CacheLoader would look up the local JDBC transaction asociated
    * with <code>tx</code> and commit that transaction<br/>
    * Non-transactional CacheLoaders could simply write the data that was previously saved transiently under the
    * given <code>tx</code> key, to (for example) a file system (note this only holds if the previous prepare() did
    * not define one_phase=true
    */
   void commit(Object tx) throws Exception;

   /**
    * Roll the transaction back. A DB-based CacheLoader would look up the local JDBC transaction asociated
    * with <code>tx</code> and roll back that transaction
    */
   void rollback(Object tx);

   /**
    * Fetch the entire state for this cache from secondary storage (disk, DB) and return it as a byte buffer.
    * This is for initialization of a new cache from a remote cache. The new cache would then call
    * storeEntireState()
    * todo: define binary format for exchanging state
    */
   byte[] loadEntireState() throws Exception;

   /** Store the given state in secondary storage. Overwrite whatever is currently in storage */
   void storeEntireState(byte[] state) throws Exception;
}

public interface Service {
   void create() throws Exception;

   void start() throws Exception;

   void stop();

   void destroy();
}

    <!-- ==================================================================== -->
    <!-- Defines TreeCache configuration                                      -->
    <!-- ==================================================================== -->

    <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
       <attribute name="CacheLoaderClass">org.jboss.cache.loader.bdbje.BdbjeCacheLoader</attribute>
       <!-- attribute name="CacheLoaderClass">org.jboss.cache.loader.FileCacheLoader</attribute -->
       <attribute name="CacheLoaderConfig" replace="false">
         location=c:\\tmp\\bdbje 
       </attribute>
       <attribute name="CacheLoaderShared">true</attribute>
       <attribute name="CacheLoaderPreload">/</attribute>
       <attribute name="CacheLoaderFetchTransientState">false</attribute>
       <attribute name="CacheLoaderFetchPersistentState">true</attribute>
       <attribute name="CacheLoaderAsynchronous">true</attribute>
    </mbean>

    <!-- ==================================================================== -->
    <!-- Defines TreeCache configuration                                      -->
    <!-- ==================================================================== -->

    <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
       ...
       <attribute name="CacheLoaderClass">org.jboss.cache.loader.FileCacheLoader</attribute>
       ...
       <attribute name="CacheLoaderPassivation">true</attribute>
       ...
    </mbean>

9.4. CacheLoader use cases

9.4.1. Local cache with store

This is the simplest case. We have a JBossCache instance, whose mode is LOCAL, therefore no replication is going on. The CacheLoader simply loads non-existing elements from the store and stores modifications back to the store. When the cache is started, depending on the CacheLoaderPreload attribute, certain data can be preloaded, so that the cache is partly warmed up.

When using TreeCacheAop, this means that entire POJOs can be stored to a database or a filesystem, and when accessing fields of a POJO, they will be lazily loaded using the CacheLoader to access a backend store. This feature effectively provides simple persistency for any POJO.

9.4.2. Replicated caches with all nodes sharing the same store

The following figure shows 2 JBossCache nodes sharing the same backend store:

Figure 3. 2 nodes sharing a backend store

Both nodes have a CacheLoader that accesses a common shared backend store. This could for example be a shared filesystem (using the FileCacheLoader), or a shared database. Because both nodes access the same store, they don't necessarily need state transfer on startup.^[8]Rather, theFetchStateOnStartup attribute could be set to false, resulting in a 'cold' cache, that gradually warms up as elements are accessed and loaded for the first time. This would mean that individual caches in a cluster might have different in-memory state at any given time (largely depending on their preloading and eviction strategies).

When storing a value, the writer takes care of storing the change in the backend store. For example, if node1 made change C1 and node2 C2, then node1 would tell its CacheLoader to store C1, and node2 would tell its CacheLoader to store C2.

9.4.3. Replicated caches with only one node having a store

Figure 4. 2 nodes but only one accesses the backend store

This is a similar case as the previous one, but here only one node in the cluster interacts with a backend store via its CacheLoader. All other nodes perform in-memory replication. A use case for this is HTTP session replication, where all nodes replicate sessions in-memory, and - in addition - one node saves the sessions to a persistent backend store. Note that here it may make sense for the CacheLoader to store changes asynchronously, that is not on the caller's thread, in order not to slow down the cluster by accessing (for example) a database. This is a non-issue when using asynchronous replication.

9.4.4. Replicated caches with each node having its own store

Figure 5. 2 nodes each having its own backend store

Here, each node has its own datastore. Modifications to the cache are (a) replicated across the cluster and (b) persisted using the CacheLoader. This means that all datastores have exactly the same state. When replicating changes synchronously and in a transaction, the two phase commit protocol takes care that all modifications are replicated and persisted in each datastore, or none is replicated and persisted (atomic updates).

Note that currently JBossCache is not an XAResource, that means it doesn't implement recovery. When used with a TransactionManager that supports recovery, this functionality is not available.

The challenge here is state transfer: when a new node starts it needs to do the following:

1. Tell the coordinator (oldest node in a cluster) to send it the state

2. The coordinator then needs to wait until all in-flight transactions have completed. During this time, it will not allow for new transactions to be started.

3. Then the coordinator asks its CacheLoader for the entire state using loadEntireState(). It then sends back that state to the new node.

4. The new node then tells its CacheLoader to store that state in its store, overwriting the old state. This is the CacheLoader.storeEntireState() method

5. As an option, the transient (in-memory) state can be transferred as well during the state transfer.

6. The new node now has the same state in its backend store as everyone else in the cluster, and modifications received from other nodes will now be persisted using the local CacheLoader.

9.4.5. Hierarchical caches

If you need to set up a hierarchy within a single VM, you can use the LocalDelegatingCacheLoader. This type of hierarchy can currently only be set up programmatically. The code below shows how a first-level cache delegates to a local second-level cache:

TreeCache firstLevel, secondLevel;
LocalDelegatingCacheLoader cache_loader;

// create and configure firstLevel
firstLevel=new TreeCache();

// create and configure secondLevel
secondLevel=new TreeCache();

// create DelegatingCacheLoader
cache_loader=new LocalDelegatingCacheLoader(secondLevel);

// set CacheLoader in firstLevel
firstLevel.setCacheLoader(cache_loader);

// start secondLevel
secondLevel.startService();

// start firstLevel
firstLevel.startService();

If you need to set up a hierarchy across VMs but within a cluster, you can use the RpcDelegatingCacheLoader, which delegates all cache loading requests from non-coordinator caches to the cluster's coordinator cache. The coordinator cache is the first cache in the cluster to come online. Note that if the coordinator cache leaves the cluster for any reason, the second cache in the cluster to come online becomes the coordinator and so on. The XML below shows how to configure a cluster using RpcDelegatingCacheLoader:

    <!-- ==================================================================== -->
    <!-- Defines TreeCache configuration                                      -->
    <!-- ==================================================================== -->

    <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
       ...
       <attribute name="CacheLoaderClass">org.jboss.cache.loader.RpcDelegatingCacheLoader</attribute>
       <attribute name="CacheLoaderConfig"></attribute>
       <attribute name="CacheLoaderShared">false</attribute>
       <attribute name="CacheLoaderPreload"></attribute>
       <attribute name="CacheLoaderFetchTransientState">false</attribute>
       <attribute name="CacheLoaderFetchPersistentState">false</attribute>
       ...
    </mbean>

If you need to set up a hierarchy across VMs without any cluster membership restrictions, starting with JBossCache 1.2.2 you will be able to use the RmiDelegatingCacheLoader.

<attribute name="CacheLoaderClass">org.jboss.cache.loader.JDBCCacheLoader</attribute>
<attribute name="CacheLoaderShared">true</attribute>
<attribute name="CacheLoaderPreload">/product/catalog/us,/product/catalog/asia</attribute>
<attribute name="CacheLoaderFetchTransientState">false</attribute>
<attribute name="CacheLoaderFetchPersistentState">false</attribute>
<attribute name="CacheLoaderConfig">
    cache.jdbc.table.name=jbosscache
    cache.jdbc.table.create=true
    cache.jdbc.table.drop=true
    cache.jdbc.fqn.column=fqn
    cache.jdbc.fqn.type=varchar(255)
    cache.jdbc.node.column=node
    cache.jdbc.node.type=blob
    cache.jdbc.parent.column=parent
    cache.jdbc.driver=oracle.jdbc.OracleDriver
    cache.jdbc.url=jdbc:oracle:thin:@localhost:1521:JBOSSDB
    cache.jdbc.user=jboss
    cache.jdbc.password=sa
</attribute>

<attribute name="CacheLoaderClass">org.jboss.cache.loader.JDBCCacheLoader</attribute>
<attribute name="CacheLoaderShared">true</attribute>
<attribute name="CacheLoaderPreload">/product/catalog/us,/product/catalog/asia</attribute>
<attribute name="CacheLoaderFetchTransientState">false</attribute>
<attribute name="CacheLoaderFetchPersistentState">false</attribute>
<attribute name="CacheLoaderConfig">
    cache.jdbc.datasource=java:/DefaultDS
</attribute>

In order to deserialize an object replicated to it from a remote cache, a TreeCache instance needs to have access to the classloader that defines the object's class. This is simple if the TreeCache's own classloader can access the required classes, but for situations where JBossCache is used as a service supporting clients that use different classloaders, the cache can be configured to use TreeCacheMarshaller.

TreeCacheMarshaller allows application code to register a classloader that should be used to handle replication for a portion of the tree.

/**
 * Sets whether a {@link TreeCacheMarshaller} instance should be created
 * to manage different classloaders to use for unmarshalling replicated
 * objects.
 * <p>
 * This property must be set to true before any call to
 * {@link #registerClassLoader(String, ClassLoader)} or
 * {@link #activateRegion(String)}
 * </p>
 */
void setUseMarshalling(boolean isTrue);

/**
 * Gets whether a {@link TreeCacheMarshaller} instance should be used
 * to manage different classloaders to use for unmarshalling replicated
 * objects.
 */
boolean getUseMarshalling();

/**
 * Registers the given classloader with TreeCacheMarshaller for 
 * use in unmarshalling replicated objects for the specified region.
 * 
 * @param fqn The fqn region. This fqn and its children will use this classloader for (un)marshalling.
 * @param cl The class loader to use
 * 
 * @throws RegionNameConflictException if fqn is a descendant of
 *                                     an FQN that already has a classloader
 *                                     registered.
 * @throws IllegalStateException if useMarshalling is false
 */
void registerClassLoader(String fqn, ClassLoader cl) throws RegionNameConflictException;

/**
 * Instructs the TreeCacheMarshaller to no longer use a special
 * classloader to unmarshal replicated objects for the specified region.
 * 
 * @param fqn The fqn of the root node of region.
 * 
 * @throws RegionNotFoundException if no classloader has been registered for
 *                                 fqn. 
 * @throws IllegalStateException if useMarshalling is false
 */
void unregisterClassLoader(String fqn) throws RegionNotFoundException;
	 		

<attribute name="UseMarshalling">true</attribute>

  
/**
 * Sets whether the entire tree is inactive upon startup, only responding
 * to replication messages after {@link #activateRegion(String)} is
 * called to activate one or more parts of the tree.
 * <p>
 * This property is only relevant if {@link #getUseMarshalling()} is
 * true.
 *
 */
public void setInactiveOnStartup(boolean inactiveOnStartup);
  
/**
 * Gets whether the entire tree is inactive upon startup, only responding
 * to replication messages after {@link #activateRegion(String)} is
 * called to activate one or more parts of the tree.
 * <p>
 * This property is only relevant if {@link #getUseMarshalling()} is
 * true.
 */
public boolean isInactiveOnStartup();   

/**
 * Causes the cache to transfer state for the subtree rooted at
 * subtreeFqn and to begin accepting replication messages
 * for that subtree.
 * <p>
 * <strong>NOTE:</strong> This method will cause the creation of a node
 * in the local tree at subtreeFqn whether or not that
 * node exists anywhere else in the cluster.  If the node does not exist
 * elsewhere, the local node will be empty.  The creation of this node will 
 * not be replicated.
 * 
 * @param subtreeFqn Fqn string indicating the uppermost node in the
 *                   portion of the tree that should be activated.
 *                   
 * @throws RegionNotEmptyException if the node subtreeFqn 
 *                                 exists and has either data or children
 *                                 
 * @throws IllegalStateException 
 *       if {@link #getUseMarshalling() useMarshalling} is false
 */
public void activateRegion(String subtreeFqn) 
    throws RegionNotEmptyException, RegionNameConflictException, CacheException;
  
/**
 * Causes the cache to stop accepting replication events for the subtree 
 * rooted at subtreeFqn and evict all nodes in that subtree.
 * 
 * @param subtreeFqn Fqn string indicating the uppermost node in the
 *                   portion of the tree that should be activated.
 * @throws RegionNameConflictException if subtreeFqn indicates
 *                                     a node that is part of another
 *                                     subtree that is being specially 
 *                                     managed (either by activate/inactiveRegion()
 *                                     or by registerClassLoader())
 * @throws CacheException if there is a problem evicting nodes
 *                                 
 * @throws IllegalStateException 
 *       if {@link #getUseMarshalling() useMarshalling} is false
 */
public void inactivateRegion(String subtreeFqn) throws RegionNameConflictException, CacheException;     	
         

<attribute name="InactiveOnStartup">true</attribute>

<?xml version="1.0" encoding="UTF-8" ?> 
<server>
  <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" /> 

  <!--  ====================================================================  --> 
  <!--  Defines TreeCache configuration                                       --> 
  <!--  ====================================================================  --> 
  <mbean code="org.jboss.cache.TreeCache" name="com.xyz.cache:service=SharedCache">
    
    .......

    <!-- Configure Marshalling --> 
    <attribute name="UseMarshalling">true</attribute>
    <attribute name="InactiveOnStartup">true</attribute>
    
    ........
    
  </mbean>
</server>
				

package example;

import javax.management.MalformedObjectNameException;
import javax.management.ObjectName;
import javax.servlet.ServletContextEvent;
import javax.servlet.ServletContextListener;

import org.jboss.cache.TreeCacheMBean;
import org.jboss.mx.util.MBeanProxyExt;

public class ActiveInactiveRegionExample implements ServletContextListener
{
   private TreeCacheMBean cache;

   public void contextInitialized(ServletContextEvent arg0) {
      try {
         findCache();
         
         cache.registerClassLoader("/example", Thread.currentThread().getContextClassLoader());
         cache.activeRegion("/example");
      }
      catch (Exception e) {
         // ... handle exception
      }

   }

   public void contextDestroyed(ServletContextEvent arg0) {
      cache.inactivateRegion("/example");
      cache.unregisterClassLoader("/example");
   }
   
   private void findCache() throws MalformedObjectNameException {
      // Find the shared cache service in JMX and create a proxy to it
      ObjectName cacheServiceName_ = new ObjectName("com.xyz.cache:service=SharedCache");
      // Create Proxy-Object for this service
      cache = (TreeCacheMBean) MBeanProxyExt.create(TreeCacheMBean.class, cacheServiceName_);
   }
}
				

   /**
    * Fetch a portion of the state for this cache from secondary storage 
    * (disk, DB) and return it as a byte buffer.
    * This is for activation of a portion of new cache from a remote cache. 
    * The new cache would then call {@link #storeState(byte[], Fqn)}.
    *
    * @param subtree Fqn naming the root (i.e. highest level parent) node of
    *                the subtree for which state is requested.
    *                
    * @see org.jboss.cache.TreeCache#activateRegion(String)
    */
   byte[] loadState(Fqn subtree) throws Exception;
   
   /**
    * Store the given portion of the cache tree's state in secondary storage. 
    * Overwrite whatever is currently in secondary storage.
    * 
    * @param state   the state to store
    * @param subtree Fqn naming the root (i.e. highest level parent) node of
    *                the subtree included in state.
    */   
   void storeState(byte[] state, Fqn subtree) throws Exception;
   
   /**
    * Sets the {@link RegionManager} this object should use to manage 
    * marshalling/unmarshalling of different regions using different
    * classloaders.
    * <p>
    * <strong>NOTE:</strong> This method is only intended to be used
    * by the TreeCache instance this cache loader is
    * associated with.
    * </p>
    * 
    * @param manager    the region manager to use, or null.
    */
   void setRegionManager(RegionManager manager);
	 		

All properties of the cache are configured via setters and can be retrieved via getters. This can be done either manually, or via the PropertyConfigurator and an XML file. A sample configuration file is shown below (stripped of comments etc):

<?xml version="1.0" encoding="UTF-8" ?> 
<server>
  <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" /> 

  <!--  ====================================================================  --> 
  <!--  Defines TreeCache configuration                                       --> 
  <!--  ====================================================================  --> 
  <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
    <depends>jboss:service=Naming</depends> 
    <depends>jboss:service=TransactionManager</depends> 


    <!-- Configure the TransactionManager --> 
    <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute> 

    <!-- 
            Node locking level : SERIALIZABLE
                                 REPEATABLE_READ (default)
                                 READ_COMMITTED
                                 READ_UNCOMMITTED
                                 NONE        
    --> 
    <attribute name="IsolationLevel">REPEATABLE_READ</attribute> 

    <!--     Valid modes are LOCAL
                             REPL_ASYNC
                             REPL_SYNC
    --> 
    <attribute name="CacheMode">LOCAL</attribute>
 
    <!-- Name of cluster. Needs to be the same for all clusters, in order
             to find each other --> 
    <attribute name="ClusterName">TreeCache-Cluster</attribute> 

    <attribute name="ClusterConfig">
      <config>
        <!-- UDP: if you have a multihomed machine,
                set the bind_addr attribute to the appropriate NIC IP address 
        --> 
        <!-- UDP: On Windows machines, because of the media sense feature
                 being broken with multicast (even after disabling media sense)
                 set the loopback attribute to true 
        --> 
        <UDP mcast_addr="228.1.2.3" mcast_port="45566" ip_ttl="64" ip_mcast="true" 
           mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000" 
           ucast_recv_buf_size="80000" loopback="false" /> 
        <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false" /> 
        <MERGE2 min_interval="10000" max_interval="20000" /> 
        <FD shun="true" up_thread="true" down_thread="true" /> 
        <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false" /> 
        <pbcast.NAKACK gc_lag="50" max_xmit_size="8192" retransmit_timeout="600,1200,2400,4800" up_thread="false"
           down_thread="false" /> 
        <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false" />
        <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" />
        <FRAG frag_size="8192" down_thread="false" up_thread="false" />
        <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true" /> 
        <pbcast.STATE_TRANSFER up_thread="false" down_thread="false" /> 
      </config>
    </attribute>

    <!--    The max amount of time (in milliseconds) we wait until the
            initial state (ie. the contents of the cache) are retrieved from
            existing members in a clustered environment
    --> 
    <attribute name="InitialStateRetrievalTimeout">5000</attribute> 

    <!--    Number of milliseconds to wait until all responses for a
            synchronous call have been received.
    --> 
    <attribute name="SyncReplTimeout">10000</attribute> 

    <!--  Max number of milliseconds to wait for a lock acquisition --> 
    <attribute name="LockAcquisitionTimeout">15000</attribute> 

    <!--  Name of the eviction policy class. --> 
    <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute> 

    <!--  Specific eviction policy configurations. This is LRU --> 
    <attribute name="EvictionPolicyConfig">
      <config>
        <attribute name="wakeUpIntervalSeconds">5</attribute> 
        <!--  Cache wide default --> 
        <region name="/_default_">
         <attribute name="maxNodes">5000</attribute> 
         <attribute name="timeToLiveSeconds">1000</attribute> 
         <!-- Maximum time an object is kept in cache regardless of idle time -->
         <attribute name="maxAgeSeconds">120</attribute>
       </region>

       <region name="/org/jboss/data">
         <attribute name="maxNodes">5000</attribute> 
         <attribute name="timeToLiveSeconds">1000</attribute> 
       </region>

       <region name="/org/jboss/test/data">
         <attribute name="maxNodes">5</attribute> 
         <attribute name="timeToLiveSeconds">4</attribute> 
       </region>
      </config>
    </attribute>

    <attribute name="CacheLoaderClass">org.jboss.cache.loader.bdbje.BdbjeCacheLoader</attribute>
    <!--       <attribute name="CacheLoaderClass">org.jboss.cache.loader.FileCacheLoader</attribute>-->
    <attribute name="CacheLoaderConfig" replace="false">
       location=c:\\tmp
    </attribute>
    <attribute name="CacheLoaderShared">true</attribute>
    <attribute name="CacheLoaderPreload">/a/b/c,/all/my/objects</attribute>
    <attribute name="CacheLoaderFetchTransientState">false</attribute>
    <attribute name="CacheLoaderFetchPersistentState">true</attribute>
    <attribute name="CacheLoaderPassivation">true</attribute>
    <attribute name="CacheLoaderAsynchronous">true</attribute>

  </mbean>
</server>     

The PropertyConfigurator.configure() method needs to have as argument a filename that needs to be found on the classpath, and will use it to configure the TreeCache from the properties defined in it. Note that this configuration file is used to configure the TreeCache both as a standalone cache, and as an MBean if run inside the JBoss container^[9] .

A list of properties is shown below:

If TreeCache is run inside JBoss then TreeCache is deployed as an MBean. The steps below illustrate how to do this. We do not deploy as a Service Archive (SAR), but as a JAR (jboss-cache.jar in the lib dir) and an XML file defining the MBean. But of course, TreeCache could also be deployed as a SAR, or even as part of a WAR, EJB or EAR.

First, the jboss-cache.jar file has to be copied to the lib directory and JBoss has to be restarted. Then a regular TreeCache configuration file in XML format has to be copied to the /deploy directory. The XML file format is the same as discussed in Configuration.

In order to be used from a client, e.g. a servlet in the Tomcat web container inside the same JBoss container, JMX can be used:

MBeanServer server=MBeanServerLocator.locateJBoss();
TreeCacheMBean cache;
cache=(TreeCacheMBean)MBeanProxyExt.create(TreeCacheMBean.class, "jboss.cache:service=TreeCache", server);
cache.put("/a/b/c", null);

The MBeanServerLocator class is a helper to find the (only) JBoss MBean server inside the current VM. The static create() method creates a dynamic proxy to the given interface and uses JMX to dynamically dispatch methods invoked against the generated interface. The name used to lookup the MBean is the same as defined in the configuration file.

 <mbean
      code="org.jboss.invocation.jrmp.server.JRMPProxyFactory"
      name="mydomain:service=proxyFactory,type=jrmp,target=factory">
      <attribute
      name="InvokerName">jboss:service=invoker,type=jrmp</attribute>
      <attribute
      name="TargetName">jboss.cache:service=TreeCache</attribute>
      <attribute name="JndiName">MyCache</attribute> <attribute
      name="InvokeTargetMethod">true</attribute> <attribute
      name="ExportedInterface">org.jboss.cache.TreeCacheMBean</attribute>
      <attribute name="ClientInterceptors"> <iterceptors>
      <interceptor>org.jboss.proxy.ClientMethodInterceptor</interceptor>
      <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
      <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
      </iterceptors> </attribute>
      <depends>jboss:service=invoker,type=jrmp</depends>
      <depends>jboss.cache:service=TreeCache</depends>
      </mbean>