This guide will show you possible ways of improving JCR performance.
It is intended to GateIn Administrators and those who wants to use JCR features.
EC2 network: 1Gbit
Servers hardware:
| 7.5 GB memory |
| 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) |
| 850 GB instance storage (2×420 GB plus 10 GB root partition) |
| 64-bit platform |
| I/O Performance: High |
| API name: m1.large |
Note
NFS and statistics (cacti snmp) server were located on one physical server.
JBoss AS configuration
JAVA_OPTS: -Dprogram.name=run.sh -server -Xms4g -Xmx4g
-XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000 -XX:+UseParallelGC
-Djava.net.preferIPv4Stack=true
Benchmark test using webdav (Complex read/write load test (benchmark)) with 20K same file. To obtain per-operation results we have used custom output from the testscase threads to CSV file.
Read operation:
| Warm-up iterations: 100 |
| Run iterations: 2000 |
| Background writing threads: 25 |
| Reading threads: 225 |
Table 51.1.
| Nodes count | tps | Responses >2s | Responses >4s |
|---|---|---|---|
| 1 | 523 | 6.87% | 1.27% |
| 2 | 1754 | 0.64% | 0.08% |
| 3 | 2388 | 0.49% | 0.09% |
| 4 | 2706 | 0.46% | 0.1% |
Read operaion with more threads:
| Warm-up iterations: 100 |
| Run iterations: 2000 |
| Background writing threads: 50 |
| Reading threads: 450 |
You can use maxThreads parameter to
increase maximum amount of threads that can be launched in AS instance.
This can improve performance if you need a high level of concurrency.
also you can use -XX:+UseParallelGC java directory to use
paralel garbage collector.
Tip
Beware of setting maxThreads too big,
this can cause OutOfMemoryError. We've
got it with maxThreads=1250 on such machine:
| 7.5 GB memory |
| 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each) |
| 850 GB instance storage (2×420 GB plus 10 GB root partition) |
| 64-bit platform |
| I/O Performance: High |
| API name: m1.large |
| java -Xmx 4g |
Cache size
JCR-cluster implementation is built using JBoss Cache as distributed, replicated cache. But there is one particularity related to remove action in it. Speed of this operation depends on the actual size of cache. As many nodes are currently in cache as much time is needed to remove one particular node (subtree) from it.
Eviction
Manipulations with eviction wakeUpInterval
value doestn't affect on performance. Performance results with values
from 500 up to 3000 are approximately equal.
Transaction Timeout
Using short timeout for long transactions such as Export/Import,
removing huge subtree defined timeout may cause
TransactionTimeoutException. [TODO] put
recomended timeout value
For performance it is better to have loadbalacer, DB server and shared NFS on different computers. If in some reasons you see that one node gets more load than others you can decrease this load using load value in load balancer.
JGroups configuration
It's recommended to use "multiplexer stack" feature present in JGroups. It is set by default in eXo JCR and offers higher performance in cluster, using less network connections also. If there are two or more clusters in your network, please check that they use different ports and different cluster names.
Write performance in cluster
Exo JCR implementation uses Lucene indexing engine to provide search capabilities. But Lucene brings some limitations for write operations: it can perform indexing only in one thread. Thats why write performance in cluster is not higher than in singleton environment. Data is indexed on coordinator node, so increasing write-load on cluster may lead to ReplicationTimeout exception. It occurs because writing threads queue in the indexer and under high load timeout for replication to coordinator will be exceeded.
Taking in consideration this fact, it is recommended to exceed
replTimeout value in cache configurations in case
of high write-load.
Replication timeout
Some operations may take too much time. So if you get
ReplicationTimeoutException try
increasing replication timeout:
<clustering mode="replication" clusterName="${jbosscache-cluster-name}">
...
<sync replTimeout="60000" />
</clustering>
value is set in miliseconds.
PermGen space size
If you intend to use Infinispan, you will have to increase the PermGen size to at least 256 Mo due to the latest versions of JGroups that are needed by Infinispan (please note that Infinspan is only dedicated to the community for now, no support will be provided). In case, you intend to use JBoss Cache, you can keep on using JGroups 2.6.13.GA which means that you don't need to increase the PermGen size.