This performance benchmark report will cover three separate areas of performance related to JBoss Remoting. Section 3 (All transports - speed results will cover basic performance when making synchronous invocations from a single client to a server and will include results for (1) all of the JBoss Remoting transports, (2) two raw Java transports, and (3) two Spring Remoting transports.[1] Section 4 (All JBoss Remoting transports - all invocation types) will cover performance differences among the various JBoss Remoting transports when making different types of client to server invocations (i.e. synchronous and asynchronous). Section 5 (All transports - scalability) will cover how well the JBoss Remoting transports, raw Java transports, and Spring Remoting transports scale as more clients are added.
This report is NOT intended to indicate how fast different transports are in absolute terms, but to compare speed and scalability of transports relative to one another. The tests were run on laptops and the results might not be indicative of performance levels achievable in a production environment. However, since all tests in each group were run on the same hardware, using the same environment and test scenario, the results should be sufficient for side by side comparisons. All the test result data are included at the end of this report along with instructions on how to reproduce the test runs in any other environment.
Since there is a limited amount of space within the graphs for full descriptions of the transports and test scenarios, the abbreviations are explained here.
socket_java: JBoss Remoting using the socket transport and default Java serialization
socket_jboss: JBoss Remoting using the socket transport and JBoss serialization
rmi_java: JBoss Remoting using the rmi transport and default Java serialization
rmi_jboss: JBoss Remoting using the rmi transport and JBoss serialization
http_java: JBoss Remoting using the http transport and default Java serialization
http_jboss: JBoss Remoting using the http transport and JBoss serialization
multiplex_java: JBoss Remoting using the multiplex transport and default Java serialization
multiplex_jboss: JBoss Remoting using the multiplex transport and JBoss serialization
raw_socket: uses straight Java Socket API with Java serialization
raw_rmi: uses straight Java RMI API with Java serialization
spring_rmi: Spring Remoting using the Spring rmi transport
spring_http: Spring Remoting using the Spring http transport
In the first performance test a single client continuously makes synchronous invocations on a single server. This test shows how quickly the transport can make remote invocations when not under load (basically showing raw data transfer speed). Three test runs were executed for each transport, and the averages are given in the following graph.
The graph shows that the JBoss Remoting socket transport with JBoss Serialization is able to make the largest number of requests per second, followed by the raw rmi and raw socket transports (which use the Java API directly and not a remoting framework).
The second performance test also consists of a single client making invocations on a single server, but the test runs span two dimensions: transport and invocation type. The transports exercised in this case are the JBoss Remoting transports, and the invocation types are the following:
Multi-threaded sync: uses 10 threads to make concurrent synchronous invocations from the client to the server. Is very similar in load to using 10 individual clients.
Async - client side: uses single threaded client to make asynchronous invocations from the client to the server. The calling thread will hand off the invocation to a client side pool of worker threads that make the actual client to server invocation.
Single threaded sync: uses single threaded client to make synchronous invocations from the client to the server.
Async - server side: uses single threaded client to make asynchronous invocations from the client to the server. The calling thread will make the actual network call to the server, where the invocation payload will be handed off to a worker thread pool where it will be picked up for processing.
The following graph indicates how long it took each JBoss Remoting transport to make 100,000 client to server invocations for each invocation type. The lower the time, the faster the combination of transport and invocation type.
The results show that the JBoss Remoting socket transport with JBoss Serialization was the fastest transport over all the different invocation types.
The final performance test is designed to indicate how well the various transports scale in response to increased load from additional clients. Each client uses a single thread to make 100,000 synchronous invocations on the server. The following graphs indicate the total number of invocations per second processed by the server as it services up to thirty clients. The numbers are averaged over three runs. For the sake of readability the results are partitioned by serialization type. [2][3]
Several phenomena may be observed in these graphs.
Since the cpu is kept at close to 100% utilization in these tests (except, perhaps, in the case of a small number of clients, depending on the speed of the hardware), the ideal in scalability would be for the total number of invocations processed by the server to remain flat as the number of clients increases, indicating that there is no overhead cost for additional clients. Nearly every transport/serialization combination shows the same pattern of an initial spike (until the cpu is saturated, presumably) followed by a gentle decline from 10 to 20 clients and a smaller decline from 20 to 30 clients. For example, the performance of the JBoss Remoting socket transport with Java serialization declines 6.6% from 10 to 20 clients and 3.2% from 20 to 30 clients. The most significant exception is the JBoss Remoting transport with JBoss serialization, which remains nearly flat as the number of clients increases from 5 to 30, decreasing by only 2.1%.
When using Java serialization none of the Remoting framework transports, JBoss or Spring, achieve the performance of the raw socket and raw rmi transports. However, the JBoss Remoting socket transport with JBoss serialization outperforms both of the raw transports at 10 clients and beyond.
The JBoss Remoting socket transport with Java serialization and the Spring rmi transport show almost identical results. The JBoss Remoting http transport with Java serialization and the Spring http transport show nearly identical results as well.
Among all of the Remoting framework transports, the JBoss Remoting socket transport with JBoss serialization is the clear favorite in these tests.
All the performance tests run for this report have the same basic structure and most share the same base code (the most notable exception being the Spring Remoting http transport, which requires a web container for server side deployment). All tests have one or more clients, each of which starts by creating a callback server and then sending the total number of invocations it intends to make. The server will then create a call tracker that will manage the test invocations as they come in from that client, ensuring that all intended invocations are received without duplicates. The server will also create a client to call back on the callback server created within that client instance.
Once the client and server have been fully initialized with their initial configuration data, the client will begin its loop to make test invocations. The number of test invocations is configurable, but for this report, the clients each make 100,000 test invocations. The payload size for each invocation is configurable as well, but for this report will be approximately 1024 bytes (plus size of payload wrapper for each transport). The time it takes the client to make all its test invocations is the main result captured for this report. When the call tracker on the server side has received all the invocations from its associated client, it will call back on that client (actually the client's callback server) and send the number of test invocations it received and number of duplicate test invocations, if any. This will indicate test run completion from the client's perspective.
All the transport tests (except the Spring Remoting http transport) are executed using the JBoss JRunit test framework. This means that each test run can be executed within ant as a junit task, which will call on a JRunit test driver which will spawn one or more client test instances and a server test instance. All configuration information (as seen below) can be modified within the JBoss Remoting build.xml file and tests can be executed via running the JBoss Remoting ant script. All the performance test source code can be found within the org.jboss.test.remoting.performance package of the JBoss Remoting project. Everything needed to run all the transport tests are included within the JBoss Remoting project (except a web container, which is needed for the Spring Remoting http transport tests).
The first two tests (speed test for all transports and invocation type tests) were run in the following environment.
Hardware: Dell Inspiron 8600, Intel Pentium processor 1.69GHz, 2GB RAM
Operating System: Microsoft Windows XP Professional (Version 2002, Service Pack 2)
Java Virtual Machine: Sun, Java HotSpot(TM) Client VM (build 1.5.0_03-b07, mixed mode, sharing)
Software:
JBoss Remoting 2.0.0.GA
Spring 1.2.8
Tomcat 5.5.17 (for Spring Remoting http transport tests)
Test script configuration:
numofcalls = 100000
payloadsize = 1024
numofclients = 1
The third test (scalability) was run in the following environment.
Hardware: Dell Inspiron 9400, Intel Core Duo T2400 processor 1.83GHz, 2GB RAM
Operating System: Microsoft Windows XP Professional (Version 2002, Service Pack 2)
Java Virtual Machine: Sun, Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode, sharing)
Software:
JBoss Remoting 2.0.0.GA
Spring 1.2.8
Tomcat 5.5.17 (for Spring Remoting http transport tests)
Test script configuration:
numofcalls = 100000
payloadsize = 1024
numofclients = 1, 5, 10, 20, 30
The test result data including all the junit result files (in xml format) and the MS Excel spreadsheet used to capture raw result data, calculate averages, and generate graphs can be found within test_results.zip.
[1] It was not possible to test the Spring Remoting hessian and burlap transports as the versions shipped with 1.2.8 of Spring do not support sending of Externalizable objects, which is what the test payload object implements.
[2] The raw_rmi and raw_socket transports use only Java serialization. They are included in the JBoss serialization table to aid in comparison.
[3] The spring transports use only Java serialization. They are included in the JBoss serialization table to aid in comparison.