This article provides detailed information about the first version of the custom TCP client/server Hot Rod protocol.
This version of the protocol is implemented since Infinispan 4.1.0.Final
All key and values are sent and stored as byte arrays. Hot Rod makes no assumptions about their types. Some clarifications about the other types:
vInt: Refers to unsigned variable length integer values as specified in here. They're between 1 and 5 bytes long.
vLong: Refers to unsigned variable length long values similar to vInt but applied to longer values. They're between 1 and 9 bytes long.
String: Strings are always represented using UTF-8 encoding.
The header for a request is composed of:
|
Magic [1b] |
Message Id [vLong] |
Version [1b] |
Opcode [1b] |
Cache Name Length [vInt] |
Cache Name [string] |
Flags [vInt] |
Client Intelligence [1b] |
Topology Id [vInt] |
Transaction Type [1b] |
Transaction Id [byte-array] |
Magic : Possible values are:
0xA0 - Infinispan Cache Request Marker
0xA1 - Infinispan Cache Response Marker
Message Id : Id of the message that will be copied back in the response. This allows for hot rod clients to implement the protocol in an asynchronous way.
Version : Infinispan hot rod server version. In this particular case, this is 10
Opcode : Possible values are only the ones on the request column:
|
Request operation codes |
Response operation codes |
|
0x01 - put request |
0x02 - put response |
|
0x03 - get request |
0x04 - get response |
|
0x05 - putIfAbsent request |
0x06 - putIfAbsent response |
|
0x07 - replace request |
0x08 - replace response |
|
0x09 - replaceIfUnmodified request |
0x0A - replaceIfUnmodified response |
|
0x0B - remove request |
0x0C - remove response |
|
0x0D - removeIfUnmodified request |
0x0E - removeIfUnmodified response |
|
0x0F - containsKey request |
0x10 - containsKey response |
|
0x11 - getWithVersion request |
0x12 - getWithVersion response |
|
0x13 - clear request |
0x14 - clear response |
|
0x15 - stats request |
0x16 - stats response |
|
0x17 - ping request |
0x18 - ping response |
|
0x19 - bulkGet request |
0x1A - bulkGet response |
|
- |
0x50 - error response |
Cache Name Length : Length of cache name. If the passed length is 0 (followed by no cache name), the operation will interact with the default cache.
Cache Name : Name of cache on which to operate. This name must match the name of predefined cache in the Infinispan configuration file.
Flags : A variable length number representing flags passed to the system. Each flags is represented by a bit. Note that since this field is sent as variable length, the most significant bit in a byte is used to determine whether more bytes need to be read, hence this bit does not represent any flag. Using this model allows for flags to be combined in a short space. Here are the current values for each flag:
|
0x0001 |
ForceReturnPreviousValue |
Client Intelligence : This byte hints the server on the client capabilities:
0x01 - basic client, interested in neither cluster nor hash information
0x02 - topology-aware client, interested in cluster information
0x03 - hash-distribution-aware client, that is interested in both cluster and hash information
Topology Id : This field represents the last known view in the client. Basic clients will only send 0 in this field. When topology-aware or hash-distribution-aware clients will send 0 until they have received a reply from the server with the current view id. Afterwards, they should send that view id until they receive a new view id in a response
Transaction Type : This is a 1 byte field, containing one of the following well-known supported transaction types (For this version of the protocol, the only supported transaction type is 0):
0 - Non-transactional call, or client does not support transactions. The subsequent TX_ID field will be omitted.
1 - X/Open XA transaction ID (XID). This is a well-known, fixed-size format.
Transaction Id : The byte array uniquely identifying the transaction associated to this call. It's length is determined by the transaction type. If transaction type is 0, no transaction id will be present.
|
Magic [1b] |
Message Id [vLong] |
Op code [1b] |
Status [1b] |
Topology Change Marker [1b] |
Opcode : Op code representing a response to a particular operation, or error condition.
Status : Status of the response, possible values:
|
0x00 - No error |
0x01 - Not put/removed/replaced |
0x02 - Key does not exist |
|
0x81 - Invalid magic or message id |
0x82 - Unknown command |
0x83 - Unknown version |
|
0x84 - Request parsing error |
0x85 - Server Error |
0x86 - Command timed out |
Exceptional error status responses, those that start with 0x8..., are followed by the length of the error message (as a vInt) and error message itself as String.
Topology Change Marker : This is a marker byte that indicates whether the response is prepended with topology change information. When no topology change follows, the content of this byte is 0. If a topology change follows, its contents are 1.
The following section discusses how the response headers look for topology-aware or hash-distribution-aware clients when there's been a cluster or view formation change. Note that it's the server that makes the decision on whether it sends back the new topology based on the current topology id and the one the client sent. If they're different, it will send back the new topology.
This is what topology-aware clients receive as response header when a topology change is sent back:
|
Response header with topology change marker |
Topology Id [vInt] |
Num servers in topology [vInt] |
|
m1: Host/IP length [vInt] |
m1: Host/IP address [string] |
m1: Port [2b - Unsigned Short] |
|
m2: Host/IP length [vInt] |
m2: Host/IP address [string] |
m2: Port [2b - Unsigned Short] |
|
...etc |
Num servers in topology : Number of Infinispan Hot Rod servers running within the cluster. This could be a subset of the entire cluster if only a fraction of those nodes are running Hot Rod servers.
Host/IP address length : Length of hostname or IP address of individual cluster member that Hot Rod client can use to access it. Using variable length here allows for covering for hostnames, IPv4 and IPv6 addresses.
Host/IP address : String containing hostname or IP address of individual cluster member that Hot Rod client can use to access it.
Port : Port that Hot Rod clients can use to communicat with this cluster member.
This is what hash-distribution-aware clients receive as response header when a topology change is sent back:
|
Response header with topology change marker |
Topology Id [vInt] |
Num Key Owners [2b - Unsigned Short] |
Hash Function Version [1b] |
Hash space size [vInt] |
Num servers in topology [vInt] |
|
m1: Host/IP length [vInt] |
m1: Host/IP address [string] |
m1: Port [2b - unsigned short] |
m1: Hashcode [4b] |
||
|
m2: Host/IP length [vInt] |
m2: Host/IP address [string] |
m2: Port [2b - unsigned short] |
m2: Hashcode [4b] |
||
|
...etc |
It's important to note that since hash headers rely on the consistent hash algorithm used by the server and this is a factor of the cache interacted with, hash-distribution-aware headers can only be returned to operations that target a particular cache. Currently ping command does not target any cache (this is to change as per ISPN-424, hence calls to ping command with hash-topology-aware client settings will return a hash-distribution-aware header with "Num Key Owners", "Hash Function Version", "Hash space size" and each individual host's hash code all set to 0. This type of header will also be returned as response to operations with hash-topology-aware client settings that are targeting caches that are not configured with distribution.
Number key owners : Globally configured number of copies for each Infinispan distributed key
Hash function version : Hash function version, pointing to a specific hash function in use. See Hot Rod hash functions for details.
Hash space size : Modulus used by Infinispan for for all module arithmetic related to hash code generation. Clients will likely require this information in order to apply the correct hash calculation to the keys.
Num servers in topology : If virtual nodes are disabled, this number represents the number of Hot Rod servers in the cluster. If virtual nodes are enabled, this number represents all the virtual nodes in the cluster which are calculated as (num configured virtual nodes) * (num cluster members). Regardless of whether virtual nodes are configured or not, the number represented by this field indicates the number of 'host:port:hashId' tuples to be read in the response.
Hashcode : 32 bit integer representing the hashcode of a cluster member that a Hot Rod client can use indentify in which cluster member a key is located having applied the CSA to it.
Common request format:
|
Header |
Key Length [vInt] |
Key [byte-array] |
Key Length : Length of key. Note that the size of a vint can be up to 5 bytes which in theory can produce bigger numbers than Integer.MAX_VALUE. However, Java cannot create a single array that's bigger than Integer.MAX_VALUE, hence the protocol is limiting vint array lengths to Integer.MAX_VALUE.
Key : Byte array containing the key whose value is being requested.
Response status:
0x00 - success, if key present/retrieved/removed
0x02 - if key does not exist
Get response:
|
Header |
Value Length [vInt] |
Value [byte-array] |
Value Length : Length of value
Value : The requested value. If key does not exist, status returned in 0x02. See encoding section for more info.
Remove response:
If ForceReturnPreviousValue has been passed, remove response will contain previous value (including value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.
ContainsKey response:
Empty
GetWithVersion response:
|
Header |
Entry Version [8b] |
Value Length [vInt] |
Value [byte-array] |
Entry Version : Unique value of an existing entry's modification. The protocol does not mandate that entry_version values are sequential. They just need to be unique per update at the key level.
Request format:
|
Header |
Entry Count [vInt] |
Entry Count : Maximum number of Infinispan entries to be returned by the server (entry == key + associated value). Needed to support CacheLoader.load(int). If 0 then all entries are returned (needed for CacheLoader.loadAll()).
Response:
|
Header |
More [1b] |
Key Size 1 |
Key 1 |
Value Size 1 |
Value 1 |
More [1b] |
Key Size 2 |
Key 2 |
Value Size 2 |
Value 2 |
More [1b] ... |
More : One byte representing whether more entries need to be read from the stream. So, when it's set to 1, it means that an entry followes, whereas when it's set to 0, it's the end of stream and no more entries are left to read.
For more information on BulkGet look here
Common request format:
|
Header |
Key Length [vInt] |
Key [byte-array] |
Lifespan [vInt] |
Max Idle [vInt] |
Value Length [vInt] |
Value [byte-array] |
Lifespan : Number of seconds that a entry during which the entry is allowed to life. If number of seconds is bigger than 30 days, this number of seconds is treated as UNIX time and so, represents the number of seconds since 1/1/1970. If set to 0, lifespan is unlimited.
Max Idle : Number of seconds that a entry can be idle before it's evicted from the cache. If 0, no max idle time.
Put response status:
0x00 if stored
Replace response status:
0x00 if stored
0x01 if store did not happen because key does not exist
PutIfAbsent response status:
0x00 if stored
0x01 if store did not happen because key was present
Put/PutIfAbsent/Replace response:
If ForceReturnPreviousValue has been passed, these responses will contain previous value (and corresponding value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.
Request format:
|
Header |
Key Length [vInt] |
Key [byte-array] |
Lifespan [vInt] |
Max Idle [vInt] |
Entry Version [8b] |
Value Length [vInt] |
Value [byte-array] |
Entry Version : Use the value returned by GetWithVersion operation.
Response status
0x00 status if replaced/removed
0x01 status if replace/remove did not happen because key had been modified
0x02 status if key does not exist
Response:
If ForceReturnPreviousValue has been passed, this responses will contain previous value (and corresponding value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.
Request format:
|
Header |
Key Length [vInt] |
Key [byte-array] |
Entry Version [8b] |
Response status
0x00 status if replaced/removed
0x01 status if replace/remove did not happen because key had been modified
0x02 status if key does not exist
Response:
If ForceReturnPreviousValue has been passed, this responses will contain previous value (and corresponding value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.
Request format:
|
Header |
Response status:
0x00 status if infinispan was cleared
Returns a summary of all available statistics. For each statistic returned, a name and a value is returned both in String UTF-8 format. The supported stats are the following:
|
Name |
Explanation |
|
timeSinceStart |
Number of seconds since Hot Rod started. |
|
currentNumberOfEntries |
Number of entries currently in the Hot Rod server. |
|
totalNumberOfEntries |
Number of entries stored in Hot Rod server. |
|
stores |
Number of put operations. |
|
retrievals |
Number of get operations. |
|
hits |
Number of get hits. |
|
misses |
Number of get misses. |
|
removeHits |
Number of removal hits. |
|
removeMisses |
Number of removal misses. |
Response
|
Header |
Number of stats [vInt] |
Name1 length [vInt] |
Name1 [string] |
Value1 length [vInt] |
Value1 [String] |
Name2 length |
Name2 |
Value2 length |
Value2 |
... |
Number of stats : Number of individual stats returned
Name length : Length of named statistic
Name : String containing statistic name
Value length : Length of value field
Value : String containing statistic value.
Application level request to see if the server is available.
Response status:
0x00 - if no errors
|
Response header |
Error Message Length vInt |
Error Message string |
Response header contains error op code response and corresponding error status number as well as the following two:
Error Message Length : Length of error message
Error message : Error message. In the case of 0x84, this error field contains the latest version supported by the hot rod server. Length is defined by total body length.
A multi-get operation is a form of get operation that instead of requesting a single key, requests a set of keys. The Hot Rod protocol does not include such operation but remote Hot Rod clients could easily implement this type of operations by either parallelizing/pipelining individual get requests. Another possibility would be for remote clients to use async or non-blocking get requests. For example, if a client wants N keys, it could send send N async get requests and then wait for all the replies. Finally, multi-get is not to be confused with bulk-get operations. In bulk-gets, either all or a number of keys are retrieved, but the client does not know which keys to retrieve, whereas in multi-get, the client defines which keys to retrieve.
Coded request
|
Byte |
0 |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
|
8 |
0xA0 |
0x09 |
0x41 |
0x01 |
0x07 |
0x4D ('M') |
0x79 ('y') |
0x43 ('C') |
|
16 |
0x61 ('a') |
0x63 ('c') |
0x68 ('h') |
0x65 ('e') |
0x00 |
0x03 |
0x00 |
0x00 |
|
24 |
0x00 |
0x05 |
0x48 ('H') |
0x65 ('e') |
0x6C ('l') |
0x6C ('l') |
0x6F ('o') |
0x00 |
|
32 |
0x00 |
0x05 |
0x57 ('W') |
0x6F ('o') |
0x72 ('r') |
0x6C ('l') |
0x64 ('d') |
|
Field explanation
|
Field Name |
Value |
Field Name |
Value |
|
Magic (0) |
0xA0 |
Message Id (1) |
0x09 |
|
Version (2) |
0x41 |
Opcode (3) |
0x01 |
|
Cache name length (4) |
0x07 |
Cache name(5-11) |
'MyCache' |
|
Flag (12) |
0x00 |
Client Intelligence (13) |
0x03 |
|
Topology Id (14) |
0x00 |
Transaction Type (15) |
0x00 |
|
Transaction Id (16) |
0x00 |
Key field length (17) |
0x05 |
|
Key (18 - 22) |
'Hello' |
Lifespan (23) |
0x00 |
|
Max idle (24) |
0x00 |
Value field length (25) |
0x05 |
|
Value (26-30) |
'World' |
Coded response
|
Byte |
0 |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
|
8 |
0xA1 |
0x09 |
0x01 |
0x00 |
0x00 |
|
|
|
Field Explanation
|
Field Name |
Value |
Field Name |
Value |
|
Magic (0) |
0xA1 |
Message Id (1) |
0x09 |
|
Opcode (2) |
0x01 |
Status (3) |
0x00 |
|
Topology change marker (4) |
0x00 |
|