Deprecation

This document is DEPRECATED.

Please consider any information here as out of date. DO NOT use this document.

Instead, refer to http://infinispan.org/documentation.

Please update your bookmarks accordingly.

Skip to end of metadata
Go to start of metadata

Introduction

This article provides detailed information about the first version of the custom TCP client/server Hot Rod protocol.

Infinispan versions
This version of the protocol is implemented since Infinispan 4.1.0.Final
All key and values are sent and stored as byte arrays. Hot Rod makes no assumptions about their types. Some clarifications about the other types:
  • vInt: Refers to unsigned variable length integer values as specified in here. They're between 1 and 5 bytes long.
  • vLong: Refers to unsigned variable length long values similar to vInt but applied to longer values. They're between 1 and 9 bytes long.
  • String: Strings are always represented using UTF-8 encoding.

Request Header

The header for a request is composed of:

Magic [1b] Message Id [vLong] Version [1b] Opcode [1b] Cache Name Length [vInt] Cache Name [string] Flags [vInt] Client Intelligence [1b] Topology Id [vInt] Transaction Type [1b] Transaction Id [byte-array]
  • Magic : Possible values are:
    • 0xA0 - Infinispan Cache Request Marker
    • 0xA1 - Infinispan Cache Response Marker
  • Message Id : Id of the message that will be copied back in the response. This allows for hot rod clients to implement the protocol in an asynchronous way.
  • Version : Infinispan hot rod server version. In this particular case, this is 10
  • Opcode : Possible values are only the ones on the request column:
    Request operation codes Response operation codes
    0x01 - put request 0x02 - put response
    0x03 - get request 0x04 - get response
    0x05 - putIfAbsent request 0x06 - putIfAbsent response
    0x07 - replace request 0x08 - replace response
    0x09 - replaceIfUnmodified request 0x0A - replaceIfUnmodified response
    0x0B - remove request 0x0C - remove response
    0x0D - removeIfUnmodified request 0x0E - removeIfUnmodified response
    0x0F - containsKey request 0x10 - containsKey response
    0x11 - getWithVersion request 0x12 - getWithVersion response
    0x13 - clear request 0x14 - clear response
    0x15 - stats request 0x16 - stats response
    0x17 - ping request 0x18 - ping response
    0x19 - bulkGet request 0x1A - bulkGet response
    - 0x50 - error response
  • Cache Name Length : Length of cache name. If the passed length is 0 (followed by no cache name), the operation will interact with the default cache.
  • Cache Name : Name of cache on which to operate. This name must match the name of predefined cache in the Infinispan configuration file.
  • Flags : A variable length number representing flags passed to the system. Each flags is represented by a bit. Note that since this field is sent as variable length, the most significant bit in a byte is used to determine whether more bytes need to be read, hence this bit does not represent any flag. Using this model allows for flags to be combined in a short space. Here are the current values for each flag:
    0x0001 ForceReturnPreviousValue
  • Client Intelligence : This byte hints the server on the client capabilities:
    • 0x01 - basic client, interested in neither cluster nor hash information
    • 0x02 - topology-aware client, interested in cluster information
    • 0x03 - hash-distribution-aware client, that is interested in both cluster and hash information
  • Topology Id : This field represents the last known view in the client. Basic clients will only send 0 in this field. When topology-aware or hash-distribution-aware clients will send 0 until they have received a reply from the server with the current view id. Afterwards, they should send that view id until they receive a new view id in a response
  • Transaction Type : This is a 1 byte field, containing one of the following well-known supported transaction types (For this version of the protocol, the only supported transaction type is 0):
    • 0 - Non-transactional call, or client does not support transactions. The subsequent TX_ID field will be omitted.
    • 1 - X/Open XA transaction ID (XID). This is a well-known, fixed-size format.
  • Transaction Id : The byte array uniquely identifying the transaction associated to this call. It's length is determined by the transaction type. If transaction type is 0, no transaction id will be present.

Response Header

Magic [1b] Message Id [vLong] Op code [1b] Status [1b] Topology Change Marker [1b]
  • Opcode : Op code representing a response to a particular operation, or error condition.
  • Status : Status of the response, possible values:
    0x00 - No error 0x01 - Not put/removed/replaced 0x02 - Key does not exist
    0x81 - Invalid magic or message id 0x82 - Unknown command 0x83 - Unknown version
    0x84 - Request parsing error 0x85 - Server Error 0x86 - Command timed out

    Exceptional error status responses, those that start with 0x8..., are followed by the length of the error message (as a vInt) and error message itself as String.

  • Topology Change Marker : This is a marker byte that indicates whether the response is prepended with topology change information. When no topology change follows, the content of this byte is 0. If a topology change follows, its contents are 1.

Topology Change Headers

The following section discusses how the response headers look for topology-aware or hash-distribution-aware clients when there's been a cluster or view formation change. Note that it's the server that makes the decision on whether it sends back the new topology based on the current topology id and the one the client sent. If they're different, it will send back the new topology.

Topology-Aware Client Topology Change Header

This is what topology-aware clients receive as response header when a topology change is sent back:

Response header with topology change marker Topology Id [vInt] Num servers in topology [vInt]
m1: Host/IP length [vInt] m1: Host/IP address [string] m1: Port [2b - Unsigned Short]
m2: Host/IP length [vInt] m2: Host/IP address [string] m2: Port [2b - Unsigned Short]
...etc
  • Num servers in topology : Number of Infinispan Hot Rod servers running within the cluster. This could be a subset of the entire cluster if only a fraction of those nodes are running Hot Rod servers.
  • Host/IP address length : Length of hostname or IP address of individual cluster member that Hot Rod client can use to access it. Using variable length here allows for covering for hostnames, IPv4 and IPv6 addresses.
  • Host/IP address : String containing hostname or IP address of individual cluster member that Hot Rod client can use to access it.
  • Port : Port that Hot Rod clients can use to communicat with this cluster member.

Hash-Distribution-Aware Client Topology Change Header

This is what hash-distribution-aware clients receive as response header when a topology change is sent back:

Response header with topology change marker Topology Id [vInt] Num Key Owners [2b - Unsigned Short] Hash Function Version [1b] Hash space size [vInt] Num servers in topology [vInt]
m1: Host/IP length [vInt] m1: Host/IP address [string] m1: Port [2b - unsigned short] m1: Hashcode [4b]
m2: Host/IP length [vInt] m2: Host/IP address [string] m2: Port [2b - unsigned short] m2: Hashcode [4b]
...etc

It's important to note that since hash headers rely on the consistent hash algorithm used by the server and this is a factor of the cache interacted with, hash-distribution-aware headers can only be returned to operations that target a particular cache. Currently ping command does not target any cache (this is to change as per ISPN-424, hence calls to ping command with hash-topology-aware client settings will return a hash-distribution-aware header with "Num Key Owners", "Hash Function Version", "Hash space size" and each individual host's hash code all set to 0. This type of header will also be returned as response to operations with hash-topology-aware client settings that are targeting caches that are not configured with distribution.

  • Number key owners : Globally configured number of copies for each Infinispan distributed key
  • Hash function version : Hash function version, pointing to a specific hash function in use. See Hot Rod hash functions for details.
  • Hash space size : Modulus used by Infinispan for for all module arithmetic related to hash code generation. Clients will likely require this information in order to apply the correct hash calculation to the keys.
  • Num servers in topology : If virtual nodes are disabled, this number represents the number of Hot Rod servers in the cluster. If virtual nodes are enabled, this number represents all the virtual nodes in the cluster which are calculated as (num configured virtual nodes) * (num cluster members). Regardless of whether virtual nodes are configured or not, the number represented by this field indicates the number of 'host:port:hashId' tuples to be read in the response.
  • Hashcode : 32 bit integer representing the hashcode of a cluster member that a Hot Rod client can use indentify in which cluster member a key is located having applied the CSA to it.

Operations

Get/Remove/ContainsKey/GetWithVersion

  • Common request format:
    Header Key Length [vInt] Key [byte-array]
    • Key Length : Length of key. Note that the size of a vint can be up to 5 bytes which in theory can produce bigger numbers than Integer.MAX_VALUE. However, Java cannot create a single array that's bigger than Integer.MAX_VALUE, hence the protocol is limiting vint array lengths to Integer.MAX_VALUE.
    • Key : Byte array containing the key whose value is being requested.
  • Response status:
    • 0x00 - success, if key present/retrieved/removed
    • 0x02 - if key does not exist
  • Get response:
    Header Value Length [vInt] Value [byte-array]
    • Value Length : Length of value
    • Value : The requested value. If key does not exist, status returned in 0x02. See encoding section for more info.
  • Remove response:
    If ForceReturnPreviousValue has been passed, remove response will contain previous value (including value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.
  • ContainsKey response:
    Empty
  • GetWithVersion response:
    Header Entry Version [8b] Value Length [vInt] Value [byte-array]
    • Entry Version : Unique value of an existing entry's modification. The protocol does not mandate that entry_version values are sequential. They just need to be unique per update at the key level.

BulkGet

  • Request format:
    Header Entry Count [vInt]
    • Entry Count : Maximum number of Infinispan entries to be returned by the server (entry == key + associated value). Needed to support CacheLoader.load(int). If 0 then all entries are returned (needed for CacheLoader.loadAll()).
  • Response:
    Header More [1b] Key Size 1 Key 1 Value Size 1 Value 1 More [1b] Key Size 2 Key 2 Value Size 2 Value 2 More [1b] ...
    • More : One byte representing whether more entries need to be read from the stream. So, when it's set to 1, it means that an entry followes, whereas when it's set to 0, it's the end of stream and no more entries are left to read.
      For more information on BulkGet look here

Put/PutIfAbsent/Replace

  • Common request format:
    Header Key Length [vInt] Key [byte-array] Lifespan [vInt] Max Idle [vInt] Value Length [vInt] Value [byte-array]
    • Lifespan : Number of seconds that a entry during which the entry is allowed to life. If number of seconds is bigger than 30 days, this number of seconds is treated as UNIX time and so, represents the number of seconds since 1/1/1970. If set to 0, lifespan is unlimited.
    • Max Idle : Number of seconds that a entry can be idle before it's evicted from the cache. If 0, no max idle time.
  • Put response status:
    • 0x00 if stored
  • Replace response status:
    • 0x00 if stored
    • 0x01 if store did not happen because key does not exist
  • PutIfAbsent response status:
    • 0x00 if stored
    • 0x01 if store did not happen because key was present
  • Put/PutIfAbsent/Replace response:
    If ForceReturnPreviousValue has been passed, these responses will contain previous value (and corresponding value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.

ReplaceIfUnmodified

  • Request format:
    Header Key Length [vInt] Key [byte-array] Lifespan [vInt] Max Idle [vInt] Entry Version [8b] Value Length [vInt] Value [byte-array]
    • Entry Version : Use the value returned by GetWithVersion operation.
  • Response status
    • 0x00 status if replaced/removed
    • 0x01 status if replace/remove did not happen because key had been modified
    • 0x02 status if key does not exist
  • Response:
    If ForceReturnPreviousValue has been passed, this responses will contain previous value (and corresponding value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.

RemoveIfUnmodified

  • Request format:
    Header Key Length [vInt] Key [byte-array] Entry Version [8b]
  • Response status
    • 0x00 status if replaced/removed
    • 0x01 status if replace/remove did not happen because key had been modified
    • 0x02 status if key does not exist
  • Response:
    If ForceReturnPreviousValue has been passed, this responses will contain previous value (and corresponding value length) for that key. If the key does not exist or previous was null, value length would be 0. Otherwise, if no ForceReturnPreviousValue was sent, the response would be empty.

Clear

  • Request format:
    Header
  • Response status:
  • 0x00 status if infinispan was cleared

Stats

Returns a summary of all available statistics. For each statistic returned, a name and a value is returned both in String UTF-8 format. The supported stats are the following:

Name Explanation
timeSinceStart Number of seconds since Hot Rod started.
currentNumberOfEntries Number of entries currently in the Hot Rod server.
totalNumberOfEntries Number of entries stored in Hot Rod server.
stores Number of put operations.
retrievals Number of get operations.
hits Number of get hits.
misses Number of get misses.
removeHits Number of removal hits.
removeMisses Number of removal misses.
  • Response
    Header Number of stats [vInt] Name1 length [vInt] Name1 [string] Value1 length [vInt] Value1 [String] Name2 length Name2 Value2 length Value2 ...
    • Number of stats : Number of individual stats returned
    • Name length : Length of named statistic
    • Name : String containing statistic name
    • Value length : Length of value field
    • Value : String containing statistic value.

Ping

Application level request to see if the server is available.

  • Response status:
    • 0x00 - if no errors

Error Handling

Response header Error Message Length vInt Error Message string

Response header contains error op code response and corresponding error status number as well as the following two:

  • Error Message Length : Length of error message
  • Error message : Error message. In the case of 0x84, this error field contains the latest version supported by the hot rod server. Length is defined by total body length.

Multi-Get Operations

A multi-get operation is a form of get operation that instead of requesting a single key, requests a set of keys. The Hot Rod protocol does not include such operation but remote Hot Rod clients could easily implement this type of operations by either parallelizing/pipelining individual get requests. Another possibility would be for remote clients to use async or non-blocking get requests. For example, if a client wants N keys, it could send send N async get requests and then wait for all the replies. Finally, multi-get is not to be confused with bulk-get operations. In bulk-gets, either all or a number of keys are retrieved, but the client does not know which keys to retrieve, whereas in multi-get, the client defines which keys to retrieve.

Example - Put request

  • Coded request
    Byte 0 1 2 3 4 5 6 7
    8 0xA0 0x09 0x41 0x01 0x07 0x4D ('M') 0x79 ('y') 0x43 ('C')
    16 0x61 ('a') 0x63 ('c') 0x68 ('h') 0x65 ('e') 0x00 0x03 0x00 0x00
    24 0x00 0x05 0x48 ('H') 0x65 ('e') 0x6C ('l') 0x6C ('l') 0x6F ('o') 0x00
    32 0x00 0x05 0x57 ('W') 0x6F ('o') 0x72 ('r') 0x6C ('l') 0x64 ('d')  
  • Field explanation
    Field Name Value Field Name Value
    Magic (0) 0xA0 Message Id (1) 0x09
    Version (2) 0x41 Opcode (3) 0x01
    Cache name length (4) 0x07 Cache name(5-11) 'MyCache'
    Flag (12) 0x00 Client Intelligence (13) 0x03
    Topology Id (14) 0x00 Transaction Type (15) 0x00
    Transaction Id (16) 0x00 Key field length (17) 0x05
    Key (18 - 22) 'Hello' Lifespan (23) 0x00
    Max idle (24) 0x00 Value field length (25) 0x05
    Value (26-30) 'World'
  • Coded response
    Byte 0 1 2 3 4 5 6 7
    8 0xA1 0x09 0x01 0x00 0x00      
  • Field Explanation
    Field Name Value Field Name Value
    Magic (0) 0xA1 Message Id (1) 0x09
    Opcode (2) 0x01 Status (3) 0x00
    Topology change marker (4) 0x00  
Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.