Chapter 9. Network Connection Monitoring

Remoting has two mechanisms for monitoring the health of estabilished connections, which inform listeners on the client and server sides when a possible connection failure has been detected.

9.1. Client side monitoring

On the client side, an org.jboss.remoting.ConnectionValidator periodically sends a PING message to the server and reports a failure if the response does not arrive within a specified timeout period. The PING is sent on one thread, and another thread determines if the response arrives in time. Separating these two activities allows Remoting to detect a failure regardless of the cause of the failure.

The creation of the ConnectionValidator is the responsibility of the org.jboss.remoting.Client class. All the application code needs to do is to register an implementation of the org.jboss.remoting.ConnectionListener interface, which has only one method:

public void handleConnectionException(Throwable throwable, Client client);

What actions the ConnectionListener chooses to take are up to the application, but disconnecting the Client might be a reasonable strategy.

The Client class has three methods for registering a ConnectionListener:

       public void addConnectionListener(ConnectionListener listener);
       public void addConnectionListener(ConnectionListener listener, int pingPeriod);
       public void addConnectionListener(ConnectionListener listener, Map metadata);
    

The second method supports configuring the frequency of PING messages, and the third method supports more general configuration of the ConnectionValidator. Note that a given Client maintains a single ConnectionValidator, so the parameters in the metadata map are applied only on the first call to Client.addConnectionListener(). The following parameters are supported by ConnectionValidator, which is where the parameter names are defined:

VALIDATOR_PING_PERIOD (actual value "validatorPingPeriod") - specifies the time, in milliseconds, that elapses between the sending of PING messages to the server. The default value is 2000.

VALIDATOR_PING_TIMEOUT (actual value "validatorPingTimeout") - specifies the time, in milliseconds, allowed for arrival of a response to a PING message. The default value is 1000.

FAILURE_DISCONNECT_TIMEOUT (actual value "failureDisconnectTimeout") - if the parameter "stopLeaseOnFailure" (see Interactions between client side and server side connection monitoring) is set to "true", then "failureDisconnectTimeout" determines the disconnect timeout value to be used by org.jboss.remoting.LeasePinger in shutting down. In particular, if "failureDisconnectTimeout" is set to "0", then LeasePinger will avoid any network i/o.

Normally, the values for these parameters are obtained either from the Client's configuration map or the metadata map passed to addConnectionListener(), with values in the metadata map taking precedence. However, another relevant parameter is defined in org.jboss.remoting.Client:

USE_ALL_PARAMS (actual value "useAllParams") - this parameter is searched for in the InvokerLocator, in the configuration map passed to the Client, and in the metadata map (in that order). If the last occurrence found is set to "true", then parameter values are first obtained from the InvokerLocator, followed by the Client's configuration map and the metadata map.

Note that ConnectionValidator creates a client invoker to sends the PING messages, and it passes the metadata map to configure the client invoker.

NOTE. The default values of VALIDATOR_PING_PERIOD and VALIDATOR_PING_TIMEOUT have often been found in practice to be too small, increasing the likelihood of spurious connection failures.

NOTE. It is important to set VALIDATOR_PING_PERIOD to a value greater than the value of VALIDATOR_PING_TIMEOUT. Doing so gives the ConnectionValidator a chance to notify all ConnectionListeners, which might result in shutting down the connection, before the next PING is sent.

For more configuration parameters, see Interactions between client side and server side connection monitoring.

9.2. Server side monitoring

A remoting server also has the capability to detect when a client is no longer available. This is done by estabilishing a lease with the remoting clients that connect to a server. On the client side, an org.jboss.remoting.LeasePinger periodically sends PING messages to the server, and on the server side an org.jboss.remoting.Lease informs registered listeners if the PING doesn't arrive withing the specified timeout period.

Server side activation. To turn on server side connection failure detection of remoting clients, it is necessary to satisfy two criteria. The first is that the client lease period is set and is a value greater than 0. The value is represented in milliseconds. The client lease period can be set by either the 'clientLeasePeriod' attribute within the Connector configuration or by calling the Connector method

public void setLeasePeriod(long leasePeriodValue);

The second criterion is that an implementation of the org.jboss.remoting.ConnectionListener interface is added as a connection listener to the Connector, either via the method

public void addConnectionListener(ConnectionListener listener)

or through the use of the ServerInvoker.CONNECTION_LISTENER parameter (actual value "connectionListener") in the Connector's configuration map or XML configuration file. Once both criteria are met, the remoting server will turn on client leasing.

The ConnectionListener will be notified of both client failures and client disconnects via the handleConnectionException() method. If the client failed, meaning its lease was not renewed within configured time period, the first parameter to the handleConnectionException() method will be null. If the client disconnected in a regular manner, the first parameter to the handleConnectionException() method will be of type ClientDisconnectedException (which indicates a normal termination). Note, the client's lease will be renewed on the server with any and every invocation made on the server from the client, whether it be a normal invocation or a ping from the client internally.

The actual lease window established on the server side is dynamic based the rate at which the client updates its lease. In particular, the lease window will always be set to lease period * 2 for any lease that does not have a lease update duration that is longer than 75% of the lease window (meaning if set lease period to 10 seconds and always update that lease in less then 7.5 seconds, the lease period will always remain 10 seconds). If the update duration is greater than 75% of the lease window, the lease window will be reset to the lease duration X 2 (meaning if set lease period to 10 seconds and update that lease in 8 seconds, the new lease window will be set to 16 seconds). Also, the lease will not immediately expire on the first lease timeout (meaning did not get an update within the lease window). It takes two consecutive timeouts before a lease will expire and a notification for client connection failure is fired. This essentially means that the time it will take before a connection listener is notified of a client connection failure will be at least 4 X lease period (no exceptions).

Client side activation. By default, the client is not configured to do client leasing. To allow a client to do leasing, either set the parameter "leasing" to "true" in the InvokerLocator or set the parameter Client.ENABLE_LEASE (actual value "enableLease") to true in the InvokerLocator or in the Client configuration map. [The use of Client.ENABLE_LEASE is recommended.] This does not mean that client will lease for sure, but will indicate the client should call on the server to see if the server has activated leasing and get the leasing period suggested by the server. It is possible to override the suggested lease period by setting the parameter org.jboss.remoting.InvokerLocator.CLIENT_LEASE_PERIOD (actual value "lease_period") to a value greater than 0 and less than the value suggested by the server. Note. If the client and server are local, meaning running within the JVM, leasing (and thus connection notification) will not be activated, even if is configured to do so.

If leasing is turned on within the client side, there is no API or configuration changes needed, unless want to override as mentioned previously. When the client initially connects to the server, it will check to see if client leasing is turned on by the server. If it is, it will internally start pinging periodically to the server to maintain the lease. When the client disconnects, it will internally send message to the server to stop monitoring lease for this client. Therefore, it is IMPORTANT that disconnect is called on the client when done using it. Otherwise, the client will continue to make its ping call on the server to keep its lease current.

The client can also provide extra metadata that will be communicated to the connection listener in case of failure by supplying a metadata Map to the Client constructor. This map will be included in the Client instance passed to the connection listener (via the handleConnectionException() method) via the Client's getConfiguration() method.

From the server side, there are two ways in which to disable leasing (i.e. turn leasing off). The first is to call:

public void removeConnectionListener(ConnectionListener listener)

and remove all the registered ConnectionListeners. Once the last one has been removed, leasing will be disabled and all the current leasing sessions will be terminated. The other way is to call:

public void setLeasePeriod(long leasePeriodValue)

and pass a value less than zero. This will disable leasing, preventing any new leases to be established but will allow current leasing sessions to continue.

The following parameter is relevant to leasing configuration on the server side:

org.jboss.remoting.ServerInvoker.CLIENT_LEASE_PERIOD (actual value "clientLeasePeriod") - specifies the timeout period used by the server to determine if a PING is late. The default value is "5000", which indicates that leasing will be activated if an org.jboss.remoting.ConnectionListener is registered with the server. This is also the suggested lease period returned by the server when the client inquires if leasing is activated.

The following parameters are relevant to leasing configuration on the client side:

org.jboss.remoting.Client.ENABLE_LEASE (actual value "enableLease") - if set to "true", will lead org.jboss.remoting.Client to attempt to set up a lease with the server, if leasing is activated on the server.

org.jboss.remoting.InvokerLocator.CLIENT_LEASE (actual value "leasing") - if set to "true" in the InvokerLocator, will lead org.jboss.remoting.Client to attempt to set up a lease with the server, if leasing is activated on the server. It is suggested that this parameter be avoided, in favor of Client.ENABLE_LEASE.

org.jboss.remoting.InvokerLocator.CLIENT_LEASE_PERIOD (actual value "lease_period") - if set to a value greater than 0 and less than the suggested lease period returned by the server, will be used to determine the time between PING messages sent by LeasePinger.

For examples of how to use server side connection listeners, reference org.jboss.test.remoting.lease.LeaseTestServer and org.jboss.test.remoting.lease.LeaseTestClient.

9.3. Interactions between client side and server side connection monitoring

As of Remoting release 2.2.2.SP7, the client side and server side connection monitoring mechanisms can be, and by default are, more closely related, in two ways.

  1. If the parameter org.jboss.remoting.ConnectionValidator.TIE_TO_LEASE (actual value "tieToLease") is set to true, then, when the server receives a PING message from an org.jboss.remoting.ConnectionValidator, it will return a boolean value that indicates whether a lease currently exists for the connection being monitored. If leasing is activated on the client and server side, then a value of "false" indicates that the lease has failed, and the ConnectionValidator will treat a returned value of "false" the same as a timeout; that is, it will notifiy listeners of a connection failure. The default value of this parameter is "true". Note. If leasing is not activated on the client side, then this parameter has no effect.
  2. If the parameter org.jboss.remoting.ConnectionValidator.STOP_LEASE_ON_FAILURE (actual value "stopLeaseOnFailure") is set to true, then, upon detecting a connection failure, ConnectionValidator will stop the LeasePinger, if any, pinging a lease on the same connection. The default value is "true".

Note. As of release 2.2.3, an important concept related to connection monitoring, connection identity, is available. Suppose that leasing is enabled and that a client invoker stops and is replaced by a new client invoker. If the replacement occurs quickly, the server side Lease may never miss a ping, in which there is no evidence that anything changed on the client side. That is, the connection is still alive, as far as the server is concerned. That semantics might be perfectly acceptable for some applications, but other applications might interpret the same events as a connection failure followed by a new connection. Remoting can be configured to treat a connection as being defined by a client/server pair, which supports the second category of applications.

More specifically, when configured to do so by setting the parameter org.jboss.remoting.Remoting.USE_CLIENT_CONNECTION_IDENTITY (actual value "useClientConnectionIdentity") to "true", Remoting identifies a connection with a LeasePinger/Lease pair. A Client participates in a connection when it is connected by way of the new method

public void connect(ConnectionListener listener, Map metadata) throws Exception;

This method serves to connect the Client to the server by way of a new or existing client invoker, and it also (1) registers the ConnectionListener with the Client's new or exiting ConnectionValidator and (2) registers the ConnectionValidator with the client invoker's LeasePinger. Subsequently, if any ConnectionValidator registered with that LeasePinger detects a connection failure, it will (if "stopLeaseOnFailure" is "true") stop the LeasePinger, and the LeasePinger will cause each registered ConnectionValidators to notify each of its registered ConnectionListeners of the connection failure. Once the LeasePinger has been shut down and all of the notifications have been made, the connection anchored by the LeasePinger is defunct, and the associated Client's should be disconnected by a call to Client.disconnect(). If such a Client is reconnected by a call to Client.connect(), it will be associated with a new LeasePinger and, therefore, a new connection.

Relevant parameters:

TIE_TO_LEASE (actual value "tieToLease") - specifies whether ConnectionValidator should treat the failure of a related lease on the server side as a connection failure. The default value is "true".

STOP_LEASE_ON_FAILURE (actual value "stopLeaseOnFailure") - specifies whether, when a ConnectionValidator detects a connection failure, it should stop the associated org.jboss.remoting.LeasePinger, if any. The default value is "true".

org.jboss.remoting.Remoting.USE_CLIENT_CONNECTION_IDENTITY (actual value "useClientConnectionIdentity") - tells Remoting to adhere to the new "connection identity" semantics.