Affinity

An "Affinity Group" is a tag (just a unique name) created by the administrator. It can then be assigned to any number of RHQ servers and agents. For example, in an environment with several data centers it may be useful to assign each data center an affinity group (e.g. "DC1", "DC2"). Agents and servers co-located in a single data center can then set their affinity groups appropriately. The partition algorithm will then show preference to assigning agents to servers withing the same affinity group. Although, based on load and availability affinity is not guaranteed.

• A server or agent can be assigned to only one affinity group.

• By default a server or agent is assigned no affinity group.

• A server can be assigned an affinity group at install time or via the HA Administration Console.

• An agent can be assigned an affinity group via the HA Administration Console.

Round Robin

The goal is to partition agents across servers while taking into consideration aspects such as affinity. Additionally, the partition algorithm will take into consideration failover topology. Agents will be assigned server lists (see below) in a weighted round robin fashion distribution.

For example, in a 3-Server cloud with no affinity then agents would be assigned server lists similar to:
A1 : S1, S2, S3
A2 : S2, S3, S1
A3 : S3, S1, S2
A4 : S1, S3, S2

If S1, S2, A1, A2, A3 all had affinity then the assigned server lists may look similar to:
A1 : S1, S2, S3
A2 : S2, S1, S3
A3 : S1, S2, S3
A4 : S3, S2, S1

Compute Power

( planned, 1.2.0)

Servers may vary in the load they can carry. If servers in a cloud vary significantly in computing power the administrator can alter the assigned compute power to distribute load appropriately. Compute power is relative, by default all servers in the cloud are assigned a compute power of 1. So, for example, in a 2-Server cloud where S1 is twice as powerful as S2 and you want S1 to assume twice the agent load, set ComputePower(S1)=2 and leave ComputePower(S2)=1. Compute powers are set as positive integers.

Algorithm

A few notes on the partition algorithm.

Affinity is strong and will always be satisfied if servers with the correct affinity are available. As shown in the example above, affinity servers will always be the first entries in the server list for an agent with defined affinity. Load will be distributed amongst affinity servers while satisfying affinity. But, as also shown in the example, if no affinity servers are available the agent will fail over to an available non-affinity server.

So, although it is possible, and potentially desirable, to have load imbalance across all servers, due to affinity, agents with affinity should have their load fairly well distributed amongst their affinity servers.

The repartition algorithm attempts to limit connection churn, it will attempt to maintain as many primary server assignments as possible while still balancing load.

All known servers and agents participate in the repartition. DOWN or MAINTENANCE_MODE servers are expected to be up and operating in NORMAL fashion and as such are included on the assumption that their unavailability will be remedied. Long term downtime should be handled by deleting the server and potentially reinstalling it at a later time.

Registration and Connection

Agents go through a registration/connection phase when initially contacting the configured server. A successful registration will return the most recent server list to the agent. The agent will then set the configured server to the primary server and attempt to connect. If connection fails it will set the configured server to the next server in the list and try again, failing over until it succeeds or must start again from the primary. Therefore, the registration server and the connected server may not be the same server.

Agent startup logic:

If the agent:

1. has no assigned token

Then

1. removes its server list (in the unlikely case that it has one persisted)

2. attempts to register with the configured server

3. attempts to connect to servers in the returned server list, starting with the primary

• if the agent name is not known to the server a new token is generated, otherwise the existing token is supplied to the agent.

• if the configured server can not be contacted the agent will not be able to connect to an RHQ server and must be reconfigured or wait for the configured server to come online.

If the agent:

1. has a token

2. has no server list

Then

1. attempts to register with the configured server and the existing token

2. attempts to connect to servers in the returned server list, starting with the primary

• if the configured server can not be contacted the agent will not be able to connect to an RHQ server and must be reconfigured or wait for the configured server to come online.

If the agent:

1. has a token

2. has a server list

Then

1. attempts to register with the configured server and the existing token

2. when registered, attempts to connect to servers in the returned server list, starting with the primary

• the configured server will be the last connected server from the server list, unless the configuration has been updated by a user

• if the configured server can not be contacted the agent will use the server list, starting with the primary, to find a server with which it can register.

• When a server list is exhausted it will be reprocessed, from the head of the list, after some (configurable) delay.

Cloud Repartition

RHQ HA will, in certain circumstances, repartition the agents amongst the cloud servers. This will result in new server lists being generated for all known agents. It is important to note that the repartition algorithm will seek to limit connection churn, and as such will re-assign the minimal number of agents to new primary servers to accomplish the re-balancing.

A repartition does not push new server lists to connected agents. This prevents large scale fail-over in large environments, potentially spiking a server with connection processing. Instead, agents will intermittently check for updated server lists, and reconnect to new primary assignments, if necessary. This disperses the connection load.

Redistribution can occur for the following reasons:

1. a new server is added (installed) to the cloud.

2. an existing server is deleted from the cloud (must already be down or in maintenance mode).

3. an admin request via HA Administration Console

Agent Behavior

New agents will be assigned a server list such that load balancing and affinity are satisfied in the same ways as if the agent had been registered during the last cloud repartition.

Connected agents check for updated server lists at (configurable) scheduled intervals. At that time if an agent is not connected to its primary server it will attempt to connect to the primary. In this fashion all agents seek to run on their assigned primary server. All agents being connected to their primary servers guarantees the best load balancing and affinity satisfaction.

Server Maintenance Mode

An HA Server can be taken out of the cloud for maintenance reasons without actually being shut down. This is done via the HA Administration Console and effectively shuts down all agent communication with the server, although the server remains up and the RHQ GUI remains usable. Agents will treat this as a downed server and will apply reconnect and failover logic as needed.

A server taken down in maintenance mode comes up in maintenance mode.

GUI

Note that it doesn't matter which server you connect to in the Server Cloud to use the RHQ GUI; the viewable resources and available options will be identical regardless of which you choose.

Commands

The RHQ Agent will have new commands introduced with HA:

1. Server List View
   View the current server list for the agent. This is the failover list the agent will consult when it needs to switch to a new server.

   • failover --list

2. Server List Regenerate
   Request a new server list be generated for the agent if for some reason the current list is stale. This will also ensure the agent is talking to its primary server (if it is not, the agent will attempt to switch to it).

   • failover --check

3. Server Switch
   Request the agent (temporarily) switch to another server immediately. This is temporarly because the primary server switchover thread is still running and when it wakes up, it'll switch the agent back to the primary. The only way this prompt command will be switch over permanently is if you configure that switchover thread to not run (by setting its period to 0).

   • {{failover --switch=<server>)

Note that the sender status command will now tell you what the configured server currently is (so you can know what server the agent is, or is attempting to, talk to.

Upgrade

The next version of RHQ plans to have automated agent updates. The RHQ agent will need to run the same version as the RHQ server (this is the "Prime Directive"), and will need to be re-installed with the new version. Ease of installation and agent backward compatibility are high priority goals for future versions.

Relative Server Power

If the server cloud is made up of servers with unequal compute power it makes sense to assign more agent load to the servers with more compute power.

Agent Load

RHQ agents can vary significantly in the load they put on a server based on number of inventoried resources, measurement collection (schedule) frequency, and other factors. HA will base agent assignment not on number of agents but on relative agent load.

Redistribution

Redistribution can occur for the following reasons:

1. in response to unbalanced agent load.

RHQ will periodically review the server-agent topology and decide whether redistribution is necessary. If so, RHQ will re-balance agent load across available servers.