Definition

"Agent Auto-Update" refers to the ability for a running agent to automatically update itself, including updates to jars, configuration files, scripts and preference values. An agent should be able to update and restart itself automatically without the need for any additional manual intervention, specifically, there should be no need for an administrator to log into the agent machine and perform tasks necessary to complete the update.

Requirements

The following are requirements that must be supported by our agent auto-update feature:

1. The update solution must be cross-platform - Linux, Windows, UNIX and hopefully even MacOS.

2. Agents must be able to update themselves while they are running. This seems obvious but is being explicitly mentioned for two reasons:

   • We must make sure the agent can overwrite jar files that are currently open by the agent JVM's classloaders (on Windows XP, and perhaps other platforms, you cannot delete a file that is currently opened; altering jar file content could screw up a classloader that is currently reading class definitions from it)

   • Agents not already running must be able to know immediately at startup they they need to update (and do so then)

3. Manual intervention must not be required to apply an update. The agent should be able to automatically download the update, apply the update and restart itself.

4. Be able to turn off the auto-update feature, from both the server and agent side. I'll need to be able to turn this off on the agent side because the embedded agent will have this feature turned off by default, in addition, this needs to be turned off to be able to run our unit tests. Be able to turn it off at the server side to force the disablement of agent auto-update even if the agent wants to.

The Prime Directive

There is one important design decision we are making in our first version of agent auto-update - we call it the Prime Directive and it is this:

Keeping the above fact in mind, this means:

1. We will not support updating agents such that there ends up being different agent versions running in the RHQ environment.

2. We will not support servers installed in a cloud that are matched to different agent versions. All servers in a cloud must support the same version of agent.

In the future, we may decide to allow violations of this Prime Directive - we might be able to support different "minor" versions of agents; however, in our first implementation, we will not support it.

In fact, we can somehow explicitly disallow a mis-versioned agent from talking to a server. The easiest way we can do this without requiring additional out-of-band data to flow over the comm connection is to have the agent kill itself if it detects it is one version but the server is another version. Unfortunately, this will not prevent really old agents (those prior to this agent-update feature) from trying to talk to the server. To prevent that, we could add an out-of-band version string to be placed in the agent's outgoing commands' configuration and if the server gets a bad version, it will disallow the messages. NOTE: we need a way to not make this too difficult for agent developers. A rebuild of an agent in a development environment should somehow be able to stay at a particular version so it can be started and not be denied access because the new agent is not of the version the server thinks is the latest version.

Agent Update Binaries

Today we ship agent distributions as a .zip. You unzip them and you have an agent installation.

We are now going to distribute "agent update binaries" that will be packaged in our server distribution. The agent update binary will be a self-executing .jar that contains not only the full agent distribution .zip but also the update code itself. The name of the jar will include version information so it is easily discernable, such as "rhq-enterprise-agentupdate-2.2.0.jar".

The agent distribution zip files will still be called "rhq-enterprise-agent-2.2.0.zip".

Agent Update Binaries can be used either within the context of an existing agent performing an update, or completely standalone (in the case of doing a fresh provisioning of an agent where no agent existed before). Example:

java -jar rhq-enterprise-agent-2.2.0.jar --install[=<new-agent-dir>]

This will tell the Main-Class (as defined in the jar's manifest) to extract the agent in the current directory without doing anything special - do not run any update-specific code. It is as if the user simply extracted the .zip distro from the jar, and unzip'ed the agent .zip distro. If you specify a <new-agent-dir>, the agent will be installed in that directory (because it simply unzips the enclosing agent.zip, the agent will really be installed in "<new-agent-dir>/rhq-agent" since the zip is rooted at "rhq-agent").

java -jar rhq-enterprise-agent-2.2.0.jar --update[=<old-agent-dir>]

This will be the command that an existing agent will issue after it downloads the jar file. This tells the Main-Class that we want to update an existing agent, where that existing agent is installed in the given <old-agent-dir> directory. The default will be the current directory (to allow a user to copy this jar file in an agent home directory and just run "java -jar rhq-enterprise-agent-2.2.0.jar --update" and have it work).

/rhq-agent-update-version.properties (information about this agent)
/org/rhq/enterprise/agent/update/... (all the RHQ update code here)
/abc-corp/... (third party libraries needed by the update code - gnu.getopt, etc...)
/rhq-enterprise-agent-#.#.#.zip (the actual agent distro .zip file)
/README.txt (some helpful information; will be displayed with --help)
/LICENSE (the RHQ license file)

rhq-agent.latest.version=1.2.0.GA
rhq-agent.latest.build-number=12345

The RHQ Server's Agent Update Servlet

Deployed in the RHQ Server's portal-war will be a standalone Agent Update Servlet. It will be accessed via simple HTTP GET - which means it is accessible using simply a browser, wget or other web client. However, its main purpose is to be accessed by the RHQ Agent - the agent will be able to ask the servlet for information about the Agent Update Binary as well as to download it (see the agent's "update" prompt command).

The servlet will be mapped to more than one URI to allow for it to support different types of requests. One request type would be to download an agent update binary. Another request will be to ask it what agent version the server supports (i.e. the version of the agent that can be downloaded from the servlet). Example:

<servlet>
  <servlet-name>AgentUpdateServlet</servlet-name>
  <servlet-class>org.rhq.enterprise.gui.agentupdate.AgentUpdateServlet</servlet-class>
</servlet>
<servlet-mapping>
  <servlet-name>AgentUpdateServlet</servlet-name>
  <url-pattern>/agentupdate/version</url-pattern>
</servlet-mapping>
<servlet-mapping>
  <servlet-name>AgentUpdateServlet</servlet-name>
  <url-pattern>/agentupdate/download</url-pattern>
</servlet-mapping>

Note that because we will deploy the servlet inside of our portal-war, our servlet will be able to make MBean and SLSB calls into the server in case it needs to do things like get the version of the server and to check to see if our server has been configured to disable the serving up of agent updates.

rhq.server.agent-downloads-limit=45

Performing Version Check To Determine If Update Is Needed

The agent has an "update" prompt command that is able to ask the server what version of the agent it has by doing an HTTP GET on http://<rhq-server>:7080/agentupdate/version (or the custom version URL if configured). This will return a simple name=version set of properties. We will make this generic so we can extend it in the future. For the first implementation, I suspect the only thing this will return in the response is:

rhq-server.version=1.2.0.GA
rhq-server.build-number=1852
rhq-agent.latest.md5=<the MD5 hashcode of the binary .jar>
rhq-agent.latest.version=1.2.0.GA
rhq-agent.latest.build-number=12345

rhq-server.version=1.2.0.GA
rhq-server.build-number=1852
rhq-agent.latest.md5=<the MD5 hashcode of the binary .jar>
rhq-agent.latest.version=1.2.0.GA
rhq-agent.latest.build-number=12345
rhq-server.agents-supported.versions=[2.0,2.3)
rhq-server.agents-supported.build-numbers=[,1205]

The version string that the servlet returns for "rhq-agent.latest.version" (and "rhq-agent.latest.build-number", if we think we need it) will be determined at servlet runtime by examining the agent update binary jar itself (in the "rhq-agent-update-version.properties" file). This version information will be cached in a file internal to the servlet, stored in the rhq-downloads/rhq-agent directory ("rhq-server-agent-versions.properties" file).

The agent doesn't need to do any of the above when it needs to automatically check to see if it needs to update. The agent will have its version checked automatically by the server during the agent's startup registration and thereafter during its connectAgent calls (in the case the agent fails over to another server different from the one it registered on). If the server's version check fails, it means the agent is attempting to violate The Prime Directive - this causes an AgentNotSupportedException to be thrown from the server to the agent.

Performing the Update

After an agent determines it is out of date and needs to update, the agent must stop everything it is doing and start the update processing. The agent will spawn a separate non-daemon thread called the "RHQ Agent Update Thread". This will need to first shutdown all internal agent components (e.g. the comm layer and the plugin container). After the agent is shutdown, with only the update thread running, the agent will do the following:

1) HTTP GET "server-transport://server-endpoint:server-port/agentupdate/download" (or whatever the configured download URL is)
2) Store the agent update binary somewhere on the local filesystem.
3) Execute "java" with "-jar" option passing in the agent update binary jar filename along with any hardcoded command line options (such as "--update").
4) After the process has been executed successfully, the agent returns from this Agent Update thread - but it also spawn a daemon "kill" thread that sleeps for a few minutes and if its still around, it calls "System.exit(1)" to make sure the Java process for the orignal agent dies.

At this point, the new agent update binary jar has been executed. It should sleep for a bit to wait for the original agent to die (can/should we perform some checks to confirm it is dead?). It should then perform its update tasks:

1) unpackage the rhq-agent directory found in the jar file, place it in some tmp location (<rhq-home>/update?).
2) copy the original plugins to the new agent (the new agent will update the plugins at startup)
3) copy the original _env.bat/sh files to the new agent; the new agent's env.bat/sh scripts should be renamed to "_env.bat/.sh.default
3a) try to execute "chmod +x *.sh" on non-Windows platforms - warn on failure but do not exit the update
4) copy the exiting data/failover.dat file (if it exists) into the new agent's data directory
5) purge the rest of the old agent (including its data/ directory and its lib, bin, logs, etc)
6) move the new agent to the old agent's location
7) launch the new agent's script

8) exit

If any of this fails, the agent will probably be dead in the water because presumably it needed to update because the server has been updated and its comm layer is different. However, this isn't always the case (most times it probably won't be) so the Agent Update Thread tries to start the original agent (via AgentMain.start()) to bring it back to where it was.

Distributions

We may decide to stop shipping agent-only distributions (our maven builds can still produce them, we just will not publish them). Because all servers must be paired with a particular version of agent, we may ship one distribution that includes both the server and agent, thus ensuring we keep the proper versions of servers and agents paired together. If we need to update an agent (say, we need to patch a bug in the agent), we will ship that new agent in a new server such that you will update the server which will then auto-update all agents with the new code. This will be true even if the server code itself did not change. What this does is:

• Reduce the chance of users installing incompatible versions of agents and servers

• Provides a mechanism for administrators to pull down agent distributions directly from the server (the agent distro will be a self-contained binary that will be available via anonymous HTTP GET from the server - thus, something like:

wget --content-disposition http://<rhq-server>:7080/agentupdate/download

will pull down an agent that can be provisioned manually if an agent does not yet exist.

Updating Agent Running As A Windows Service / UNIX Boot Time Process

The auto-update feature needs to determine how it will be affected if the agent is running as a Windows Service or UNIX boot time process.

Testing

Testing this stuff is a bit difficult because you have to get an agent to think its an older version than what the server is expecting. In order to artificially violate the Prime Directive, you can use a small ANT script to modify an existing agent distribution or agent update binary so that agent thinks it's a particular version. Read the comments at the top of that ANT script to learn how to use it.

ant -Dagent.home.dir=<your-agent-install-directory>

ant -Dserver.home.dir=<your-server-install-directory>

Future Enhancements

Older Notes That May Or May Not Be Relevent Now

Here are some issues that we need to solve before we can implement this feature:

• Persisted data (command-spool.dat, inventory.dat, plugin specific data) cannot be blindly kept or deleted - the update must be able to tell the agent what persisted data it can keep and what data it should purge. Note that we might be able to blindly delete inventory.dat always because the agent can sync itself fully from the server; however, if the agent spooled data while the server was in MM, we should keep and resend that spooled data if possible. But a new agent update MAY in fact invalidate the spool data (for example, if a domain object changed its serializable UID, the agent will not be able to unspool the data without getting errors; nor can the agent send the data up to the server because the server would not be able to deserialize it). Therefore, we cannot blindly keep OR delete /data files - there must be a way that we can tell the agent, for example, "you must delete your spool data but you can keep your inventory data". This is true for plugin-specific data too (can we provide a way to say, "delete the JbossAS plugin's persisted data in /data).

  • Perhaps we add a simple piece of data to the agent's .zip (update-metadata.properties) or .jar (META-INF/MANIFEST.MF) that indicates things like "purge-inventory=true", "purge-all-data=true", "purge-file=data/some-specific-file", "purge-config=true", etc.

• On Windows, you can't delete a jar file while the VM's classloaders has it open.

• You cannot update/overwrite a jar file because if a VM's classloader has it open, the class definitions will get corrupted or fail to load.

• How does an agent know its got a new update waiting? Where does the agent look to to find out what its current version is? Where does it look to ask the server what versions are available for download?

Here are some ideas to think about when deciding how to implement:

• Have the agent send a "version" string in the out-of-band command config properties. the server's command authenticator can throw a "UnsupportedVersionException" if the version the agent is is an old one that is no longer supported. We can pass things in the exception, like a URL, so the agent can be told immediately where it can go to download a newer agent.

• Can the agent download the update to a tmp location, call System.exit(42), have the rhq-agent.sh/bat see the special exit code, know that it means it should update and restart and restart the agent immediately? The script could also purge the old agent, unzip the new agent distro and restart it.

• Have the agent distro located in a simple location on the Server so a simple HTTP GET can download it. I don't think we need to go through the agent-server comm layer to simply download the agent.

  • The agent should be configurable to be told where to look for the updates. If we publish the updates in rhq-downloads, we must make it so Tomcat can serve it directly (I do not think we want it to go through our comm layer). In addition, I suspect we'll want to be able to serve the updates from another web server, not the RHQ Server - because its possible 100s, 1000s of agents will request these update files and we do not want to clobber the RHQ Servers just so they can serve file on the order of 10s of MBs in size, via HTTP GET. I envision being able to put an agent zip on a web server, along side its metadata file that indicates its version information. Agent starts up, goes to its configured "rhq.agent.update-url", grabs the metadata info file, determines if this update is applicable and pulls down/applies the update if appropriate.

• Rather than ask the agent script to perform the update, can we do it in Java? We first must completely unload the agent from the VM (PC must be stopped, all agent threads killed and AgentMain's classloader completely unloaded - we then load in a very small, simple AgentUpdate class and have it perform the update. At which time, it restarts AgentMain - or asks the script to restart it via System.exit(42).

• We should ship with a standalone jar (agent-update.jar) that goes in /lib (maybe /lib/update). This is so small we should never have to update it (because if we do, we are then screwed if we have to ever update it, itself . But I envision AgentUpdate class just being a simple loader class that loads ANT and we ship an ANT script in the agent distribution. That ANT script will get extracted from the zip and run. The ANT script can perform all the update stuff. Our installer embeds ANT so we can look at its code and duplicate what it does when needing to launch ANT from within Java code.

• Careful with the System.exit() idea - that would kill the server if the agent is embedded.

• Before the agent attempts to make the initial connectAgent, it could make a HTTP GET request to "server-transport://server-endpoint:server-port/agentupdate/version". The default URL used by the agent will be a URL "/agentupdate/version" of the RHQ Server it is talking to. This URL is configurable via agent preference rhq.agent.agent-update.version-url. The download URL is also configurable as rhq.agent.agent-update.download-url. This is to allow a deployment environment where all agents can get their updates from a separate web server, apart from the RHQ server. The agent will take the response, compare it to the agent's own version (which it can get via the Version class) and determine if the agent needs to update. If it does not need to update, it continues. If it does need to update, then the agent must abort everything it is doing, and begin the update process.