JBoss Community Archive (Read Only)

RHQ 4.8

Design-DeployAgentPluginsInDatabase

Deploying Agent Plugins In The Database

The Issues

The agent plugins are deployed in the server's file system (rhq.ear/rhq-downloads/rhq-plugins). The ProductPluginDeployer is a JBossAS deployer object that detects when files are added/modified/deleted from that directory and when it does, it properly modifies the plugin metadata in the database. This mechanism has some major flaws:

  1. You must keep plugins identical across all servers in the cloud.

  2. You must shutdown all servers in the cloud before updating a plugin because within seconds, as soon as you copy a new plugin to the rhq-plugins directory, the metadata in the DB is updated and thus all the servers will see that new DB metadata. But if another server tries to server up its plugin content to an agent, it will not have the correct agent plugin (the Db metadata it sees is out of sync with the old plugin that it still has). You need to change all servers plugins at the same exact time, atomically.

  3. If you bring up 2+ servers at the same time, they all try to deploy/manipulate the metadata in the database at the same time (see previous point as a corolary)

If we had a mechanism such that the plugin content is stored in the database instead of the file system, that would mean

  1. Its very easy to update a plugin - tell a server (any server) about the new plugin and the new plugin content as well as the metadata can be updated atomically to the database

  2. We can easily add a plugin (or delete a plugin) in the same way (though this has complications with respect to deleting a plugin that has resources that are already in inventory - in addition, the agents still have the plugins and need to be told to delete their plugins and the resources from those plugins)

  3. We can (possibly) support agent failover while downloading plugins (since we may be able to stream chunked plugin content per agent request - e.g. agent can ask for plugin content bytes #0-1000 from server A, and if server A goes down, the agent can ask for bytes #1001-#2000 from server B. Helps the agent be a bit more fault tolerant of server failures during plugin download.

Cases

This section will discuss the different use-cases for deploying the agent plugins in the server.

Assume all agent plugins are still shipped in the "rhq-plugins" directory on the server as they always have been, but assume that the RHQ_PLUGINS table now has a BLOB column that can store the actual content of the plugin jar files. The "plugin deployer" (the custom JBoss Deployer that processes .jar files found in rhq-plugins) runs in the server and handles the processing of plugins that are deployed in the server.

Single server coming up new

This use-case is the typical one when installing a server for the first time. You have a single server in the cloud and you are starting it up for the first time. There is no content in RHQ_PLUGINS but all rhq-plugin jar files are up to date.

  • The plugin deployer needs to insert all the plugin content into the RHQ_PLUGINS table

Deploy new plugin and restart server

While the server is down, you copy a new plugin to the rhq-plugins directory then restart the server. RHQ_PLUGINS has the original plugin content for all plugins, which is up to date, but it is missing the content for the new plugin.

  • The plugin deployer will need to insert a new row into RHQ_PLUGINS to represent the new plugin.

Deploy an updated plugin and restart server

While the server is down, you overwrite a currently existing plugin with an updated version of that plugin then restart the server. RHQ_PLUGINS has the original plugin content for all plugins, which are all up-to-date except for the newly updated plugin.

  • RHQ_PLUGINS needs to be updated with the updated plugin content. No new rows will be inserted in RHQ_PLUGINS, since there is already a row for the existing plugin that is being updated.

Install and start a new server in the cloud

A second server is installed and started in the cloud. No new plugins are installed in this new server. The second server has the identical set of plugins that are found in the first server. RHQ_PLUGINS has the full set of plugins which is up-to-date.

  • The second server's plugin deployer should see that all jar files found on the file system are the same as that found in the database and thus nothing on the file system needs to change

Deploy a new plugin in second server

With a 2-server cloud, you deploy a new plugin in the second server. The first server can either be running or shutdown.

  • The first server does not need to know anything about what happened to the second server's plugins. At this point, however, the first server's set of plugins are different than the second's set of plugins (on the file system) - the second server has one more plugin.

  • If an agent requests the new plugin from the first server, the first server must provide the new plugin content. The first server will have to now download the new plugin content from the database and stream it to the filesystem.

  • When the first server restarts (and an agent never asked it for the new plugin), it should see that it is missing a plugin from RHQ_PLUGINS, so it should stream the new plugin contents from the DB and write the plugin on the first server's filesystem. This helps keep it up to date and ensures the first servers' startup processes the correct set of current plugins.

Deploy an updated plugin in second server

With a 2-server cloud, you deploy an updated plugin in the second server, overwriting the original plugin jar. The first server can either be running or shutdown.

  • The first server does not need to know anything about what happened to the second server's plugins. At this point, however, the first server's set of plugins are different than the second's set of plugins (on the file system) - the second server has the same number of plugins as the first but one of the plugins is different.

  • If an agent requests the updated plugin from the first server, the first server must provide the new plugin content. The first server will have to now download the new plugin content from the database and stream it to the filesystem.

  • When the first server restarts (and an agent never asked it for the updated plugin), it should see that it is has an outdated plugin, different from RHQ_PLUGINS, so it should stream the updated plugin contents from the DB and write the plugin on the first server's filesystem (overwriting the old plugin jar file). This helps keep it up to date and ensures the first servers' startup processes the correct set of current plugins.

Hot Deploying a new plugin in 1-server cloud

While the server is still running, copy a new plugin to the server.

Hot Deploying a new plugin in 2-server cloud

While the second server is still running, copy a new plugin to it.

Hot Deploying an updated plugin in a 1-server cloud

While the server is still running, copy an updated plugin to the server, overwriting the original plugin.

Hot Deploying an updated plugin in a 2-server cloud

While the second server is still running, copy an updated plugin to it, overwriting the original plugin.

Removing a Plugin

Removing a plugin, either in a single-server cloud or N-server cloud is much more complicated than adding new plugins because it not only involves manipulating the RHQ_PLUGINS table, but it also involves possibly removing resource types AND resources that are already in inventory. Due to this added complexity, the "remove plugins" use-cases will not be discussed on this page.

JBoss.org Content Archive (Read Only), exported from JBoss Community Documentation Editor at 2020-03-12 14:57:20 UTC, last content change 2009-01-28 14:51:15 UTC.