The RHQ Server, at its core, is a JEE application. It has two main deployment units - the main EAR and an Installer WAR - and they are deployed inside a JBoss Application Server (currently on JBossAS version 4.2.3, but soon to be 7.x). The main EAR contains the full server-side business logic (all the EJBs, JPA entities, web GUI, etc). The Installer WAR is technically only needed one time when the RHQ Server is run the very first time after the RHQ Server distribution is unzipped onto the filesystem. The EAR should not be deployed until after the Installer WAR has finished its work.
Consider that out-of-box, we don't know certain environmental settings the user wants to use. The main example to look at is the type of database being used. RHQ supports both Postgres and Oracle for its backend database. However, we don't know what the user wants to use hence we don't know what configuration settings should be used for RHQ's app server datasource (for example: What's the JDBC URL? What's the username/password for the database? Should it be using the Postgres or Oracle JDBC driver?). Therefore, out-of-box, we don't have RHQ's datasource even defined yet.
Also, for new installations, the user's database won't have the RHQ schema - e.g. it won't have the necessary tables that RHQ needs. If the installation is an upgrade from a previous RHQ version, the schema may exist in the database, but it may be old and would thus need to be upgraded.
Other minor issues are things like: What port does the user want the web GUI to listen to? What's the SMTP hostname to be used by RHQ when it needs to send emails?
This is where the Installer WAR comes in. Out-of-box, RHQ Server's Installer WAR will be deployed/enabled but the main EAR is not enabled. The main EAR cannot be deployed yet because it will fail immediately (e.g. the JPA initialization will fail because the datasource has not be created yet and the database itself may not even have the RHQ tables yet). When the user points her browser to the RHQ Server, the Installer GUI will be displayed. The user will be able to fill in the form specifying things like the database type, JDBC URL, SMTP hostname, etc. Once the answers are supplied to the questions in the Installer's form, the user will click the "Install" button and server code running in the Installer WAR will perform all the necessary work to setup the application server for use by RHQ. This includes things like telling the app server to create a datasource for the appropriate database, setting up the email service with the appropriate SMTP hostname, running database schema creation/upgrade tasks to create the necessary tables in the database, etc.
RHQ's Installer WAR can be run in "auto-install" mode. This means if the rhq-server.properties file (which is nothing more than a file of system properties passed into the app server via the -P option) is fully configured already, the Installer WAR will detect this and automatically do the same things as if the user pressed the "Install" button. Because all settings are defined in rhq-server.properties, the Installer WAR can perform the install without user intervention. So technically we could decide to not provide the user with a Installer GUI at all, but these tasks performed by the Installer WAR will still need to be performed regardless.
Once the Installer WAR completes all the install tasks, it will then immediately deploy the main EAR. It will also attempt to undeploy itself since it is no longer needed - the installation tasks are done and the Installer WAR has nothing else to do for the life of that RHQ Server. At this point in time, the main EAR can now successfully start up because things like the datasource and email service are already created and fully configured and the database now has the proper RHQ schema. The main EAR, once finished initialization, can now serve up the main RHQ GUI and we now have a fully functional RHQ Server.
We are currently porting the RHQ Server from our previous JBossAS 4.2.3 container to JBossAS 7 (specifically, we are currently working on JBossAS 7.1.1.Final).
We still have an Installer WAR and a main EAR bundled with an AS7. The Installer WAR and the main EAR are stored in the AS7's standalone/deployments directory to be picked up by the deployment scanner. The Installer WAR has an associated .dodeploy marker file and the main EAR has an associated .skipdeploy marker file (this is how we get the Installer WAR to be deployed and the main EAR not deployed when initially running the RHQ Server for the first time).
When the Installer WAR's "Install" button is pressed (or the "auto-install" mode is triggered), the same kinds of install tasks are performed (as described above). Once the Installer WAR is done, it will rename the main EAR's .skipdeploy marker file to .dodeploy, thus effectively deploying the main EAR once the deployment scanner detects the new .dodeploy file. The Installer WAR will also give itself a .skipdeploy file so it will no longer be deployed (it will also remove its .deployed file so it gets undeployed immediately).
This is currently working today in RHQ's rhq-on-as7 branch.
Today, the RHQ build will package AS7 under the main installation directory of the RHQ Server. The RHQ Server has its own files that it ships with (like a startup script and its rhq-server.properties file where its initial settings are stored). Along with these files of its own, RHQ comes with AS7 under the subdirectory "jbossas". Currently, RHQ will not ship with any modified AS7 files - it will ship with a stock AS7. The only thing changed under this "jbossas" directory is that the Installer WAR and main EAR are found in the "standalone/deployments" subdirectory. For example here's a portion of what the RHQ Server installation looks like on the file system:
<rhq-install-dir>
|___ /bin (contains startup script 'rhq-server.sh' and startup config 'rhq-server.properties')
|___ /modules (contains custom AS7 modules - e.g. RHQ's Oracle and Postgres JDBC drivers)
|___ /jbossas (a stock AS7 distribution - under here is modules/, standalone/, etc.)
|___ /standalone/deployments (the Installer WAR and main EAR are in this directory)
RHQ is using a stock AS7 (that is, not customizing the files under "jbossas") because it should make it easier to upgrade to a newer version of AS7 in the future. With less customizations the RHQ build system is required to make in RHQ's accompanying AS7 distribution, the less work it should be to upgrade to a newer AS7 version in the future. This also should make it easier for RHQ to support being installed via RPM where the AS7 is to be installed separated via its own RPM (see below for more on this). The less customizations RHQ needs out-of-box, the less its own RPM needs to do when being asked to be deployed in an external AS7 that is supplied separately.
One important thing to note is that RHQ does not currently come bundled with its own customized standalone-full.xml configuration file. RHQ will rely on the stock version of that file that comes with the AS7 distribution. The Installer WAR will make ModelController API calls (using the normal DMR ModelNode API) to do things like create and configure datasources and JMS queues, setting up the email service's SMTP hostname and port, etc. We do this for a couple of reasons:
First, so we do not have to maintain our own copy of standalone-full.xml which would put us at risk of having this custom configuration file grow obsolete over time as new AS7 versions are released. We simply want to make runtime management API calls to the AS7 instance that RHQ is running in and let AS7 maintain the full configuration. This allows us to customize AS7 only where we need, without needing to configure the entire AS7 infrastructure (which again, risks us growing obsolete over time as new internal AS7 extensions are added, removed or modified).
Second, this will also allow us to be able to support mutiple versions of the AS app server (assuming the management API calls we make remain backward compatible). If we maintain our own modified copy of standalone-full.xml, we risk not being able to deploy on different versions of AS7 since our copy of standalone-full.xml may not be applicable to other versions of AS7 (other than the one which our standalone-full.xml was based on). This could be important when it comes to RPM installations (see below).
One new requirement on the RHQ Server is that it must be installable via RPM and, not only that, but it should be installable in an AS7 instance that, itself, was installed via an RPM.
What this means is that our RHQ RPM distribution does not need to be shipped with an internally packaged AS7 distribution. It also means that RHQ will not be able to dictate the AS7 distribution it will run on - it will be supplied externally by the user (not by the RHQ packaging). The RHQ RPM can have a dependency on the AS7 RPM - when installing RHQ via RPM, it will first require AS7 be installed via RPM.
Note that if RHQ were to package its own customized standalone-full.xml, this would restrict the versions of AS7 RHQ could be installed in (e.g. if a user's environment is restricted to only allow a specific version of AS7 to be installed, RHQ wants to have the best chance to support that version - having a full, customized standalone-full.xml based on a specific (and perhaps different) AS7 version than what the user has adds risk to this). This is also the reason why RHQ is trying to maintain the goal of shipping with a stock AS7 in its normal distribution. The less customization we have to do in the normal "black box" RHQ distribution, the easier is should be to support an RPM installation in the future.
Currently, there is no RHQ Server RPM distribution. This is a new requirement and will be supported in the future.
There is a patching mechanism being developed for AS7. In that wiki, there is a section "Patching Deployments" that tells us the preferred way for patching applications is not to use the deployment scanner and the deployments/ directory, but rather to deploy things using a custom module/subsystem extension (here's another page that talks about extending AS7). What this means is, it is highly recommended that we ship our deployments (Installer WAR and main EAR) in a custom module and that custom module's subsystem extension will make the necessary management API calls to deploy the Installer WAR and the main EAR. This avoids the use of the deployment scanner (which means we can't use the .dodeploy/.skipdeploy markers) and it will enable RHQ to be patched using this new proposed patching mechanism.
Brian Stansberry has a prototype that shows how to deploy an EAR using a custom module subsystem extension - see the "Mixed Approach" section in the Extending AS7 wiki. I have taken this prototype (along with reading the Extending AS7 docs and looking at the rtfilter-subsystem that Ian wrote awhile ago) and have the start of RHQ's own custom module subsystem extension. However, I've come to see that there are things we need to consider before embarking on completing this work.
First, the "Mixed Approach" section (and its associated prototype) assumes that the deployment will always be deployed. In our case, we have two deployments (Installer WAR and main EAR) but only one can be deployed at startup (the Installer WAR). The main EAR must not be deployed until the Installer WAR completes its tasks (as explained in the above earlier sections). Note that this might not be until the user fills in the form on the GUI and presses the "Install" button.
Once the installation is done once, the Installer WAR should never be deployed again (it is only needed that one initial time). So for each subsequent restart of the RHQ Server, it is the main EAR that now should always be deployed. The custom module subsystem extension needs to be able to know if the Installer WAR has already previously been run - if it has, then it needs to only deploy the main EAR.
Because we have two deployments with different times when they should be deployed, there needs to be some workflow that the custom module subsystem extension must handle. When the subsystem extension deploys the Installer WAR, it must be able to be told (or it must be able to figure out) when the Installer WAR finished its installation tasks. At that point, it needs to undeploy the Installer WAR and then immediately deploy the main EAR. If you recall in the earlier description of the current implementation, it is the Installer WAR that does this. The Installer WAR knows when it is finished doing its installation tasks, so today it modifies the .dodeploy/.skipdeploy/.deployed marker files to switch around which deployments are deployed and which are not. Now, the Installer WAR must make a call back into the subsystem extension to tell it "the installation is done", at which point the subsystem extension should undeploy the Installer WAR and deploy the main EAR.
This approach affects packaging. The Installer WAR and the main EAR will be stored, not in the standalone/deployments directory under "jbossas" (as explained earlier) but they now will be stored in the custom module itself. This is actually nicer because this can simply be just another module that gets stored next to our already shipping custom modules for the Postgres and Oracle JDBC driver modules (<rhq-installation-dir>/modules) and we don't have to modify the stock AS7's deployments/ directory (i.e. we can leave it empty). However, the question now becomes, how do we install this custom extension? For the AS7 that ships directly with the RHQ Server (that is, under "jbossas" subdirectory in the RHQ install directory) it means we need to now make a custom modification to standalone-full.xml (and if you recall, we are trying to avoid making any kind of modifications to the stock AS7 configuration files). But, so be it - if we need the RHQ build system to add a custom <extension> tag to the stock AS7 standalone-full.xml that ships with RHQ Server distribution, then we can do that (this probably will have no effect when moving to future versions of AS7 since it is doubtful that the <extension> schema supported by AS7 will change and render our custom modifications obsolete). However, what about the case when a user is providing an external AS7 for RHQ to run in (as is the case when installing via RPM)? This means we need to customize the external AS7 that RHQ is going to be running in. In the RPM instance, it means we will probably need an RPM script to modify the AS7's standalone-full.xml file. Either that, or we ship with our own full, customized standalone-full.xml (but as explained earlier, we want to avoid that for reasons explained).
I confirmed that the following CLI commands will install our startup subsystem in a running AS7 instance:
/:add-namespace(namespace=rhqstartup,uri=urn:org.rhq:startup:1.0) /extension=org.rhq.rhq-enterprise-server-startup-subsystem/:add(module=org.rhq.rhq-enterprise-server-startup-subsystem) /subsystem=rhq-startup:add()
If we are to do this from within a script, assuming jboss-cli.sh is on the PATH, we can do this:
jboss-cli.sh --connect --command='/:add-namespace(namespace=rhqstartup,uri=urn:org.rhq:startup:1.0)' jboss-cli.sh --connect --command='/extension=org.rhq.rhq-enterprise-server-startup-subsystem/:add(module=org.rhq.rhq-enterprise-server-startup-subsystem)' jboss-cli.sh --connect --command='/subsystem=rhq-startup:add()'
All of the above requires a lot of work - coding, testing and maintenance. If you notice, the bulk of this work is in order to provide the user with a friendly user interface during the installation phase. We give the user a way to fill in a graphical form, press "Install" and we make all the necessary modifications to the underlying app server service configurations. But it doesn't have to be this way.
What if we threw away the requirement that we ship with a separate Installer WAR? What if we simply ship with a standalone-full.xml that isn't quite 100% fully configured (it would have all the RHQ services and settings, but missing some of the things we need the user to provide - such as the database username/password, JDBC URL, port values, SMTP hostname, etc.)? We would require the user to edit the standalone-full.xml to fill in the missing pieces (with the aide of documentation of course). There would be no need for an Installer WAR (with one caveat I'll mention below). The user preconfigures the RHQ Server prior to running it by simply editing standalone-full.xml. When starting the RHQ Server via rhq-server.sh, we will just pass in our custom standalone-full.xml file for the AS7 instance to use.
What does this solve? It immediately means we no longer have to ship with a separate Installer WAR. The main EAR can be deployed at startup, even the first time (this means we can use the mechanism that is demonstrated in that custom module subsystem extension prototype found on the Extension AS7 "Mixed Approach" wiki mentioned above). We no longer would have to design, develop and maintain an elaborate Installer WAR.
This does not solve the problem of preparing the database schema. Somehow, we need to create the tables in the database (if its an upgrade, we need to upgrade the existing schema). We could ship with a standalone utility to do this - so as part of the installation phase, the documentation would have to explain to the user that they not only have to edit standalone-full.xml to properly configure the RHQ Server but they would have to run this utility to prepare the database. Such a utility would not be hard to write - we have most of the code in place for this.
While not user-friendly to the end-user, it solves alot of problems from an implementation point of view. It saves alot of time and work but it does make the installation of the RHQ Server more difficult for an end-user (because now they need to edit XML and potentially run a standalone DB utility, as opposed to answering some questions on a GUI page and clicking an "Install" button).
We could add a standalone installer to avoid having to deploy a second, separate Installer WAR in the app server (which eliminates all the complexities that would involve). We could refactor the actual installation code into a standalone Java app that could be called from a script. This would be cross-platform supportable (i.e. we could do this for both Windows and UNIX/Linux based platforms). This would require the user to run a separate rhq-installer.sh script (rhq-installer.bat if on Windows) rather than rhq-server.sh (.bat) the very first time. The workflow could be something like this:
User unzips the RHQ distribution on the filesystem. This would come with a stock AS7 (note - if AS7 was installed via rpm, our RHQ distro would need to be told where it is - say JBOSS_HOME or something like that. If RHQ is installed via an RPM, perhaps the RPM install script would be able to store information about the location of the AS7 that RHQ depends on).
User runs "rhq-installer.sh"
Installer script runs the stock AS7 server (this has no RHQ cusotmizations in it at all)
Note that we might want to run the AS7 in admin-only mode
Installer script runs the Installer Java code (which is to be bundled in a standalone jar). This Installer code would:
Ask the user (via console prompts - similar to RHQ agent setup) for settings such as JDBC/DB settings and server settings
Perform the database setup/upgrade ensuring the schema is prepared properly
Run the AS7 DMR/ModelController commands to configure AS7 for RHQ (install datasources, JMS queues, setup email service, etc)
Run the AS7 DMR/ModelController commands to install the new extension subsystem (which comes with the main EAR)
Restart the AS7 in non-admin mode
At this point, the AS7 has been fully configured, is deployed with the main RHQ EAR and is ready to accept requests.
Some things to consider:
We would have to rework the way we do our dev builds. Today, our dev builds are nothing special - they just turn on the auto-installer which is run by the Installer WAR. We would have to do something different since the Installer WAR is going away. Perhaps we bundle our dev builds with a pre-configured standalone.xml. Since its dev mode, this isn't too bad. Developers that want to run/test on a different AS7 have the option to run through the new installation procedures.
We would need to build a new console-input, standalone installer app. We would need to ask the user for all the things our GWT installer GUI asks for.
We would need to refactor out the Installer WAR code that does the actual installation and put it into a standalone jar module. This way we can bundle it in a standalone jar that our new installer .sh script would call. So we'd need something like these maven modules:
We would need to build a new rhq-installer.sh/.bat set of scripts