Define a Bundle

The application architect or engineering services team will provide the definition and content of my application/project/business service to the RHQ provisioning system allowing it to be deployed as part of the development lifecycle. This is assuming a typical JEE application with JBoss EAP or EWS. Most likely at least one database is involved and potentially other resources such as ldap, mail servers or web services.

1. The application construction and configuration will be reviewed to decide on the deployment file layout and the files that must be configured for specific environments

   1. The application recipe will be written to define this model and will be tested in development deployments to ensure that the application is fully deployed and working (including upgrade and clustering scenarios when appropriate)

   2. Check that the service definition configuration will be valid for all environments (shared service installation and definition)

2. The environment will be defined and documented with the set of configurations needed for deployment to other environments (e.g. qa, stage, perf and prod)

   1. Will we need to let the bundle provider document the meaning of a configuration property? (Now which database is this property supposed to point to?)

3. Deployments to QA using alternate infrastructure will provide proof that nothing is hard coded for dev deployment scenarios

4. Integrate bundle construction into build system. Track recipe in SCM. Build bundle package.

5. Update bundle in RHQ with up-to-date automatically build bundle and release information.

The recipe should be a system of storing bundle data outside of the RHQ database. RHQ can not be the system of record for the bundles themselves. This implies a certain amount of data that the bundle should include and potentially argues for another abstraction layer between the recipe and the system such as a bundle metadata file. The following data is needed for a bundle.

• Name, description, (tags?)

• Version

• compatibility, requirements, dependencies (e.g. supported os'es, disk space reqs)

• if bundles might be shared publicly

  • More info, web site links

  • license

• Questions for property replacement / environment

Deploy a new Bundle

1. Assume the machines to be deployed to are already in inventory

2. The deployer will review the deployment plan of machines to deploy against

   1. Possibly need to create a group for those machines to allow group deployments

   2. Review the state and utilization on those machines (multiplexed deployments are the common case)

   3. Review other utilization like ports on those machines

3. Review dependent infrastructure and predeploy

   1. Is the load balancer in place and ready for configuration?

   2. Is the database up and available from the machines to be deployed to and running the proper schema?

   3. Are LDAP, webservice dependencies and other services in place?

4. Define deployment configurations

   1. Set the production database password

   2. Set the production admin domain password

5. Review predeploy plan

   1. What machines is this going out to and what affect will the deployment have on them?

   2. Is there enough memory available on the machines to be deployed to? How about disk space? Network utilization?

   3. What will the configuration files look like once processed for deployment?

6. Verify service functionality

   1. Check deployment validity, review start logs

   2. Confirm function (on all cluster nodes) and access through the load balancer

   3. Confirm dependent functionality (ldap, web services, etc)

   4. Import and confirm inventory integration into RHQ for monitoring

   5. Setup RHQ monitoring. Add alert conditions. Define cluster groupings

Upgrade a bundle

1. A new build is available from development and is provided by release engineering into the RHQ bundle system as an update

2. Confirm recipe validity in dev testing. (Recipe should be kept up to date as development alters configuration requirements)

3. Add any necessary deployment documentation to the recipe

4. Bundle version is marked as ready for further deployment

   1. As a new version is tested through the lifecycle... how do we keep the prod deployer from accidentally pushing it to production?

   2. How are notes passed on the deployments between developer, release engineering and deployment?

Deployment upgrade

1. A deployer is updating a production deployment and has confirmation on the bundle version to be released to that environment

2. Confirm that any dependency changes that are different are reviewed (is that new web service dependency up?)

   1. It's a load balanced application? Are we putting up the "down for upgrade" message?

   2. This release also requires a schema upgrade for the application. Are we shutting down the old app and waiting for the upgrade to be run by DBAs?

   3. What if the new release will be adding a few nodes to the cluster?

3. Jump to step 5 of the first time deployment

Deployment fails

1. Deployment does not start properly

   1. A port conflict

   2. File permissions fail

   3. db schema not upgraded successfully

   4. Dependent service not upgraded in sync

   5. gremlins

2. How do i get to both the deployment logs and the start logs for the services i deployed?

3. Let the recipe define "important paths" where detected file changes could be dangerous. (e.g. a new file in the deploy directory is bad... one in the work directory is ok)

Deployment Revert

1. Reverting of deployments is necessary (particularly in deployment failures)

   1. This goes back to the state right before the deploy (not the previously installed version)

   2. Either we save an entire backup, or more likely, we install the previous version and add in the backed up files

Workflows

For those who won't have a feed of their builds into the bundle system, a user will have to upload specific releases into the system. For bundles to work outside the rhq deployment path more information will also be needed. Local inventory implies version info should be in the bundle so a local deployment can properly inventory itself. An answer file format should be supported for testing deployments or for export/import for other environments.

Uploading a bundle package should either create a new bundle and completely set it up or add a new version to an existing bundle. The wizard would allow the user to confirm the operation to be executed.

Deployment Defs should be managed separately to allow sys admins to define their environments and the question responses for a bundle. It'd be really useful for these def's to support an environment tagging system so there can be correlation between different defs of the same environment (dev, test, stage, prod, dr).

A major question exists on how to manage what environments a bundle version is ready for? A typical five environment system will often have 2 to 3 active versions between the environments and tracking where a version belongs at any given point is critical. Otherwise they'll accidentally push version 1.1 to prod which is still in early testing instead of that 1.0.1 patch that was already confirmed for deployment. A separate workflow should be designed for managing those deployment assignments.

Recommend we build the triple tagging system discussed before to support folksonomy and assignment workflows. Add to bundle versions, resources and deployment defs. See Design - Tagging for details.

• environment=[dev,qa,stag,perf,prod,dr]

• qaStatus=[untested,tested,confirmedForProduction]

• department=[consumerBanking, comercialBanking, investmentBanking, hr, it]

• project=[SpecialExplorationProject, DataConsistencyProject, PublicWebsite, LdapServices]

When installing, the user should be walked through the expected outcome. The question response screen may have already been filled out for an environment, reuse it. Allow user to confirm file munging results (a screen with all the files to be munged and a way to preview them across the deployment (for each box)). Detailed steps to install should be exposed to the user. The ant workflows should include the results of each step. Preview the steps before it starts? Watch the results as the system progresses. (Do we have a way to say upgrade/install all servers at once or one at a time? Can we stop or continue on failure?)

Error handling workflows will mostly involve collecting and displaying logs and step status. Which step failed? What was the outcome? What was the detailed logging (from Ant)? Need a system to quickly revert to a previous version on failure to deploy the next version.

Supported Scenarios

1. Execution features

   1. Install a bundle to a new directory

   2. Check install status and report diffs to server

   3. Pre-deploy check allowing you see what the files will look like after property replacement

   4. Install a bundle over top an existing install (upgrading and backing up as described above)

   5. Install via Ant can be run offline and disconnected from the server

   6. Install steps supported via Ant tasks

      1. Shutdown

      2. Check service status

      3. Check deploy file statuses

      4. Install or Upgrade install directory

      5. Inventory of install

      6. install / upgrade / remove OS service

      7. Input properties (via file or interactive QA)

      8. File property replacement

      9. Startup

2. CLI

   1. Upload bundle package with content and ant recipe (new bundle or new version)

   2. deploy to resource or group

3. GUI

   1. Bundle / Bundle version upload via bundle packages

   2. Setup deployments w/ tagging

   3. Resource / Resource Group tagging  and tagging display in deployment screens

   4. Choosing resource or group for new deployment

   5. Choosing a new version / deployment def for a previously deployed resource / deployment combination

   6. Deployment execution

      1. Preview of changes and status

      2. Preview of steps to install (targets & tasks)

      3. Review of inventory file status report (e.g. file was changed since last deploy)

      4. Execution status (across groups, all together or one at a time)

      5. Detailed step review including error status and log messages

   7. Resource / Deployment relationship

   8. Bundle upload via URL (give the wizard a URL and have it suck)

4. Other

   1. Resource to deployment linking through install inventory metadata and discovery utilities (custom resource relationships or fields?)

   2. Resource tagging management

Future Scenarios

1. Content Providers that can automatically pull release builds from external build management systems or SCM's