Some stream of consciousness design:
Ant becomes the preferred recipe language...
It's extensible and has a large set of tasks available for it (http://ant.apache.org/external.html)
Known by developers and release engineers
Should be testable outside of the bundle system (test locally and push when it works)
Issues with Ant
How do we track status and errors? Need task listeners and error handlers that can report on the details of each task or target
Still need a system for upgrades (see local inventory below)
Can we run the recipe execution as another user to limit security risks?
Should we build in service-installation, status checking, etc. as a first class concept? That would let it be outside the recipe and os-agnostic.
Other Ant tasks
Dependency checks (db there and responding?)
A set of prerequisites (e.g. confirm diskspace available)
Service management tasks... install/status/etc.
Local inventory
The idea here is to keep a file inventory on the box that contains information about this installation for upgrades. On the initial install, the files are laid down in the directory and then a process discovers all the files and records their paths and MD5 sums. This information will later be used for upgrades and change checks. The md5 inventory file could be stored in a .rhqinstall directory in the installation path... or perhaps a proper system location dependent on operating system (/var/rhq/).
To perform a bundle status check, the md5-er process comes along and creates another full md5/path file. These files can then be transferred to the server to let the user see the list of files that are [new, edited, missing]. We might want to also store this file somewhere in our deployment record. The file should also be kept on the system because it'll be an ant task. This would also let someone create a "change" report locally on the box cli with Ant.
Based on the above, we can offer a "fix" that would revert changed files back to their original states.
"Fixes" and "Upgrades" would operate along similar lines to RPM upgrades: http://www.rpm.org/max-rpm/ch-rpm-upgrade.html
When upgrading, conflict files would be backed up to the inventory directory... e.g. ".rhqinstall/upgrade_v1_to_v2/path/to/file/my-ds.xml.backup". If we do this directory properly we should be able to offer (now or in the future) full reverting to previous installs (including local changes).
Also, the important server-based plugins would be updated to integrate with the .rhqinstall directory. For example, that directory will contain a bundle.info file. This file will have textual information about the installation, but also contain the id of the bundle deployment which can be reported back as a new set of traits. "rhqinstall.bundle.id", "rhqinstall.bundle.deploy.id". This information would then later be linkable so that the discovered resources can be linked back to their deployments. (i.e. You'll be able to jump directly from the bundle deployment to the EAP instance in inventory and do the reverse... as well as make a decision to do a "bundle upgrade" right from the EAP inventory pages) The "discovery" of this information would be a provided plugin utility so that any plugin could support discovery of this information. (we also have the option of making this a first class piece of data attached to resource... but traits wouldn't be a horrible implementation... it's already got a discovery interval and does historical storage and is hooked into DynaSearch and alerts)
Testing outside of the bundle system
The Ant files should be runnable outside of the bundle system with an easy to get ant framework. (the basic ant plus our extension jars) We should ship this as a downloadable package from the server. It'd be a custom package because it may differ from what is used locally to do Ant based builds. We should customize the start scripts to use rhqinstall_ant_home instead of ant_home for example. There should be a task/target that checks all variable inputs. This would be a verification step in the normal deploy, but in a test developer deploy it would be used to allow the developer an alternate way to pass in the custom config properties and file paths. These should be passed in via a matching properties or yaml file. These files would be able to be exported from the UI for a given bundle deploy profile. That way local testing could happen against any set of input properties to be used in the real deployment.