RHQ is fully open source - community contributions and help are greatly appreciated and needed.
If you are a developer, there is a full set of additional RHQ Developer Documentation available.
RHQ consists of the core server engine and agent plus some additional plugins used the manage some middleware components.
The RHQ project utilizes Maven for its build infrastructure. You must have Maven installed in order to build.
As part of your Maven installation, you should have a $HOME/.m2/settings.xml file to customize your Maven build environment. You typically do not need a settings.xml when building RHQ because the defaults normally suffice. However, because the RHQ plugins require some JBoss dependency libraries, you'll need to point to some JBoss specific repositories in your own settings.xml. Look at the RHQ source for an example settings.xml file.
To learn how to build RHQ, please see the Building RHQ wiki page. You will also probably want to read the Advanced Build Notes as well.
If you put the source to your custom agent plugins along side the core RHQ plugins (and essentially tweek a copy of one of the existing plugin pom.xml files to match your custom plugin information), you can build it at the same time you build the core RHQ code. Otherwise, you'll have to run your own custom build to package your plugin jar and manually copy the plugin into the appropriate location within the server (e.g. in the plugin dropbox directory at <install-dir>/plugins - or if you are using RHQ 2.3 or earlier, <install-dir>/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins). See the skeleton plugin maven module in /etc/samples for an example of an independent plugin project layout.