git clone git://git.fedorahosted.org/rhq/rhq.git
Running a lot of Agents
In order to test scalability of the server, we want to be able to run hundreds of agents without incurring the cost of hundreds of physical machines.
There are two different toolsets we can use to do this:
The source code and test scripts for those two toolsets are found in the git repo under the directory etc/agentspawn. You will need to at least git clone the repository so you can get to the source and scripts:
Notice
When running a high number of agents (which are highly multi-threaded) you may run into process limits on some Linux systems. In that case either use ulimit to change the limits for the agent user or configure the limits.conf file to do the same (on Fedora/RHEL, this file is located in /etc/security/limits.conf).
AgentSpawn
agentspawn contains a Maven POM as well as an ANT script that you can use to run multiple VMs, which can have multiple agents running within them. Assuming you have a machine with a lot of memory and CPU capacity, you could run a good number of agents using this mechanism.
AgentSpawn Setup
You should make sure you perform a full trunk build of RHQ so you have an up-to-date RHQ Agent built in modules/enterprise/agent/target/ - it will be used as the template agent that will be spawned by these agentspawn tools.
If you do not want to perform a full build, just ensure you have an agent distribution that you can point the "perftest.agent.dist.zip" setting to. If you opt not to have a full build environment, you still need to checkout the agentspawn source module because you will need to build that module next.
You need to "mvn install" the agentspawn module to build the jar file containing the class that will spawn its internal agents. After that completes, you will have a jar file in the "target" directory located in the agentspawn module's root directory.
AgentSpawn Settings You Should Know About
Whether you are going to run a single VM with multiple agents, or you are going to run mulitple VMs each with multiple agents running within them, you will need to understand what some of the settings are that are needed by agent spawn. These settings are defined either as -D system properties (when running a single VM via the jar) or within agentspawn.properties (when running multiple VMs via the ANT script).
|
Property name |
Jar System Property? |
agentspawn.properties? |
Description |
|
perftest.agent.dist.zip |
|
|
The full path to your agent distribution zip |
|
perftest.agentdist |
|
|
The full path to your exploded agent distribution (not to a zip/tar) |
|
perftest.agent.vm.dir |
|
|
The full path pointing to the root directory where all your agent's will be spawned on your file system |
|
perftest.agent.vm.spawncount |
|
|
The number of agents that are to be run inside of each VM |
|
perftest.startport |
|
|
The port that the first agent will bind to. The rest of the agents are assigned ports incremented from this port number |
|
perftest.startports |
|
|
Comma-separated list of start ports. This defines the number of VMs you want to start - for each value represents a VM and its start port it will use |
|
perftest.bindaddress |
|
|
The address your agents will bind to |
|
perftest.pidfile |
|
|
If you want the VM to write its pid in a file, set this to a full path to the pidfile |
|
perftest.startpause |
|
|
The pause (in number of milliseconds) that will occur before the start of each agent within the VM |
|
perftest.cmdline |
|
|
The command line arguments to pass to each agent |
|
rhq.agent.server.bind-address |
|
|
The hostname or IP of the RHQ Server the agents should initially register with |
|
rhq.agent.server.bind-port |
|
|
The RHQ Server port that the agents talk over when initially registering |
|
rhq.* |
|
|
Any rhq.* setting (as defined in a standard agent-configuration.xml) can be set to override the standard agent defaults |
Running a Single VM
The jar artifact built by mvn is standalone and can be run simply by using the command java -jar target/org.rhq.agentspawn-1.0-SNAPSHOT.jar where org.rhq.agentspawn-1.0-SNAPSHOT.jar is the name of the jar built by this mvn module. Run that command without any command line arguments to see its usage help message as well as some help text that shows you what system properties you can set to configure its behavior (like how many internal agents it should spawn, what ports it should listen to, etc). You should also read the AgentSpawn javadoc for more information.
If you only want to run one VM with multiple agents, the above is all you need to run. You simply build the jar and run it, configuring it as you see fit via system properties and command lines arguments. There is even some sample scripts for you to use (one-vm.bat,one-vm.sh). The agent spawn jar does not read or otherwise use the agentspawn.properties file - that .properties file is only used in conjunction with the ant script (see below for more on that).
Running Mulitple VMs
If you want to run multiple VMs (each with multiple agents inside them), you need to next use the ANT script found in the scripts directory. Read the agentspawn.properties and build.xml files found here for details on how to run it and what settings you can set to define your spawned agents' configurations. Execute within the scripts directory the command "ant -projecthelp" for usage help.
AgentSpawn Step-by-Step Basics
Here are some of the basic steps you should perform to run multiple VMs that are running multiple agents per VM:
-
Get an agent distribution .zip file. You can either perform a full RHQ build, an RHQ agent build or get an agent binary zip file from somewhere. Remember the path to that .zip file.
-
"mvn install" inside the rhq/trunk/etc/agentspawn module. If you do not have the code for this, you can check it out of git at git://git.fedorahosted.org/git/rhq/rhq.git
-
Edit the etc/agentspawn/src/script/agentspawn.properties with the settings you want. This file defines where your agent distribution zip file is located (see the first step), how many VMs you want to launch, how many agents per VM to start, etc. This file is commented to help you decipher what each setting is for.
-
Open a command line shell and change to the etc/agentspawn/src/scripts directory.
-
Run "ant copy" to spawn multiple copies of your agent onto your file system.
-
Run "ant start" to start your VMs/agents.
-
When you want to stop your agents, run "ant stop"
-
To clean up your file system and purge your agents, run "ant clean"
Additional Notes
-
When you run multiple agents in a single VM, you will see this in the logs:
[Starting Agent 36166] (rhq.core.pc.measurement.MeasurementManager) - Unable to register MeasurementManagerMBean javax.management.InstanceAlreadyExistsException: rhq.pc:type=MeasurementManager
This is to be expected. In addition, when you import your agents into inventory, all but one of your VM's MeasurementManager service resources will show "red" availability. Again, this is to be expected.
AgentCopy
agentcopy is similiar to agentspawn except it only involves an ANT script. There is no additional Java code used with agentcopy. Agentcopy is used to run multiple agents (just as with agentspawn), but only a single agent ever runs within one VM. The benefit to using agentcopy over agentspawn is that agentcopy produces agents that are installed in the exact manner as a "normal" agent is installed (its files are laid out on the filesystem exactly as a normal agent, and the agents are run exactly as a production agent is expected to run). Assuming you have a machine with a lot of memory and CPU capacity, you could run a good number of agents using this mechanism, with each agent running in exactly the same manner as you would expect production agents to be running. One reason why you would use agentcopy over agentspawn is to test agent auto-update functionality (agentspawn agents cannot be auto-updated).
AgentCopy Setup
You should make sure you perform a full trunk build of RHQ so you have an up-to-date RHQ Agent built in modules/enterprise/agent/target/rhq-agent - it will be used as the template agent that will be copied by the agentcopy.
If you do not want to perform a full build, just ensure you have an agent installation (fully unzipped) that you can point the "agentcopy.agent.dist.dir" setting to. If you opt not to have a full build environment, you still need the ANT script and its supporting files found at http://git.fedorahosted.org/cgit/rhq/rhq.git/tree/etc/agentspawn/src/scripts.
Unlike agentspawn, you do not need to fully checkout and "mvn install" the agentspawn module. There is no additional Java code or jars needed to run agentcopy. Just make sure you have the agentcopy* files.
AgentCopy Settings You Should Know About
These are the settings defined within agentcopy.properties and are used by the ANT script agentcopy.xml.
|
Property name |
Description |
|
agentcopy.agent.dist.dir |
The full path to your exploded agent distribution (not to a zip/tar). If you do not set this at all, agentcopy will download an agent binary update from the server and use that for the agent distro. |
|
agentcopy.agent.spawn.dir |
The full path pointing to the root directory where all your agent's will be copied on your file system |
|
agentcopy.agent.spawn.list |
Comma-separated list of start ports. This defines the number of agents you want to start - for each value represents an agent and its start port it will use |
|
agentcopy.agent.sleep-before-launch-default |
The pause (in number of milliseconds) that will occur before the start of each agent. If you want a specific agent to have its own value, replace "-default" with ".###" where ### is the agent's port. |
|
agentcopy.agent.javaopts-default |
The Java options to pass to each agent. If you want a specific agent to have its own value, replace "-default" with ".###" where ### is the agent's port. |
|
agentcopy.agent.sysprops-default |
The system properties to pass to each agent. If you want a specific agent to have its own value, replace "-default" with ".###" where ### is the agent's port. |
|
agentcopy.agent.cmdlineopts-default |
The command line arguments to pass to each agent |
|
agentcopy.bindaddress |
The address your agents will bind to. If you want a specific agent to have its own value, replace "-default" with ".###" where ### is the agent's port. |
|
agentcopy.server.bind-address |
The hostname or IP of the RHQ Server the agents should initially register with |
|
agentcopy.server.bind-port |
The RHQ Server port that the agents talk over when initially registering |
|
agentcopy.agent.override.###.rhq.* |
Any rhq.* setting (as defined in a standard agent-configuration.xml) can be set to override the standard agent defaults. Only the agent whose start port is ### will get this value. |
AgentCopy Step-by-Step Basics
Here are some of the basic steps you should perform to run multiple agents using agentcopy.
-
Get an agent distribution .zip file. You can either perform a full RHQ build, an RHQ agent build or get an agent binary zip file from somewhere. Unzip that agent somewhere and remember the path to that installation directory. This will be the base agent that all agentcopy agent's will derive from.
-
This step is actually optional. If you already have your server(s) running, you can comment out (i.e. do not set) the property "agentcopy.agent.dist.dir". This will tell agentcopy to download the agent binary update directly from the server and use that as the agent distribution that will be copied for all its agents.
-
-
Edit the etc/agentspawn/src/script/agentcopy.properties with the settings you want. This file defines where your agent distribution directory is located (see the first step), how many agents you want to launch, what their start ports should be, etc. This file is commented to help you decipher what each setting is for.
-
Open a command line shell and change to the etc/agentspawn/src/scripts directory.
-
Run "ant -f agentcopy.xml copy" to spawn multiple copies of your agent onto your file system.
-
Run "ant -f agentcopy.xml start" to start your agents
-
When you want to stop your agents, run "ant -f agentcopy.xml stop" (this does not work on Windows)
-
To clean up your file system and purge your agents, run "ant -f agentcopy.xml clean"