Current Environment (July 31st)

All perf related .sh files referred to below can be found at http://jon01.DOMAIN.COM:8042/perf/ (/home/TEST_USER/httpd/htdocs/perf) and attached to this page.

Prepare Server and Agent build

Process: Need to do an initial check and build, then need to do an update(optionally clean) build and upload

1. Do this on jon01, since that's the nfs server you'll likely to have fewer file system problems. Also it has Apache running already which we will utilize to distribute the files we'll generate

2. Ensure you have Java, SVN and Maven set up in your environment. If necessary, add this to your environment:

   export JAVA_HOME=/qa/tools/opt/jdk1.5.0_15
   export MAVEN_HOME=/qa/tools/opt/maven-2.0.9
   export SUBVERSION_HOME=/qa/tools/opt/subversion-1.4.0
   export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$SUBVERSION_HOME/bin:$PATH
   export LD_LIBRARY_PATH=$SUBVERSION_HOME/lib
   export LANG=en_US.iso88591

3. Make room for the src

   mkdir -p ~/perf/src
   cd ~/perf/src

4. Check out the code. If this is the first time:

   svn co http://svn.rhq-project.org/repos/rhq/trunk/ rhq/trunk
   svn co http://anonsvn.jboss.org/repos/jopr/trunk/ jopr/trunk

   Otherwise, you'll typically want to do something like the following, replacing the revision numbers with ones taken from a working TeamCity build.

   svn update -r 3355 rhq/trunk
   svn update -r 239 jopr/trunk

5. Copy the settings.xml attached to this page to here: ~/.m2/settings.xml

   1. Update rhq.rootDir appropriately to point at rhq/trunk

6. Build RHQ

   1. Need to specify the ojdbc-driver profile, and jon01 repo in settings.xml, to get the oracle jdbc driver.

      cd ~/perf/src/rhq/trunk
      mvn -Pdev,enterprise clean
      mvn -Pdev,enterprise,jon05-oracle,ojdbc-driver install

   2. If you get

      [ERROR] BUILD ERROR
      [INFO] ------------------------------------------------------------------------
      [INFO] An Ant BuildException has occured: The following error occurred while executing this line:
      /home/ccrouch/perf/src/rhq/trunk/modules/core/dbutils/src/main/scripts/dbsetup-build.xml:321: Database schema is not up-to-date. Use -Ddbsetup

      then you're database is out of date you'll eventually need to run mvn -Ddbsetup-upgrade -Pjon05-oracle install, but for right now you can do the following to get RHQ built:

      mvn -Pdev,enterprise clean
      mvn -Pdev,enterprise,jon05-oracle,ojdbc-driver -Ddbsetup-do-not-check-schema install

7. Build Jopr. If this is not the first time you are building you should probably add the 'clean' target before 'install'.

   cd ~/perf/src/jopr/trunk
   mvn -Pdev clean install

8. Distribute Server and Agent

   cd ~/perf/src
   # Download the necessary bits
   wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-server.sh
   wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-agent.sh
   chmod +x *.sh
   ./bundle-server.sh
   ./bundle-agent.sh
   # The Server (dev-container.tar.gz) and Agent (agent.zip) should be available at http://jon01.DOMAIN.COM:8042/perf/

Prepare AgentSpawn build

cd ~/perf/src/rhq/trunk/etc/agentspawn
# build the AgentSpawn src
mvn clean install
cd ~/perf/src
# Download the necessary bits
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-spawn.sh
chmod +x *.sh
./bundle-spawn.sh
# The AgentSpawn distro (agentspawn.tar.gz) should be available at http://jon01.DOMAIN.COM:8042/perf/

Using Server build

Process on each box: Need to do an initial run, then deal with having an existing server in place and running a new one

1. First time through make sure you have

   1. Make sure sshd doesnt timeout and kill your session too quickly

      1. Edit /etc/ssh/sshd_config and update the properties as follows

      2. ClientAliveInterval 300

      3. Then restart sshd: /sbin/service sshd reload

      4. Logout and in again

   2. Java installed (in the PATH and JAVA_HOME set via .bash_profile)

   3. any firewalls turned off (/etc/init.d/iptables stop)

   4. date/time on the box is accurate (need root for this)

      1. if date is more than 1000s (15mins) away from current time then set it approximately e.g. : date -s "Wed Jul 15 10:21:46 EDT 2009"

      2. set date accurately: ntpd -qd

      3. start ntpd server: ntpd

      4. make sure ntpd starts on reboot: chkconfig --level 35 ntpd on

   5. then do

      mkdir -p `hostname -s`/perf

1. Each time through

   cd ~/`hostname -s`/perf
   # Download the necessary bits
   wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-server.sh
   wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-server.sh
   chmod +x *.sh
   # setup the server
   ./unbundle-server.sh
   ./tweak-server.sh
   # startup the server
   ./server/trunk/dev-container/bin/rhq-server.sh start

cd ~/`hostname -s`/perf
cd ./server/trunk/dev-container/bin
./rhq-server.sh stop
sleep 30
cd ~/`hostname -s`/perf
# you may need to do the next step on jon01 if you get NFS errors
rm -rf ./server_old
mv server server_old
# Repeat steps above

Using Agent build

This is a prerequisite for using the AgentSpawn build

1. First time through make sure

   1. All the server specific first time steps above have been done

   2. Ant 1.6.5 installed (in the PATH and ANT_HOME set via .bash_profile)

   3. then

      mkdir -p `hostname -s`/perf

1. Each time through

   cd ~/`hostname -s`/perf
   # Download the necessary bits
   wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-agent.sh
   chmod +x *.sh
   ./unbundle-agent.sh

Using AgentSpawn build

1. First time through make sure the prerequisites for using the Agent above have been done

2. Each time through

   1. Get the bits

      cd ~/`hostname -s`/perf
      # Download the necessary bits
      wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-spawn.sh
      wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-spawn.sh
      chmod +x *.sh
      ./unbundle-spawn.sh

   2. setup for the appropriate agent

      #jon07
      ./tweak-spawn.sh 10 27000 50 10 "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
      #jon08
      ./tweak-spawn.sh 10 28000 50 10 "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
      #jon09
      ./tweak-spawn.sh 10 29000 50 10 "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"

   3. start 'em up

      cd ./spawn/trunk/etc/agentspawn/src/scripts
      # distribute the spawned agents
      ant copy
      # start them up
      ant start

cd ~/`hostname -s`/perf
cd ./spawn/trunk/etc/agentspawn/src/scripts
ant kill
# get rid of any existing ones
ant clean
# Optionally delete the agent and get a new version of that
# Repeat steps above

Quick Re-Start

Here's a quick set of instructions that should work to get you going in the minimum number of copy-n-paste steps. You must have already performed all the build initialization steps mentioned above at least once. But once you have your build environment setup and you are building the perf environment, you can just copy-n-paste the steps below to do the builds quicker.

# BUILD SERVER/AGENT
cd ~/perf/src/rhq/trunk
svn up
cd ~/perf/src/jopr/trunk
svn up
cd ~/perf/src/rhq/trunk
mvn -Pdev,enterprise,jon05-oracle,ojdbc-driver clean install
cd ~/perf/src/jopr/trunk
mvn -Pdev clean install
cd ~/perf/src
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-server.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-agent.sh
chmod +x *.sh
./bundle-server.sh
./bundle-agent.sh
# BUILD AGENT SPAWN
cd ~/perf/src/rhq/trunk/etc/agentspawn
mvn clean install
cd ~/perf/src
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-spawn.sh
chmod +x *.sh
./bundle-spawn.sh

# START NEW SERVER
cd ~/`hostname -s`/perf
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-server.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-server.sh
chmod +x *.sh
./unbundle-server.sh
./tweak-server.sh
./server/trunk/dev-container/bin/rhq-server.sh start

# RUN AGENT SPAWN
cd ~/`hostname -s`/perf
cd ./spawn/trunk/etc/agentspawn/src/scripts
ant stop
sleep 10
ant kill
ant clean
cd ~/`hostname -s`/perf
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-agent.sh
chmod +x *.sh
./unbundle-agent.sh
cd ~/`hostname -s`/perf
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-spawn.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-spawn.sh
chmod +x *.sh
./unbundle-spawn.sh
# !!!!!!!!!!!!!!! START: THIS IS DIFFERENT FOR EACH AGENT BOX !!!!!!!!!!!!!!!
./tweak-spawn.sh.CHANGEME xx xxxxx xx xx "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
# !!!!!!!!!!!!!!! END: THIS IS DIFFERENT FOR EACH AGENT BOX !!!!!!!!!!!!!!!
cd ./spawn/trunk/etc/agentspawn/src/scripts
ant copy
ant start