Chapter 13. Support and Patch Management

13.1. Introduction

Customer requested fix can be made in a cumulative patch, one-off patch, or both. Normally, most of the bug fixes are included into the cumulative patch, so there is no need to create one-off patches. One-off patches can still be created for the emergency production and security issues. However, it's strongly advised to merge all the bug fixes into the cumulative patches.

13.1.1. Cumulative Patch

  1. Cumulative patches contain only customer requested bug fixes and security patches
  2. Cumulative patches are additive, i.e. JBossAS-4.0.4.GA_CP02 includes all of the fixes from JBossAS-4.0.4.GA_CP01
  3. Released on 2nd Tuesday of each month. Code freeze for all cumulative patch branches is on 1st day of each month.
  4. Separate cumulative patch branch is created for every supported product release, i.e.
  5. All customers get access to available cumulative patches
  6. All the fixes included into a cumulative patch need to be included into the next maintenance release
  7. It's required to have test case(s) for each fix included into the cumulative patch.
  8. Cumulative patches are tested by QA so all fixes are tested in combination

13.1.2. One-off Patch

  1. Isolated branch is created for every patch, i.e.
  2. Every one-off patch is tested by QA
  3. It's advised to minimize number of one-off patches, and virtually all one-off patches are supposed to be merged into the cumulative patches

13.2. Support Workflow

  1. Customer requests a fix.
  2. If the issue is fixed in a newer GA/SP version of the product, Support verifies if upgrade is an option
  3. Support engineer raises issue in the patch project (ASPATCH for the application server)
  4. If support engineer has concerns, they mark the patch as "Request Triage". Patch is assigned to patch-triage team for resolution
  5. If issues are resolved, the patch continues through the CP process. Support notifies customer when the next cumulative patch release is scheduled for
  6. Otherwise, the patch is done as a one-off. For instance, if customer requires an emergency fix and can’t wait until the next cumulative patch release, a one-off patch is created. The fix is also merged into the upcoming cumulative patch
  7. After Development, QA, and JBN are done, Support provides patch download link to the customer
  8. If customer has a problem with the cumulative patch release, create a bug against it. For the app server, create a bug under the ASPATCH project. Make sure that every ASPATCH issue is public.

13.3. Cumulative Patch Process

13.3.1. Development Phase

  1. Create an issue in JIRA with a type "Task" under the appropriate PATCH project. For the App Server it's ASPATCH project. Such JIRA issues need to be public.
  2. Set Fix Version field value appropriately. For instance, JBossAS-4.0.4.GA_CP03. It's easy to find out which Cumulative Patch release is next by looking at the roadmap in JIRA.
  3. Link the issue with the appropriate bug issue using "incorporates".
  4. Click the "Start Progress" link

    Check out appropriate cumulative patch branch, for instance for the AS4.0.2 release

    			     svn checkout 
  5. Implement the fix and check it in.

    If a bug is in app server component such as Web Services or Hibernate, a cumulative patch branch must be created off of the component release that was included into the app server. For instance, for Hibernate 3.1.3:

    1. Create Hibernate_3_1_3_CP branch
    2. Implement or merge the fix into the Hibernate_3_1_3_CP branch
    3. Tag Hibernate_3_1_3_CP branch as Hibernate_3_1_3_CP0x where x is a number to be incremented with every fix.
    4. Follow normal release process for the component but treat the release as internal. For instance, you should end up with Hibernate 3.1.3_CP01 component in the repository.
    5. Update App Server's build-thirdparty.xml with the appropriate version of the Hibernate CP release, i.e. 3.1.3_CP01

  6. If the issue being resolved does not have a test case, it should be added to the testsuite.
  7. If a regression is introduced, CruiseControl will notify.
  8. Click "Resolve". In the "Patch Instructions" field enter installation instructions on how to apply the patch.

    Use the Support Patch Template as a basis for these instructions.

13.3.2. QA Phase

  1. Within 48 hours after the cumulative branch code freeze, QA will assign the release task and set a due date for the QA process to be complete.
  2. QA will tag cumulative patch branch upon code freeze. For instance,

    			     svn copy 
  3. QA will run the testsuite to verify that the cumulative patch produces no regressions. If the tests fail, QA will submit a blocker issue and it will be assigned to the developer for root cause analysis.
  4. See How To QA a Cumulative Patch..
  5. QA will mark the issue as "QA Tests Passed" if the tests passed.
  6. QA will assemble cumulative patch package for the JBN
  7. QA will test deployment of the cumulative patch with JBN team

13.3.3. JBN Phase

  1. The patch will be deployed by the JBossNetwork team.

13.4. One-Off Patch Process

13.4.1. Development Phase

  1. Create an issue in JIRA with a type "Support Patch" under the appropriate project. For the App Server, it's JBAS project.
  2. Be sure to make the issue either "Customers Only" or "JBoss Internal"
  3. Click the "Start Progress" link to enter the CVS or SVN branch information.

    Enter the branch for the patch. Using that JIRA ID and the tag for the release being patched, create a branch for the patch using the format ReleaseTag_JiraID.

    For instance, if the patch issue ID is JBAS-1234 for JBoss 4.0.3, the patch's branch should be JBoss_4_0_3_JBAS-1234.

    				 # svn
    				 svn copy
    				 svn checkout jbas-1234_local_dir
    				 # cvs
    				 cvs rtag -r JBoss_4_0_3 -b JBoss_4_0_3_JBAS-1234 jboss-4.0.x
    				 cvs co -r JBoss_4_0_3_JBAS-1234 jboss-4.0.x

    If the project being patched is not in JBoss hosted CVS, attach the source diff to this case.

  4. If the issue being resolved does not have a test case, it should be added to the testsuite.
  5. When committing the patch to the branch, be sure to include the JIRA ID of the patch in the commit comment.
  6. Build the patched jars from the above branch and attach them to the JIRA issue.

  7. Merge the patch into the appropriate cumulative patch branch(es) by following steps described in the cumulative patch section
  8. Click "Hand Over to QA". In the "Patch Instructions" field enter installation instructions on how to apply the patch to an existing installation.

    Use the Support Patch Template as a basis for these instructions.

  9. Mark the issue as "Pass to QA".

13.4.2. QA Phase

  1. Within 48 hours, QA will assign the task and set a due date for the QA process to be complete. If the due date is more than 3 days after the development is complete, QA will comment on the case and let the customer know when the patch will be ready.
  2. QA will run the testsuite to verify that the patch produces no regressions. If the tests fail, QA will mark the issue as "QA Tests Failed" and it will be reassigned to the developer.
  3. See How To QA a One-Off Support Patch..
  4. QA will mark the issue as "QA Tests Passed" if the tests passed.
  5. If tests pass, attach JARS to Support Portal case. Download jars and verify that MD5 sums match originals.

13.4.3. JBN Phase

  1. The patch will be deployed by the JBossNetwork team. This should include linking the issue to the JBN patch.

13.5. Support Patch Instructions Template

Below is the skeleton of the directions that should be entered in the "Patch Instructions" field in JIRA.

        [Not needed for the Cumulative Patch tasks]
        [Not needed for the Cumulative Patch tasks]
        JBoss Application Server
        [Not needed for the Cumulative Patch tasks]
        [What problem this patch fixes.]
        ex: "A NullPointerException is no longer thrown when the password field is left blank."
        [Detailed explanation of the problem.]
        ex: "Prior to this fix, blah happened. With this fix blah will happen instead.  This is because blah blah blah.
        [How a user should manually install this patch.]
        ex: "Rename %JBOSS_HOME%/lib/someJar.jar to "someJar.replacedBy.JBAS-xxxx.jar.old"
        Copy the new someJar.jar to %JBOSS_HOME%/lib/"
        [known usages and known combinations that don't work]
        ex: "portal 2.x in a given jems bundle does not work with this change"
        [list any patches this patch is dependent on. Not needed for the Cumulative Patch tasks]
        ex: 4.0.2-SP2
        [list any patches this patch supersedes]
        ex: JBAS-1450
        [list any patches this patch is superseded by. Not needed for the Cumulative Patch tasks]
        ex: 4.0.2-SP3
        [author of this patch]
        [date these instructions were written]

13.6. How To QA a One-Off Support Patch

# Howto test a patch in the QA lab

# create a directory which corresponds to the JIRA id for the patch
mkdir JBAS-1234
cd JBAS-1234

#set your JAVA_HOME correctly 
#if jboss 3.x, use jdk 1.3
#export JAVA_HOME=/opt/jdk1.3.1_13/

#if jboss 4.x use jdk 1.4
export JAVA_HOME=/opt/j2sdk1.4.2_09/

#Make sure you are testing on a 32-bit box

# get the source distribution of the *targetted* version of jboss
tar xvzf /opt/src/jboss-4.0.2-src.tar.gz

# build Jboss & save the original output
cd jboss-4.0.2-src/build; sh
cp -r output output-orig
cd ../..

# download the patched binaries into a binaries directory
mkdir binaries; cd binaries; #download them locally & upload via scp (if in qa lab)

#TODO: look at using wget to retrieve patches

# capture the md5 of each jar and add it as comment to jira
md5sum *.jar

#use 'find' to locate where in the output each jar is, in
#order to create the right mirror of the install distro

find ../jboss-4.0.2-src/build/output -name jboss.jar

# create a mirror of the install distro using the patched jars
#for each configuration 'find' listed
mkdir -p jboss-4.0.2/server/all/lib
cp jboss.jar jboss-4.0.2/server/all/lib

# copy the jars to the output, verifying each jar copies correctly
# NOTE: make sure to replace all the jars in the source tree
# not just the ones under build/output since the client classpath
# is affected by the jars under /thirdparty and */output/lib.
cp -ivr jboss-4.0.2 ../jboss-4.0.2-src/build/output

# save the patched version for later (to switch back & forth)
cd ../jboss-4.0.2-src/build
cp -r output output-patched

# run the testsuite against the patched version (let node0 default to localhost)
cd ../testsuite
sh -Dnode1=$MYTESTIP_1 tests

# tests will take 1-2 hours, to save time, verify pass rate every 15 min
# if you see several failures, make sure no one else is on
ps --columns 1000 -ef | grep run.jar

# once tests complete, save the text report and take a look at it
cp output/reports/text/TESTS-Testsuite.txt ../../TESTS-patched.txt

# verify any failures also fail on the unpatched version
# upon any failure contact QA Lead for help
cd ../build;
mv output output-patched
cp -r output-orig output

#find out which mode the failing tests were using and
#start the server using the particular configuration
#the example below starts the 'all' mode
cd output/jboss-4.0.2/
./bin/ -c all

#different shell
cd jboss-4.0.2-src/testsuite
sh -Dtest=org.jboss.tests.the.FailingTest one-test

# if failure only occurs on patched version, reject the patch
# Attach necessary information to JIRA issue
# ie, testuite/output/reports/TESTS-org.jboss.test.the.FailingTest.txt

#  When failing the patch notify developer that patched client jars 
#  were note used, ie, add this as a comment:
#  NOTE: Patched client jars were not used to validate compatibility.

# Upon failure tar up the issue directory and copy to /home/issues
tar cvzf JIRA-1234.tar.gz JIRA-1234
cp JIRA-1234.tar.gz /home/issues

#note path to tar.gz on JIRA issue

13.7. How To QA a Cumulative Patch

To be filled in by QA.