The RHQ CLI provides an excellent way for users to automate tasks as well as extend the functionality of RHQ. In addition to these capabilities, the CLI serves as an excellent end-to-end, integration testing tool for RHQ. As the CLI was developed, a customized test harness was built along side it in order to satisfy three goals:
- Test CLI-specific functionality (e.g., verify that particular objects are bound into the script context)
- Test the remote APIs used by the CLI
- Document CLI usage and remote APIs used by the CLI
This document provides detailed information on writing and running CLI tests. It will also consider various potential improvements for testing of and with the CLI.
One or more test failures will cause the build to fail as is the case when running tests via the surefire plugin. Test results are reported using the standard TestNG reports. There is no integration yet with the SureFire reports.
If you look at some of the existing test scripts, you will notice that they are organized in the following format,
The majority of tests require a running server and a logged in user. The latter is handled by the first line. exceuteAllTests is a helper function that is made available to scripts by the CLI. It looks through the script and executes any function that follows the well-known JUnit naming convention of testXXX. executeAllTests will continue executing all test functions even if there are failures. The error message for each failure is captured and included in the exception thrown by executeAllTests. The errror messages are formatted to make it easy to see which functions failed as well as to provide line numbers of where the failures occurred.
If you want to execute a subset of the test functions in a script you can use executeTests instead of executeAllTests. It expects a single parameter that is array of the function names to include in the test run. The example below demonstrates its usage,
Alternatively, if you want to explicitly exclude functions from the test run, you can use the implicit variable skippedTests as follows,
You may have noted use of the Assert object in the test script. The Assert object is another implicit variable provided by the CLI that offers most if not all of the assertXXX methods found in the org.testng.Assert class.
The biggest problem right is that CLI tests are not run as part of a continuous integration build due to the required set up that includes a running RHQ server and at least one registered agent. Work is under way to get the CLI tests running in CI build.
Each test script file is added as a separate test in the generated TestNG test suite; Test functions however are not treated separately as is the case with a test method of a regular TestNG test class. So if your test script has five functions, three of which fail, the TestNG report will only show the one failure. The output in the test failure will specify which functions failed, but it would be nice to see each failed function reported separately.