Introduction

The RHQ CLI provides an excellent way for users to automate tasks as well as extend the functionality of RHQ. In addition to these capabilities, the CLI serves as an excellent end-to-end, integration testing tool for RHQ. As the CLI was developed, a customized test harness was built along side it in order to satisfy three goals:

1. Test CLI-specific functionality (e.g., verify that particular objects are bound into the script context)

2. Test the remote APIs used by the CLI

3. Document CLI usage and remote APIs used by the CLI

This document provides detailed information on writing and running CLI tests. It will also consider various potential improvements for testing of and with the CLI.

Running Tests

CLI integration tests live in the scripts modules located at rhq/modules/enterprise/remoting/scripts. All of the tests are written as CLI scripts in JavaScript which is currently the only scripting language supported by the CLI. The test scripts reside under the scripts/src/test/script directory. The tests are executed as a customized TestNG suite that can be run from maven using a command such as mvn test. The antrun maven plugin is used to execute an instance of the class ScriptTestRunner which also lives in the scripts module. ScriptTestRunner recursively scans the directory tree containing the test scripts and generates a TestNG test suite based on its findings. Any script that has an extension of .js will be included in the test suite. Alternatively if you only want to run a single script you can specify the test system property as follows,

$ mvn test -Dtest=test_alerts.js

One or more test failures will cause the build to fail as is the case when running tests via the surefire plugin. Test results are reported using the standard TestNG reports. There is no integration yet with the SureFire reports.

Writing Tests

If you look at some of the existing test scripts, you will notice that they are organized in the following format,

rhq.login('rhqadmin', 'rhqadmin');

executeAllTests();

rhq.logout();

function testAlerts() {
    var alert = // ...
    // ...
    Assert.assertNotNull(alert);
}

function testOperations() {
    var operation = // ...
    // ...
    Assert.assertEquals(operation, expectedOp);
}

The majority of tests require a running server and a logged in user. The latter is handled by the first line. exceuteAllTests is a helper function that is made available to scripts by the CLI. It looks through the script and executes any function that follows the well-known JUnit naming convention of testXXX. executeAllTests will continue executing all test functions even if there are failures. The error message for each failure is captured and included in the exception thrown by executeAllTests. The errror messages are formatted to make it easy to see which functions failed as well as to provide line numbers of where the failures occurred.

If you want to execute a subset of the test functions in a script you can use executeTests instead of executeAllTests. It expects a single parameter that is array of the function names to include in the test run. The example below demonstrates its usage,

executeTests(['testX', 'testY', 'testZ']);

Alternatively, if you want to explicitly exclude functions from the test run, you can use the implicit variable skippedTests as follows,

skippedTests.push('testX');
executeAllTests();

You may have noted use of the Assert object in the test script. The Assert object is another implicit variable provided by the CLI that offers most if not all of the assertXXX methods found in the org.testng.Assert class.

Future Improvements

The biggest problem right is that CLI tests are not run as part of a continuous integration build due to the required set up that includes a running RHQ server and at least one registered agent. Work is under way to get the CLI tests running in CI build.

Each test script file is added as a separate test in the generated TestNG test suite; Test functions however are not treated separately as is the case with a test method of a regular TestNG test class. So if your test script has five functions, three of which fail, the TestNG report will only show the one failure. The output in the test failure will specify which functions failed, but it would be nice to see each failed function reported separately.

Currently the CLI supports only JavaScript, but other languages might be supported in the future. Note that I am referring to scripting languages that a compatible script engine for running on the JVM. Some of the testing utilities described in this document are implemented in JavaScript which would make them harder to use across different languages. Ideally, these utilities should be reusable across languages so that we do not have to re-implement them for each language we want to support and use. I believe that we can solve this problem as well as the problem of reporting failed functions by porting most of this functionality to ScriptTestRunner.

By pushing the JavaScript test utility into the Java code base, it can be more easily reused across different scripting languages should we choose to support additional language bindings in the future. For reporting failed test functions, the test runner code can be updated to use a TestNG data provider. The test method that executes the CLI script would be passed the name of the function to execute. This approach will enable TestNG to report on failures of individual functions. The one caveat is that the script engine must implement Invocable so that we can execute individual functions defined in the script. The JavaScript engine that we support does implement Invocable . For a script engine that does not implement that interface, we would resort to executing the entire script as a single test.