SeamFramework.orgCommunity Documentation
This chapter explains the purpose of the test harness and describes its key features.
The JBoss Test Harness is a testing framework based on TestNG that provides a series of extensions that allow runtime packaging and deployment of Java EE artifacts (EAR or WAR) for in-container testing. It's important to note that the JBoss Test Harness has no relation with, or dependency on, the JBoss Application Server (JBoss AS).
You'll often see the term in-container used in this reference guide. This term refers to running the test suite in any of the aforementioned environments, whilst standalone refers to running the tests outside the container via an implementation-specific standalone bootstrap. The standalone mode only runs those tests which the CDI RI can run without deployment in a Java EE container.
The last thing Java developers want is yet another testing framework to make their life more complicated. That's why the JBoss Test Harness is built entirely upon TestNG. TestNG is one of two prominent test frameworks for Java (the other being JUnit). Furthermore, what developers want is a good integration with their Integrated Development Environment (IDE). These days, if a tool doesn't have an IDE plugin, then it won't get the attention it deserves. TestNG plugins are available for all major IDEs and build tools (Ant and Maven 2). Again, a motivating factor for extending TestNG.
Because it leverages the existing TestNG ecosystem, there is no need for a special test launcher for the JBoss Test Harness. You simply use the IDE or build tool of your choice (so long as it has TestNG support). You also get reporting and debugging for free (various reporting plugins are provided for TestNG).
You can read more about TestNG at testng.org.
The JBoss Test Harness supports the following features:
Test activation via any method supported by the TestNG configuration descriptor (package, group, class)
Exclusion of in-container tests in standalone mode
Exclusion of individual tests labeled as under investigation
Integration with any TestNG plugin (Eclipse, IntelliJ, Ant, Maven)
Automated reporting capability as provided by TestNG
Standalone and in-container test mode
Container pluggability
Declarative packaging of additional resources and classes in artifact
Declarative deployment exception trapping
Artifact dumping for failure and packaging analysis
A test is designated by a method annotated with
@org.testng.annotations.Test
in a class which extends
org.jboss.testharness.AbstractTest
and is annotated with
@org.jboss.testharness.impl.packaging.Artifact
.
Test suites may often choose to extend AbstractTest
and require tests to extend that base class. In fact, both the CDI TCK
and the Bean Validation TCK provide base classes that extend
AbstractTest
to provide functionality specific to
the needs of the TCK.
The
@Test
annotation is provided by TestNG, the
@Artifact
annotation is provided by the JBoss Test Harness and the
AbstractTest
is part of the JBoss Test Harness. There is a one-to-one mapping between a
TestNG test class and an artifact. The packaging type is defined by the
@org.jboss.testharness.impl.packaging.Packaging
annotation on the test class, defaulting to a WAR if not specified.
Prior to executing the tests for a given class, the JBoss Test Harness
packages the class as a deployable artifact (EAR or WAR), along with any
extra resources specified, and deploys the artifact to the container. The
harness provides test execution and result reporting via HTTP
communication to a simple Servlet using a thin layer over the TestNG test
launcher. The test harness can also catch and enforce expected deployment
exceptions. This setup and tear down activity is provided by the super
class
org.jboss.testharness.AbstractTest
,
which all test classes must extend (directly or indirectly).
If the annotation
@org.jboss.testharness.impl.packaging.IntegrationTest
is not present on the test class, then it means the test class can be
executed in standalone mode. In standalone mode, the deployable artifact
is assembled on the local classpath and the tests execute in the same JVM
as the launcher, just as though it were a regular TestNG test case. The
standalone mode is provided for convenience and efficiency, allowing you
the speed of mock-based testing and the confidence of an in-container test,
using the same test objects and tests.
The basic procedure of an in-container test is as follows. The JBoss
Test Harness produces a deployable artifact from an
@Artifact
test class and any declared dependent classes, descriptors or other
resources. Then it deploys the artifact to the container using the
Containers
SPI, negotiates with the container to execute the test and return the
result and, finally, undeploys the artifact. TestNG collects the results
of all the tests run in the typical way and produces a report.
The question is, how does the JBoss Test Harness negotiate with the container to execute the test when TestNG is being invoked locally? Technially the mechanism is pluggable, but JBoss Test Harness provides a default implementation that uses HTTP communication that you will likely use. Here's how the default implementation works.
The artifact generator bundles and registers (in the web.xml
descriptor) an
HttpServlet
,
org.jboss.testharness.impl.runner.servlet.ServletTestRunner
,
that responds to test execution GET requests. TestNG running on the
client side delegates to a test launcher (more on that in a moment)
which originates these text execution requests to transfer control to
the container JVM. The name of the test method to be executed is
specified in a request query parameter named
methodName
.
When the test execution request is received, the servlet delegates to
an instance of
org.jboss.testharness.impl.runner.TestRunner
,
passing it the name of the test method.
TestRunner
reads the name of the test class from the resource
META-INF/jboss-test-harness.properties, which is bundled in the
artifact by the artifact generator. It then combines the class name
and the method name to produce a TestNG test suite and runs the suite
(within the context of the container).
TestNG returns the results of the run as an
ITestResult
object.
ServletTestRunner
translates this object into a
org.jboss.testharness.api.TestResult
and passes it back to the test launcher on the client side by encoding
the translated object into the response. The object gets encoded as
either html or a serialized object, depending on the value of the
outputMode
request parameter that was passed to the servlet. Once the result has
been transfered to the client-side TestNG, TestNG wraps up the run of
the test as though it had been executed in the same JVM.
There's one piece missing. How does TestNG on the client side know to
submit a request to the
ServletTestRunner
servlet to get TestNG to execute the test in the container JVM? That's
the role of the test launcher.
The test launcher is the API that allows test suite to launch the test
in a pluggable fashion.
AbstractTest
,
the super class of
AbtractJSR299Test
,
implements
IHookable
,
a TestNG interface which allows the execution of the test method to
be intercepted. Using that mechanism, AbstractTest
delegates execution
of the test method (a method annotated with
@Test
in an
@Artifact
class) to an implementation of
org.jboss.testharness.api.TestLauncher
if the tests are being executed in-container. As you might anticipate,
the implementation is specified using a property with the same name as
the interface in a META-INF/jboss-test-launcher.properties resource.
The JBoss Test Harness provides a default implementation,
org.jboss.testharness.impl.runner.servlet.ServletTestLauncher
,
that hooks into the HTTP communication infrastructure described
above. It invokes the
ServletTestRunner
servlet for each method annotated with
@Test
in the
@Artifact
that is not otherwise disabled.
If you wish to implement the runner yourself, you must return a
TestResult
as a result of executing the method in
the container. You must also ensure that any exception which occurs
during deployment is wrapped as a
org.jboss.testharness.api.DeploymentException
, and
that any communication problem is rethrown as an
IOException
. The deployment exception may be
transformed by an implementation of the
org.jboss.testharness.api.DeploymentExceptionTransformer
interface, which is specified using the
org.jboss.testharness.container.deploymentExceptionTransformer
property. The default implementation passes on the original
exception unchanged. The implementation for JBoss AS used with the CDI
TCK, on the other hand, deciphers the exception thrown by the JBoss
deployer and converts it to one of the catagory exceptions defined in
the CDI TCK API.
So in short, JBoss Test Harness takes care of all the interfaces you
need to execute tests in-container except for the implementation of
the Containers
SPI. That is, unless you are
deploying to one of the containers supported by the JBoss Test Harness
(TODO we need a table showing supported containers).