Hibernate.orgCommunity Documentation
This chapter covers the two types of reports that can be generated from the TCK, an assertion coverage report and the test execution results.
A specification can be distilled into a collection of assertions that define the behavior of the software. This section introduces the Bean Validation TCK coverage report, which documents the relationship between the assertions that have been identified in the JSR 349 specification document and the tests in the TCK test suite.
The structure of this report is controlled by the assertion document, so we'll start there.
The Bean Validation TCK developers have analyzed the JSR 349 specification document and identified the assertions that are present in each chapter. Here's an example of one such assertion found in section 2.1: “Every constraint annotation must define a message element of type String”
The assertions are listed in the XML file
tck-audit.xml in the Bean Validation TCK
distribution. Each assertion is identified by the section of the
specification document in which it resides and assigned a unique
paragraph identifier to narrow down the location of the assertion
further. To continue with the example, the assertion shown above is
listed in the tck-audit.xml file using this XML
fragment:
<section id="2.1.1" title="Constraint definition properties">
...
<assertion id="c">
<text>Every constraint annotation must define a message element of type String</text>
</assertion>
...
</section>
The strategy of the Bean Validation TCK is to write a test which
validates this assertion when run against an implementation. A test case
(a method annotated with @Test) is correlated with an
assertion using the
@org.jboss.test.audit.annotations.SpecAssertion
annotation as follows:
@Test
@SpecAssertion(section = "2.1.1", id = "c")
public void testConstraintDefinitionWithoutMessageParameter() {
try {
Validator validator = TestUtil.getValidatorUnderTest();
validator.validate( new DummyEntityNoMessage() );
fail( "The used constraint does not define a message parameter. The validation should have failed." );
}
catch ( ConstraintDefinitionException e ) {
// success
}
To help evaluate the distribution of coverage for these assertions, the TCK provides a detailed coverage report. This report is also useful to help implementors match tests with the language in the specification that supports the behavior being tested.
The coverage report is an HTML report generated as part of the TCK project build. Specifically, it is generated by an annotation processor that attaches to the compilation of the classes in the TCK test suite.
You can find the source code for this processor in the GitHub repository https://github.com/jboss/jboss-test-audit
The report is written to the file
target/coverage-report/coverage-beanvalidation.html.
The report itself has five sections:
Chapter Summary - List the chapters (that contain assertions) in the specification document along with total assertions, tests and coverage percentage.
Section Summary - Lists the sections (that contain assertions) in the specification document along with total assertions, tests and coverage percentage.
Coverage Detail - Each assertion and the test that covers it, if any.
Unmatched Tests - A list of tests for which there is no matching assertion (useful during TCK development).
Unversioned Tests - A list of
tests for which there is no @SpecVersion
annotation on the test class (useful during TCK development).
The coverage report is color coded to indicate the status of an assertion, or group of assertions. The status codes are as follows:
Covered - a test exists for this assertion
Not covered - no test exists for this assertion
Unimplemented - a test exists, but is unimplemented
Untestable - the assertion has been deemed untestable, a note, explaining why, is normally provided
For reasons provided in the tck-audit.xml
document and presented in the coverage report, some assertions are not
testable.
The coverage report does not give any indication as to whether the tests are passing. That's where the TestNG reports come in.
As you by now know, the Bean Validation TCK test suite is really just a TestNG test suite. That means an execution of the Bean Validation TCK test suite produces all the same reports that TestNG produces. This section will go over those reports and show you were to go to find each of them.
When the Bean Validation TCK test suite is executed during the Maven 2 test phase of the TCK runner project, TestNG is invoked indirectly through the Maven Surefire plugin. Surefire is a test execution abstraction layer capable of executing a mix of tests written for JUnit, TestNG, and other supported test frameworks.
Why is this relevant? It means two things. First, it means that you are going to get a summary of the test run on the commandline. Here's the output generated when the tests are run using standalone mode.
------------------------------------------------------- T E S T S ------------------------------------------------------- Running TestSuite Tests run: 697, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.413 sec Results : Tests run: 697, Failures: 0, Errors: 0, Skipped: 0
The number of tests executed, the execution time, and the output will differ when you run the tests using in-container mode as the Bean Validation TCK requires.
If the Maven reporting plugin that compliments Surefire is
configured properly, Maven will also generate a generic HTML test result
report. That report is written to the file test-report.html in the
target/surefire-reports directory of the TCK runner
project. It shows how many tests were run, how many failed and the
success rate of the test run.
Copyright © 2009 - 2014 Red Hat, Inc.