Hibernate.orgCommunity Documentation
This chapter covers the two types of reports that can be generated from the TCK, an assertion coverage report and the test execution results. The chapter also justifies why the TCK is good indicator of how accurately an implementation conforms to the JSR-303 specification.
The Bean Validation TCK coverage has been measured as follows:
Assertion Breadth Coverage
The Bean Validation TCK provides at least 100% coverage of identified assertions with test cases.
Assertion Breadth Coverage Variance
The coverage of specification sub-sections shows at least a normal distribution (centered around 75%).
Assertion Depth Coverage
The assertion depth coverage has not been measured, as, when an assertion requires more than one testcase, these have been enumerated in an assertion group and so are adequately described by the assertion breadth coverage.
API Signature Coverage
The Bean Validation TCK covers 100% of all API public methods using the Java CTT Sig Test tool.
A specification can be distilled into a collection of assertions that define the behavior of the software. This section introduces the Bean Validation TCK coverage report, which documents the relationship between the assertions that have been identified in the JSR-303 specification document and the tests in the TCK test suite.
The structure of this report is controlled by the assertion document, so we'll start there.
The Bean Validation TCK developers have analyzed the JSR-303 specification document and identified the assertions that are present in each chapter. Here's an example of one such assertion found in section 2.1:
Every constraint annotation must define a message element of type String
The assertions are listed in the XML file
tck-audit.xml in the Bean Validation TCK
distribution. Each assertion is identified by the section of the
specification document in which it resides and assigned a unique
paragraph identifier to narrow down the location of the assertion
further. To continue with the example, the assertion shown above is
listed in the tck-audit.xml file using this XML fragment:
<section id="2.1.1" title="Constraint definition properties">
...
<assertion id="c">
<text>Every constraint annotation must define a message element of type String</text>
</assertion>
...
</section>
The strategy of the Bean Validation TCK is to write a test which
validates this assertion when run against an implementation. A test case
(a method annotated with @Test in an
@Artifact class) is correlated with an assertion
using the
@org.jboss.test.audit.annotations.SpecAssertion
annotation as follows:
@Test
@SpecAssertion(section = "2.1.1", id = "c")
public void testConstraintDefinitionWithoutMessageParameter() {
try {
Validator validator = TestUtil.getValidatorUnderTest();
validator.validate( new DummyEntityNoMessage() );
fail( "The used constraint does not define a message parameter. The validation should have failed." );
}
catch ( ConstraintDefinitionException e ) {
// success
}
To help evaluate the distribution of coverage for these assertions, the TCK provides a detailed coverage report. This report is also useful to help implementors match tests with the language in the specification that supports the behavior being tested.
The coverage report is an HTML report generated as part of the TCK project build. Specifically, it is generated by an annotation processor that attaches to the compilation of the classes in the TCK test suite, another tool from the JBoss Test Utils project. You can enable this report by setting the commandline property tck-audit to true when running the Maven 2 build in the tck directory.
mvn clean install -Dtck-audit=true
You must run clean first because the annotation processor performs it's work when the test class is being compiled. If compilation is unnecessary, then the assertions referenced in that class will not be discovered.
The report is written to the file target/coverage.html in the same project. The report has five sections:
Chapter Summary - List the chapters (that contain assertions) in the specification document along with total assertions, tests and coverage percentage.
Section Summary - Lists the sections (that contain assertions) in the specification document along with total assertions, tests and coverage percentage.
Coverage Detail - Each assertion and the test that covers it, if any.
Unmatched Tests - A list of tests for which there is no matching assertion (useful during TCK development).
Unversioned Tests - A list of
tests for which there is no @SpecVersion
annotation on the test class (useful during TCK development).
The coverage report is color coded to indicate the status of an assertion, or group of assertions. The status codes are as follows:
Covered - a test exists for this assertion
Not covered - no test exists for this assertion
Problematic - a test exists, but is currently disabled. For example, this may be because the test is under development
Untestable - the assertion has been deemed untestable, a note, explaining why, is normally provided
For reasons provided in the tck-audit.xml
document and presented in the coverage report, some assertions are not
testable.
The coverage report does not give any indication as to whether the tests are passing. That's where the TestNG reports come in.
As you by now, the Bean Validation TCK test suite is really just a TestNG test suite. That means an execution of the Bean Validation TCK test suite produces all the same reports that TestNG produces. This section will go over those reports and show you were to go to find each of them.
When the Bean Validation TCK test suite is executed during the Maven 2 test phase of the TCK runner project, TestNG is invoked indirectly through the Maven Surefire plugin. Surefire is a test execution abstraction layer capable of executing a mix of tests written for JUnit, TestNG, and other supported test frameworks.
Why is this relevant? It means two things. First, it means that you are going to get a summary of the test run on the commandline. Here's the output generated when the tests are run using standalone mode.
------------------------------------------------------- T E S T S ------------------------------------------------------- Running TestSuite Tests run: 237, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.062 sec Results : Tests run: 237, Failures: 0, Errors: 0, Skipped: 0
The number of tests executed, the execution time, and the output will differ when you run the tests using in-container mode as the Bean Validation TCK requires.
If the Maven reporting plugin that compliments Surefire is configured properly, Maven will also generate a generic HTML test result report. That report is written to the file test-report.html in the target/surefire-reports directory of the TCK runner project. It shows how many tests were run, how many failed and the success rate of the test run.
The one drawback of the Maven Surefire report plugin is that it buffers the test failures and puts them in the HTML report rather than outputting them to the commandline. If you are running the test suite to determine if there are any failures, it may be more useful to get this information in the foreground. You can prevent the failures from being redirected to the report using the following commandline switch:
mvn test -Dsurefire.useFile=false
The information that the Surefire provides is fairly basic and the detail pales in comparison to what the native TestNG reports provide.
TestNG produces several HTML reports for a given test run. All the reports can be found in the target/surefire-reports directory in the TCK runner project. Below is a list of the three types of reports:
Test Summary Report
Test Suite Detail Report
Emailable Report
The first report, the test summary report is written to the file index.html. It produces the same information as the generic Surefire report.The summary report links to the test suite detail report, which has a wealth of information. It shows a complete list of test groups along with the classes in each group, which groups were included and excluded, and any exceptions that were raised, whether from a passed or failed test. A partial view of the test suite detail report is shown below.
The test suite detail report is very useful, but it borderlines on complex. As an alternative, you can have a look at the emailable report, which is a single HTML document that shows much of the same information as the test suite detail report in a more compact layout.
Copyright © 2009 - 2011 Red Hat, Inc.