Hibernate.orgCommunity Documentation

TCK Reference Guide for JSR 349: Bean Validation


This guide describes how to download, install, configure, and run the Technology Compatibility Kit (TCK) used to verify the compatibility of an implementation of JSR 349: Bean Validation 1.1.

The Bean Validation TCK is built atop Arquillian, a portable and configurable automated test suite for authoring unit and integration tests in a Java EE environment.

The Bean Validation TCK is provided under the Apache Public License 2.0.

If you are running the Bean Validation TCK for the first time, read Chapter 1, Introduction completely for the necessary background information about the TCK. Once you have reviewed that material, perform the steps outlined in the remaining chapters.

  • Chapter 1, Introduction gives an overview of the principles that apply generally to all Technology Compatibility Kits (TCKs), outlines the appeals process and describes the Bean Validation TCK architecture and components.

  • Chapter 2, Appeals Process explains the process to be followed by an implementor should they wish to challenge any test in the TCK.

  • Chapter 3, Installation explains where to obtain the required software for the Bean Validation TCK and how to install it.

  • Chapter 4, Reports explains the test reports that are generated by the TCK test suite and introduces the TCK audit report as a tool for measuring the completeness of the TCK in testing the JSR 349 specification and in understanding how test cases relate to the specification.

  • Chapter 5, Running the TCK test suite details the configuration of the test harness and documents how to create a TCK runner for executing the TCK test suite, either in standalone or container mode.

  • Chapter 6, Running the Signature Test finally documents how to use the SigTest tool for ensuring compatibility of types provided in the package javax.validation.* with the official API signature defined by the specification.

This chapter explains the purpose of a TCK and identifies the foundation elements of the Bean Validation TCK.

The goal of any specification is to eliminate portability problems so long as the program which uses the implementation also conforms to the rules laid out in the specification.

Executing the TCK is a form of compatibility testing. It's important to understand that compatibility testing is distinctly different from product testing. The TCK is not concerned with robustness, performance or ease of use, and therefore cannot vouch for how well an implementation meets these criteria. What a TCK can do is to ensure the exactness of an implementation as it relates to the specification.

Compatibility testing of any feature relies on both a complete specification and a complete reference implementation. The reference implementation demonstrates how each test can be passed and provides additional context to the implementor during development for the corresponding assertion.

The Bean Validation TCK is designed as a portable, configurable and automated test suite for verifying the compatibility of an implementation of JSR 349. The test suite is built atop TestNG and provides a series of extensions that allow runtime packaging and deployment of JEE artifacts for in-container testing (Arquillian).

Each test class in the suite acts as a deployable unit. The deployable units, or artifacts, are defined in a declarative way using annotations.

The declarative approach allows many of the tests to be executed in a standalone implementation of Bean Validation, accounting for a boost in developer productivity. However, an implementation is only valid if all tests pass using the in-container execution mode. The standalone mode is merely a developer convenience.

In order to pass the Bean Validation TCK (which is one requirement for becoming a certified Bean Validation provider), you need to:

Note

The designated reference runtime for compatibility testing of the Bean Validation specification is the Sun Java Platform, Enterprise Edition (Java EE) 7 reference implementation (RI), aka Glassfish 4.

While the Bean Validation TCK is rigourous about enforcing an implementation's conformance to the JSR 349 specification, it's reasonable to assume that an implementor may discover new and/or better ways to validate the assertions. This chapter covers the appeals process, defined by the Specification Lead, Red Hat, Inc., which allows implementors of the JSR 349 specification to challenge one or more tests defined by the Bean Validation TCK.

The appeals process identifies who can make challenges to the TCK, what challenges to the TCK may be submitted, how these challenges are submitted, how and by whom challenges are addressed and how accepted challenges to the TCK are managed.

Following the recent adoption of transparency in the JCP, implementors are encouraged to make their appeals public, which this process facilitates. The JCP community should recognize that issue reports are a central aspect of any good software and it's only natural to point out shortcomings and strive to make improvements. Despite this good faith, not all implementors will be comfortable with a public appeals process. Instructions about how to make a private appeal are therefore provided.

To submit a challenge, a new issue of type Bug should be created against BVTCK in the Hibernate JIRA instance. The appellant should complete the Summary, Component (TCK Appeal), Environment and Description fields only. Any communication regarding the issue should be added in the comments of the issue for accurate record.

To submit an issue in the Hibernate JIRA, you must have a (free) JIRA member account. You can create a member account using the on-line registration.

If you wish to make a private challenge, you should follow the above procedure, setting the Security Level to Private. Only the issue reporter, TCK Project Lead and designates will be able to view the issue. Should there be need for clarification or discussions before actually entering a bug into the issue tracker, an email can be sent to beanvalidation-tck@redhat.com.

This chapter explains how to obtain the TCK and supporting software and provides recommendations for how to install/extract it on your system.

You can obtain a release of the Bean Validation TCK project from the from the download page on the Hibernate Validator website. The Bean Validation TCK is distributed as a ZIP file, which contains the TCK artifacts (the test suite binary and source, the test suite descriptor, the audit source and report), the TCK library dependencies in /lib and documentation in /doc. The contents should look like:

artifacts/
changelog.txt
docs/
lib/
license.txt
setup-examples/
src/
readme.md

You can also download the source code from GitHub - https://github.com/beanvalidation/beanvalidation-tck.

The JSR 349 reference implementation (RI) project is named Hibernate Validator. You can obtain the Hibernate Validator release used as reference implementation from the Hibernate Validator download page.

Note

Hibernate Validator is not required for running the Bean Validation TCK, but it can be used as a reference for familiarizing yourself with the TCK before testing your own Bean Validation implementation.

The TCK requires the following two Java runtime environments:

You should refer to vendor instructions for how to install the runtime.

The rest of the TCK software can simply be extracted. It's recommended that you create a dedicated folder to hold all of the jsr349-related artifacts. This guide assumes the folder is called jsr349. Extract the src folder of the TCK distribution into a sub-folder named tck or use the following git commands:

git clone git://github.com/beanvalidation/beanvalidation-tck tck
git checkout 1.1.0.Final

You can also check out the full Hibernate Validator source into a subfolder ri. This will allow you to run the TCK against Hibernate Validator.

git clone git://github.com/hibernate/hibernate-validator.git ri
git checkout 5.0.0.Final

The resulting folder structure is shown here:

jsr349/
   ri/
   tck/

Now lets have a look at one concrete test of the TCK, namely ConstraintInheritanceTest (found in tck/tests/src/main/java/org/hibernate/beanvalidation/tck/tests/constraints/inheritance/ConstraintInheritanceTest.java):

package org.hibernate.beanvalidation.tck.tests.constraints.inheritance;

import java.lang.annotation.Annotation;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import javax.validation.ConstraintViolation;
import javax.validation.Validator;
import javax.validation.constraints.DecimalMin;
import javax.validation.constraints.NotNull;
import javax.validation.constraints.Size;
import javax.validation.metadata.BeanDescriptor;
import javax.validation.metadata.ConstraintDescriptor;
import javax.validation.metadata.PropertyDescriptor;

import org.jboss.arquillian.container.test.api.Deployment;
import org.jboss.arquillian.testng.Arquillian;
import org.jboss.shrinkwrap.api.spec.WebArchive;
import org.jboss.test.audit.annotations.SpecAssertion;
import org.jboss.test.audit.annotations.SpecAssertions;
import org.jboss.test.audit.annotations.SpecVersion;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import org.hibernate.beanvalidation.tck.util.TestUtil;
import org.hibernate.beanvalidation.tck.util.shrinkwrap.WebArchiveBuilder;

import static org.hibernate.beanvalidation.tck.util.TestUtil.assertCorrectConstraintTypes;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;

/**
 * @author Hardy Ferentschik
 */
@SpecVersion(spec = "beanvalidation", version = "1.1.0")
public class ConstraintInheritanceTest extends Arquillian {

    private Validator validator;

    @Deployment
    public static WebArchive createTestArchive() {
        return new WebArchiveBuilder()
                .withTestClassPackage( ConstraintInheritanceTest.class )
                .build();
    }

    @BeforeMethod
    public void setupValidator() {
        validator = TestUtil.getValidatorUnderTest();
    }

    @Test
    @SpecAssertion(section = "4.3", id = "b")
    public void testConstraintsOnSuperClassAreInherited() {
        BeanDescriptor beanDescriptor = validator.getConstraintsForClass( Bar.class );

        String propertyName = "foo";
        assertTrue( beanDescriptor.getConstraintsForProperty( propertyName ) != null );
        PropertyDescriptor propDescriptor = beanDescriptor.getConstraintsForProperty( propertyName );

        Annotation constraintAnnotation = propDescriptor.getConstraintDescriptors()
                .iterator()
                .next().getAnnotation();
        assertTrue( constraintAnnotation.annotationType() == NotNull.class );
    }

    @Test
    @SpecAssertions({
            @SpecAssertion(section = "4.3", id = "a"),
            @SpecAssertion(section = "4.3", id = "b")
    })
    public void testConstraintsOnInterfaceAreInherited() {
        BeanDescriptor beanDescriptor = validator.getConstraintsForClass( Bar.class );

        String propertyName = "fubar";
        assertTrue( beanDescriptor.getConstraintsForProperty( propertyName ) != null );
        PropertyDescriptor propDescriptor = beanDescriptor.getConstraintsForProperty( propertyName );

        Annotation constraintAnnotation = propDescriptor.getConstraintDescriptors()
                .iterator()
                .next().getAnnotation();
        assertTrue( constraintAnnotation.annotationType() == NotNull.class );
    }

    @Test
    @SpecAssertions({
            @SpecAssertion(section = "4.3", id = "a"),
            @SpecAssertion(section = "4.3", id = "c")
    })
    public void testConstraintsOnInterfaceAndImplementationAddUp() {
        BeanDescriptor beanDescriptor = validator.getConstraintsForClass( Bar.class );

        String propertyName = "name";
        assertTrue( beanDescriptor.getConstraintsForProperty( propertyName ) != null );
        PropertyDescriptor propDescriptor = beanDescriptor.getConstraintsForProperty( propertyName );

        List<Class<? extends Annotation>> constraintTypes = getConstraintTypes( propDescriptor.getConstraintDescriptors() );

        assertEquals( constraintTypes.size(), 2 );
        assertTrue( constraintTypes.contains( DecimalMin.class ) );
        assertTrue( constraintTypes.contains( Size.class ) );
    }

    @Test
    @SpecAssertions({
            @SpecAssertion(section = "4.3", id = "a"),
            @SpecAssertion(section = "4.3", id = "c")
    })
    public void testConstraintsOnSuperAndSubClassAddUp() {
        BeanDescriptor beanDescriptor = validator.getConstraintsForClass( Bar.class );

        String propertyName = "lastName";
        assertTrue( beanDescriptor.getConstraintsForProperty( propertyName ) != null );
        PropertyDescriptor propDescriptor = beanDescriptor.getConstraintsForProperty( propertyName );

        List<Class<? extends Annotation>> constraintTypes = getConstraintTypes( propDescriptor.getConstraintDescriptors() );

        assertEquals( constraintTypes.size(), 2 );
        assertTrue( constraintTypes.contains( DecimalMin.class ) );
        assertTrue( constraintTypes.contains( Size.class ) );
    }

    @Test
    @SpecAssertion(section = "4.6", id = "a")
    public void testValidationConsidersConstraintsFromSuperTypes() {
        Set<ConstraintViolation<Bar>> violations = validator.validate( new Bar() );
        assertCorrectConstraintTypes(
                violations,
                DecimalMin.class, DecimalMin.class, ValidBar.class, //Bar
                NotNull.class, Size.class, ValidFoo.class, //Foo
                NotNull.class, Size.class, ValidFubar.class //Fubar
        );
    }

    private List<Class<? extends Annotation>> getConstraintTypes(Iterable<ConstraintDescriptor<?>> descriptors) {
        List<Class<? extends Annotation>> constraintTypes = new ArrayList<Class<? extends Annotation>>();

        for ( ConstraintDescriptor<?> constraintDescriptor : descriptors ) {
            constraintTypes.add( constraintDescriptor.getAnnotation().annotationType() );
        }

        return constraintTypes;
    }
}

Each test class is treated as an individual artifact (hence the @Deployment annotation on the class). In most tests the created artifact is a standard Web application Archive build via WebArchiveBuilder which in turn is a helper class of the TCK itself alleviating the creation of of the artifact. All methods annotated with @Test are actual tests which are getting run. Last but not least we see the use of the @SpecAssertion annotation which creates the link between the tck-audit.xml document and the actual test (see Section 1.1, “TCK Primer”).

Running the TCK against the Bean Validation RI (Hibernate Validator) and JBoss AS 7

This chapter covers the two types of reports that can be generated from the TCK, an assertion coverage report and the test execution results.

A specification can be distilled into a collection of assertions that define the behavior of the software. This section introduces the Bean Validation TCK coverage report, which documents the relationship between the assertions that have been identified in the JSR 349 specification document and the tests in the TCK test suite.

The structure of this report is controlled by the assertion document, so we'll start there.

The Bean Validation TCK developers have analyzed the JSR 349 specification document and identified the assertions that are present in each chapter. Here's an example of one such assertion found in section 2.1: “Every constraint annotation must define a message element of type String

The assertions are listed in the XML file tck-audit.xml in the Bean Validation TCK distribution. Each assertion is identified by the section of the specification document in which it resides and assigned a unique paragraph identifier to narrow down the location of the assertion further. To continue with the example, the assertion shown above is listed in the tck-audit.xml file using this XML fragment:


<section id="2.1.1" title="Constraint definition properties">
  ...
   <assertion id="c">
      <text>Every constraint annotation must define a message element of type String</text>
   </assertion>
  ...
</section>

The strategy of the Bean Validation TCK is to write a test which validates this assertion when run against an implementation. A test case (a method annotated with @Test) is correlated with an assertion using the @org.jboss.test.audit.annotations.SpecAssertion annotation as follows:

@Test

@SpecAssertion(section = "2.1.1", id = "c")
public void testConstraintDefinitionWithoutMessageParameter() {
   try {
       Validator validator = TestUtil.getValidatorUnderTest();
       validator.validate( new DummyEntityNoMessage() );
       fail( "The used constraint does not define a message parameter. The validation should have failed." );
   }
   catch ( ConstraintDefinitionException e ) {
       // success
}

To help evaluate the distribution of coverage for these assertions, the TCK provides a detailed coverage report. This report is also useful to help implementors match tests with the language in the specification that supports the behavior being tested.

The coverage report is an HTML report generated as part of the TCK project build. Specifically, it is generated by an annotation processor that attaches to the compilation of the classes in the TCK test suite.

The report is written to the file target/coverage-report/coverage-beanvalidation.html. The report itself has five sections:

  1. Chapter Summary - List the chapters (that contain assertions) in the specification document along with total assertions, tests and coverage percentage.

  2. Section Summary - Lists the sections (that contain assertions) in the specification document along with total assertions, tests and coverage percentage.

  3. Coverage Detail - Each assertion and the test that covers it, if any.

  4. Unmatched Tests - A list of tests for which there is no matching assertion (useful during TCK development).

  5. Unversioned Tests - A list of tests for which there is no @SpecVersion annotation on the test class (useful during TCK development).

The coverage report is color coded to indicate the status of an assertion, or group of assertions. The status codes are as follows:

  • Covered - a test exists for this assertion

  • Not covered - no test exists for this assertion

  • Unimplemented - a test exists, but is unimplemented

  • Untestable - the assertion has been deemed untestable, a note, explaining why, is normally provided

For reasons provided in the tck-audit.xml document and presented in the coverage report, some assertions are not testable.

The coverage report does not give any indication as to whether the tests are passing. That's where the TestNG reports come in.

As you by now know, the Bean Validation TCK test suite is really just a TestNG test suite. That means an execution of the Bean Validation TCK test suite produces all the same reports that TestNG produces. This section will go over those reports and show you were to go to find each of them.

This chapter lays out how to run and configure the TCK harness against a given Bean Validation provider in a given Java EE container. If you have not by now made yourself familiar with the Arquillian documentation, this is a good time to do it. It will give you a deeper understanding of the different parts described in the following sections.

The TCK distribution comes with a directory setup-examples which contains two example projects for running the TCK. If you followed the instructions in Chapter 3, Installation you find the directory under jsr349/tck/setup-examples. Both setups are using Hibernate Validator as Bean Validation Provider and Glassfish 4 as EE constainer. However, one is using Maven as build tool to run the TCK, the other Ant. Depending which of the examples you want to use, you need to install the corresponding build tool.

Each example comes with a readme.md containing the prerequisites for using this setup, how to run the TCK against Hibernate Validator and Glassfish. The readme in setup-examples itself contains information about what needs to be changed to use a different Bean Validation provider and EE container.

The following chapters contain some more information about the general structure of the TCK which will give you a deeper understanding above the simple readme files.

The Bean Validation test harness is built atop TestNG, and it is TestNG that is responsible for selecting the tests to execute, the order of execution, and reporting the results. Detailed TestNG documentation can be found at testng.org.

The tck-tests.xml artifact provided in the TCK distribution must be run by TestNG (described by the TestNG documentation as "with a testng.xml file") unmodified for an implementation to pass the TCK. For testing purposes it is of course ok to modify the file (see also the TestNG documentation)

<suite name="JSR-349-TCK" verbose="1">
    <test name="JSR-349-TCK">

        <method-selectors>
            <method-selector>
                <selector-class name="org.hibernate.beanvalidation.tck.util.IntegrationTestsMethodSelector"/>
            </method-selector>
        </method-selectors>

        <packages>
            <package name="org.hibernate.beanvalidation.tck.tests"/>
        </packages>
    </test>
</suite>

TestNG provides extensive reporting information. Depending on the build tool or IDE you use, the reporting will take a different format. Please consult the TestNG documentation and the tool documentation for more information.

After setting the ValidationProvider you have to make a choice on the right DeployableContainer. Arquillian picks which container it is going to use to deploy the test archive and negotiate test execution using Java's service provider mechanism. Concretely Arquillian is looking for an implementation of the DeployableContainer SPI on the classpath. The setup examples use a remote Glassfish container adapter, which means that Arquillian tries to deploy the test artifacts onto a specified remote Glassfish instance, run the tests remotely and report the results back to the current JVM. The installation directory of the remote container is specified via the container.home property in the example build files.

One of the requirements of an implementation passing the TCK is for it to pass the Bean Validation signature test. This section describes how the signature file is generated and how to run it against your implementation.