A TCK, or Technology Compatibility Kit, is one of the three required pieces for any JSR (the other two being the specification document and the reference implementation). The TCK is a set of tools and tests to verify that an implementation of the technology conforms to the specification. The tests are the primary component, but the tools serve an equally critical role of providing a framework and/or set of SPIs for executing the tests.

The tests in the TCK are derived from assertions in the written specification document. The assertions are itemized in an XML document, where they each get assigned a unique identifier, and materialize as a suite of automated tests that collectively validate whether an implementation complies with the aforementioned assertions, and in turn the specification. For a particular implementation to be certified, all of the required tests must pass (i.e., the provided test suite must be run unmodified).

A TCK is entirely implementation agnostic. Ideally, it should validate assertions by consulting the specification's public API. However, when the information returned by the public API is not low-level enough to validate the assertion, the implementation must be consulted directly. In this case, the TCK provides an independent API as part of a porting package that enables this transparency. The porting package must be implemented for each CDI implementation. Section 1.3.2, “CDI TCK Components” introduces the porting package and Section 4.3, “The Porting Package” covers the requirements for implementing it.

The goal of any specification is to eliminate portability problems so long as the program which uses the implementation also conforms to the rules laid out in the specification.

Executing the TCK is a form of compatibility testing. It's important to understand that compatibility testing is distinctly different from product testing. The TCK is not concerned with robustness, performance or ease of use, and therefore cannot vouch for how well an implementation meets these criteria. What a TCK can do is to ensure the exactness of an implementation as it relates to the specification.

Compatibility testing of any feature relies on both a complete specification and a complete reference implementation. The reference implementation demonstrates how each test can be passed and provides additional context to the implementor during development for the corresponding assertion.

The CDI TCK is designed as a portable, configurable and automated test suite for verifying the compatibility of an implementation of the JSR 346: Contexts and Dependency Injection for Java EE 1.1 specification. The test suite is built atop TestNG framework and Arquillian platform.

Each test class in the suite acts as a deployable unit. The deployable units, or artifacts, can be either a WAR or an EAR.

Any implementor may submit an appeal to challenge one or more tests in the CDI TCK. In fact, members of the JSR 346 Expert Group (EG) encourage this level of participation.

Any test case (e.g., test class, @Test method), test case configuration (e.g., beans.xml), test beans, annotations and other resources may be challenged by an appeal.

What is generally not challengeable are the assertions made by the specification. The specification document is controlled by a separate process and challenges to it should be handled through the JSR 346 EG by sending an e-mail to jsr346-comments@jcp.org.

To submit a challenge, a new issue should be created in the CDI TCK project of the JBoss JIRA using the Issue Type: Bug. The appellant should complete the Summary, Component (TCK Appeal), Environment and Description Field only. Any communication regarding the issue should be pursed in the comments of the filed issue for accurate record.

To submit an issue in the JBoss JIRA, you must have a (free) JBoss.org member account. You can create a member account using the on-line registration.

If you wish to make a private challenge, you should follow the above procedure, setting the Security Level to Private. Only the issue reporter, TCK Project Lead and designates will be able to view the issue.

The challenges will be addressed in a timely fashion by the CDI TCK Project Lead, as designated by Specification Lead, Red Hat Middleware LLC. or his/her designate. The appellant can also monitor the process by following the issue report filed in the CDI TCK project of the JBoss JIRA.

The current TCK Project Lead is listed on the CDI TCK Project Summary Page on the JBoss JIRA.

Accepted challenges will be acknowledged via the filed issue's comment section. Communication between the CDI TCK Project Lead and the appellant will take place via the issue comments. The issue's status will be set to "Resolved" when the TCK project lead believes the issue to be resolved. The appellant should, within 30 days, either close the issue if they agree, or reopen the issue if they do not believe the issue to be resolved.

Resolved issue not addressed for 30 days will be closed by the TCK Project Lead. If the TCK Project Lead and appellant are unable to agree on the issue resolution, it will be referred to the JSR 346 specification lead or his/her designate.

Periodically, an updated TCK will be released, containing tests altered due to challenges - no new tests will be added. Implementations are required to pass the updated TCK. This release stream is named 1.1.x, where x will be incremented.

Additionally, new tests will be added to the TCK improving coverage of the specification. We encourage implementations to pass this TCK, however it is not required. This release stream is named 1.y.z where y > 1.

You can obtain a release of the CDI TCK project from the download page on Seam Framework website. The release stream for JSR 346 is named 1.1.x. The CDI TCK is distributed as a ZIP file, which contains the TCK artifacts (the test suite binary and source, porting package API binary and source, the test suite descriptor, the audit source and report) in /artifacts and documentation in /doc. The TCK library dependencies are not part of the distribution and can be downloaded on demand (see readme.txt file in /lib).

You can also download the current source code from GitHub repository.

The TCK project is also available in the Maven Central repository as org.jboss.cdi.tck:cdi-tck-impl. The POM file defines all dependencies required to build the TCK.

Executing the TCK requires a Java EE 7 or better runtime environment (i.e., application server), to which the test artifacts are deployed and the individual tests are invoked. The TCK does not depend on any particular Java EE implementation.

The JSR 346: Contexts and Dependency Injection for Java EE 1.1 reference implementation (RI) project is named Weld. The release stream for JSR 346 is named 2.x. You can obtain the latest release from the download page on Seam Framework website.

Naturally, to execute Java programs, you must have a Java SE runtime environment. The TCK requires Java 7 or better, which you can obtain from the Java Software website.

The TCK requires the following two Java runtime environments:

You should refer to vendor instructions for how to install the runtime environment.

The rest of the TCK software can simply be extracted. It's recommended that you create a folder named jsr346 to hold all of the jsr346-related projects. Then, extract the TCK distribution into a subfolder named tck. If you have downloaded the Weld distribution, extract it into a sibling folder named weld. The resulting folder structure is shown here:

jsr346/
   weld/
   tck/

Each test class is treated as an individual artifact. All test methods (i.e., methods annotated with @Test) in the test class are run in the application, meaning bean discovery occurs exactly once per artifact and the same BeanManager is used by each test method in the class.

Eclipse, or any other IDE, is not required to execute or pass the TCK. However an implementor may wish to execute tests in an IDE to aid debugging the tests. This section introduces two essential Eclipse plugins, TestNG and Maven, and points you to resources explaining how to install them.

mvn eclipse:eclipse

Again, the Eclipse plugins are not required to execute the TCK, but can be very helpful when validating an implementation against the TCK test suite and especially when using the modules from the project.

System properties and/or the resource META-INF/cdi-tck.properties, a Java properties file, are used to configure the TCK.

You should set the following properties:

The Arquillian testing platform will look for configuration settings in a file named arquillian.xml in the root of your classpath. If it exists it will be auto loaded, else default values will be used. This file is not a requirement however it's very useful for container configuration. See an example configuration for JBoss TCK runner:

        weld/jboss-tck-runner/src/test/jbossas7/arquillian.xml
      

The CDI TCK relies on an implementation of the porting package to function. There are times when the tests need to tap directly into the CDI implementation to manipulate behavior or verify results. The porting package includes a set of SPIs that provide the TCK with this level of access without tying the tests to a given implementation.

The SPI classes in the CDI TCK are as follows:

Please consult the JavaDoc for these interfaces for the implementation requirements.

You can configure the CDI TCK to just run tests appropriate to the Java EE Web Profile by excluding TestNG group javaee-full, e.g. via maven-surefire-plugin configuration:

      <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <excludedGroups>javaee-full</excludedGroups>
            </configuration>
       </plugin>
        

The CDI TCK is built atop Arquillian and TestNG, and it's TestNG that is responsible for selecting the tests to execute, the order of execution, and reporting the results. Detailed TestNG documentation can be found at testng.org.

The artifacts/cdi-tck-impl-suite.xml artifact provided in the TCK distribution must be run by TestNG 6.3 (described by the TestNG documentation as "with a testng.xml file") unmodified for an implementation to pass the TCK. This file also allows tests to be excluded from a run:

<suite name="CDI TCK" verbose="0" configfailurepolicy="continue">
   <test name="CDI TCK">
      ...
      <classes>
         <class name="org.jboss.cdi.tck.tests.context.application.ApplicationContextTest">
            <methods>
               <exclude name="testApplicationScopeActiveDuringServiceMethod"/>
            </methods>
         </class>
      </classes>
      ...
   </test>
</suite>

TestNG provides extensive reporting information. Depending on the build tool or IDE you use, the reporting will take a different format. Please consult the TestNG documentation and the tool documentation for more information.

It's beyond the scope of this guide to describe in how to set up your build environment to run the TCK. The TestNG documentation provides extensive information on launching TestNG using the Java, Ant, Eclipse or IntelliJ IDEA.

The TCK makes use of the Java 1.4 keyword assert; you must ensure that the JVM used to run the application server is started with assertions enabled. See Programming With Assertions for more information on how to enable assertions.

Tests within the jms test group require some basic Java Message Service configuration. A connection factory, a queue destination for PTP messaging domain and a topic destination for pub/sub messaging domain must be available via JNDI lookup. The corresponding JNDI names are specified with configuration properties - see Section 4.1, “TCK Properties”.

Tests within the persistence test group require basic data source configuration. The data source has to be valid and JTA-based. The JNDI name of the DataSource is specified with configuration property - see Section 4.1, “TCK Properties”.

Tests within the installedLib test group require the CDI TCK cdi-tck-ext-lib artifact to be installed as a library (see also Java EE 6 specification, section EE.8.2.2 "Installed Libraries").

Tests within the systemProperties test group require the following system properties to be set:

The CDI TCK coverage has been measured as follows:

A specification can be distilled into a collection of assertions that define the behavior of the software. This section introduces the CDI TCK coverage report, which documents the relationship between the assertions that have been identified in the JSR 346 specification document and the tests in the TCK test suite.

The structure of this report is controlled by the assertion document, so we'll start there.

         <section id="declaring_bean_qualifiers" title="Declaring the qualifiers of a bean">

          ...

          <assertion id="d">

             <text>Any bean may declare multiple qualifier types.</type>

          </assertion>

          ...

            </section>

         

@Test

@SpecAssertion(section = DECLARING_BEAN_QUALIFIERS, id = "d")

public void testMultipleQualifiers()

{

  Bean<?> model = getBeans(Cod.class, new ChunkyBinding(true), new WhitefishBinding()).iterator().next();

  assert model.getBindings().size() == 3;

}

mvn clean install

5.2.3. TestNG Reports

The CDI TCK test suite is really just a TestNG test suite. That means an execution of the CDI TCK test suite produces the same reports as TestNG does. This section will go over those reports and show you where to find each of them.

5.2.3.1. Maven, Surefire and TestNG

When the CDI TCK test suite is executed during the Maven test phase of the TCK runner project, TestNG is invoked indirectly through the Maven Surefire plugin. Surefire is a test execution abstraction layer capable of executing a mix of tests written for JUnit, TestNG, and other supported test frameworks.

Why is this relevant? It means two things. First, it means that you are going to get a summary of the test run on the commandline. Here's the output generated when the tests are run using standalone mode.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
[XmlMethodSelector] CLASSNAME:org.jboss.testharness.impl.testng.DisableIntegrationTestsMethodSelector
[XmlMethodSelector] SETTING PRIORITY:0
[XmlMethodSelector] CLASSNAME:org.jboss.testharness.impl.testng.ExcludeIncontainerUnderInvestigationMethodSelector
[XmlMethodSelector] SETTING PRIORITY:0
Tests run: 441, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.816 sec

Results :

Tests run: 441, Failures: 0, Errors: 0, Skipped: 0

Note

The number of tests executed, the execution time, and the output will differ when you run the tests using in-container mode as the CDI TCK requires.

If the Maven reporting plugin that complements Surefire is configured properly, Maven will also generate a generic HTML test result report. That report is written to the file test-report.html in the target/surefire-reports directory of the TCK runner project. It shows how many tests were run, how many failed and the success rate of the test run.

The one drawback of the Maven Surefire report plugin is that it buffers the test failures and puts them in the HTML report rather than outputting them to the commandline. If you are running the test suite to determine if there are any failures, it may be more useful to get this information in the foreground. You can prevent the failures from being redirected to the report using the following commandline switch:

mvn test -Dsurefire.useFile=false

The information that the Surefire provides is fairly basic and the detail pales in comparison to what the native TestNG reports provide.

5.2.3.2. TestNG HTML Reports

TestNG produces several HTML reports for a given test run. All the reports can be found in the target/surefire-reports directory in the TCK runner project. Below is a list of the three types of reports:

• Test Summary Report

• Test Suite Detail Report

• Emailable Report

The first report, the test summary report, shown below, is written to the file index.html. It produces the same information as the generic Surefire report.

The summary report links to the test suite detail report, which has a wealth of information. It shows a complete list of test groups along with the classes in each group, which groups were included and excluded, and any exceptions that were raised, whether from a passed or failed test. A partial view of the test suite detail report is shown below.

The test suite detail report is very useful, but it borderlines on complex. As an alternative, you can have a look at the emailable report, which is a single HTML document that shows much of the same information as the test suite detail report in a more compact layout. A partial view of the emailable report is shown below.

Now that you have seen two ways to get test results from the Maven test execution, let's switch over to the IDE, specifically Eclipse, and see how it presents TestNG test results.

5.2.3.3. Test Results in the TestNG Plugin View

After running a test in Eclipse, the test results are displayed in the TestNG plugin view, as shown below.

The view offers two lists. The first is a list of all methods (tests) in the class flagged as either passed or failed. The second is a list of methods (tests) in the class that failed. If there is a test failure, you can click on the method name to get the stacktrace leading up to the failure to display in the lower frame.

You can also find the raw output of the TestNG execution in the IDE console view. In that view, you can click on a test in the stacktrace to open it in the editor pane.

One of the nice features of TestNG is that it can keep track of which tests failed and offer to run only those tests again. You can also rerun the entire class. Buttons are available for both functions at the top of the view.