JBoss.orgCommunity Documentation

Chapter 3. Testing

3.1. Unit tests
3.2. Integration tests
3.3. Writing tests
3.4. Technology Compatibility Kit (TCK) tests

The ModeShape project uses automated testing to verify that the software is doing what it's supposed to and not doing what it shouldn't do. These automated tests are run continuously and also act as regression tests, ensuring that we know if any problems we find and fix reappear later. All of our tests are executed as part of our Maven build process, and the entire build process (including the tests) is automatically run using Hudson continuous integration system.

Unit tests verify the behavior of a single class (or small set of classes) in isolation from other classes. We use the JUnit 4.4 testing framework, which has significant improvements over earlier versions and makes it very easy to quickly write unit tests with little extra code. We also frequently use the Mockito library to help create mock implementations of other classes that are not under test but are used in the tests.

Unit tests should generally run quickly and should not require large assemblies of components. Additionally, they may rely upon the file resources included in the project, but these tests should require no external resources (like databases or servers). Note that our unit tests are run during the "test" phase of the standard Maven lifecycle. This means that they are executed against the raw .class files created during compilation.

Developers are expected to run all of the ModeShape unit tests in their local environment before committing changes to SVN. So, if you're a developer and you've made changes to your local copy of the source, you can run those tests that are related to your changes using your IDE or with Maven (or any other mechanism). But before you commit your changes, you are expected to run a full Maven build using mvn clean install (in the "trunk/" directory). Please do not rely upon continuous integration to run all of the tests for you - the CI system is there to catch the occasional mistakes and to also run the integration tests.

While unit tests test individual classes in (relative) isolation, the purpose of integration tests are to verify that assemblies of classes and components are behaving correctly. These assemblies are often the same ones that end users will actually use. In fact, integration tests are executed during the "integration-test" phase of the standard Maven lifecycle, meaning they are executed against the packaged JARs and artifacts of the project.

Integration tests also use the JUnit 4.4 framework, so they are again easy to write and follow the same pattern as unit tests. However, because they're working with larger assemblies of components, they often will take longer to set up, longer to run, and longer to tear down. They also may require initializing "external resources", like databases or servers.

Note, that while external resources may be required, care should be taken to minimize these dependencies and to ensure that most (if not all) integration tests may be run by anyone who downloads the source code. This means that these external resources should be available and set up within the tests. For example, use in-memory databases where possible. Or, if a database is required, use an open-source database (e.g., MySQL or PostgreSQL). And when these external resources are not available, it should be obvious from the test class names and/or test method names that it involved an external resource (e.g., "MySqlConnectorIntegrationTest.shouldFindNodeStoredInDatabase()").

As mentioned in the introduction, the ModeShape project doesn't follow any one methodology or process. Instead, we simply have a goal that as much code as possible is tested to ensure it behaves as expected. Do we expect 100% of the code is covered by automated tests? No, but we do want to test as much as we can. Maybe a simple JavaBean class doesn't need many tests, but any class with non-trivial logic should be tested.

We do encourage writing tests either before or while you write the code. Again, we're not blindly following a methodology. Instead, there's a very practical reason: writing the tests early on helps you write classes that are testable. If you wait until after the class (or classes) are done, you'll probably find that it's not easy to test all of the logic (especially the complicated logic).

Another suggestion is to write tests so that they specify and verify the behavior that is expected from a class or component. One challenge developers often have is knowing what they should even test and what the tests should look like. This is where Behavior-driven development (BDD) helps out. If you think about what a class' behaviors are supposed to be (e.g., requirements), simply capture those requirements as test methods (with no implementations). For example, a test class for sequencer implementation might have a test method shouldNotThrowAnErrorWhenTheSuppliedStreamIsNull() { }. Then, after you enumerate all the requirements you can think of, go back and start implementing the test methods.

If you look at the existing test cases, you'll find that the names of the unit and integration tests in ModeShape follow a naming style, where the test method names are readable sentences. Actually, we try to name the test methods and the test classes such that they form a concisely-worded requirement. For example,


is easily translated into a readable requirement:

InMemorySequencer should not throw an error when the supplied stream is null.

In fact, at some point in the future, we'd like to process the source to automatically generate a list of the behavior specifications that are asserted by the tests.

But for now, we write tests - a lot of them. And by following a few simple conventions and practices, we're able to do it quickly and in a way that makes it easy to understand what the code is supposed to do (or not do).

Many Java specifications provide TCK test suites that can be used to check or verify that an implementation correctly implements the API or SPI defined by the specification. These TCK tests vary by technology, but JSR-283 does provide TCK tests that ensure that a JCR repository implementation exhibits the correct and expected behavior.

ModeShape now implements all of the required JCR 2.0 features:

  • repository acquisition

  • authentication

  • reading/navigating

  • query

  • export

  • node type discovery

  • permissions and capability checking

and implements most of the optional JCR 2.0 features:

  • writing

  • import

  • observation

  • workspace management

  • versioning

  • locking

  • node type management

  • same-name siblings

  • orderable child nodes

ModeShape supports the JCR-SQL2 and JCR-QOM query languages defined in JSR-283, plus the XPath and JCR-SQL languages defined in JSR-170 but deprecated in JSR-283.

The ModeShape project also frequently runs the JCR TCK unit tests from the reference implementation. (These tests are not the official TCK, but are used within the official TCK.) Most of these unit tests are run in the modeshape-jcr module against the in-memory repository to ensure our JCR implementation behaves correctly, and the same tests are run in the modeshape-integration-tests module against a variety of connectors to ensure they're implemented correctly. The modeshape-jcr-tck module runs all of these TCK unit tests, and currently there are only a handful of failures due to known issues (see the JCR specification support section for details).

The ModeShape project has not yet been certified to be fully-compliant with the JCR 2.0 specification, but does plan on attaining this certification in the very near future.