As components of the Application Server mature into their own projects, it will become increasingly difficult to manage them as a part of the "jboss" project. They will have their own contributors, releases, components and libraries. Coordinating all of this effort in a single project is simply not scalable.
The obvious solution is to decouple these "super subprojects" from the main Application Server project. With respect to the application server, they will become external dependencies much like JGroups and Hibernate.
However, splitting the Application Server into multiple projects presents new challenges and requires a fresh approach to the build system. Some of the issues which need addressing:
Continuous Integration -- Multiple parallel lines of development must be integrated at regular intervals. What was once accomplished by a cvs update and rebuild must now be done by other means.
Version Alignment -- Projects need the ability to manage their own dependencies. However, the versions of these dependencies must align when projects are aggregated into the Application Server.
Source vs Binary Tradeoff -- A developer shouldn't have to checkout the source for all dependencies to build a given project. Likewise, it should be possible to checkout a dependency and quickly test modifications to that dependency with the project.
This document will define a Build System which will address the above issues.
- Project
From the perspective of the build system, a project is a directory containing source code which produces one or more artifacts. A project may have dependencies on the artifacts of other projects or third-party libraries.
- Component
A source tree which exists only as a part of another project. For example, JTA is a component of the application server.
- Dependency
A requirement that a library or artifact must be available during compilation of a given project. This term can also refer to the artifact or library itself.
- Artifact
The compiled product of a project. Generally, artifacts are JAR files. The Application Server project will have a dependency on artifacts produced by the MicroContainer project. An artifact may be a JAR, SAR, or WAR.
- Library
A JAR file which is not generated from source. Many projects will have a dependency on the log4j library.
A primary goal of the new build system is to support the decomposition of the JBoss Application Server source tree into multiple standalone projects. It should be possible to checkout and build any project independently of any container projects. However, the build system should support integrated builds between dependent projects. For instance, it should still be possible to build the application server and all of its subprojects with one invocation.
The new build system should simplify the way that thirdparty libraries are managed. Specifically, the build system should allow projects to declare which versions of what dependencies they have. The build system should be able to automatically download dependencies from an online repository.
Build scripts should be as declarative as possible, while still allowing for customization to an individual project's needs. The inputs and outputs of a component should be clearly documented.
Projects should be able to declare dependencies without being concerned with where those dependencies reside, or in what form. Dependency information is also needed for documentation and for use by IDE's, so it should be easy to automatically extract this information for use in other contexts.
A project should only need to declare its direct dependencies. Indirect dependencies should be included automatically.
Dependency information is stored and versioned along with the project source. The dependencies themselves will be resolved by the automatic update feature below.
Multiple parallel lines of development need to be constantly integrated to tighten the feedback loop. As the number of projects grows, it will be impractical for each developer to update and build each project. The build system should provide the capability for a developer to update dependencies from an online repository.
For example, instead of having to checkout and build the MicroContainer to test if a recent change affects the Application Server, it should be possible to configure the build system to download the most recent build from the online repository.
As illustrated in the figure below, this requires that the continuous build system (cruisecontrol) be enhanced to publish binary artifacts of each component to the online repository. The repository will include both snapshot and released versions of each project's artifacts.
There will be cases where an updated snapshot dependency will introduce critical bugs which prohibit a downstream developer from moving forward. Therefore, it should also be possible to revert an artifact dependency to a previously working snapshot.
It should be possible to work with multiple projects at once. A developer should not be required to copy artifacts by hand when testing the integration between two projects. Therefore, it should be possible to switch between using a binary snapshot of a dependency (from the online repository) and integrating the local build of that project.
Succinctly, it should be possible to use an artifact of a project instead of a the pre-compiled library simply by checking out the project from CVS and configuring the build of the integration project to call the dependent project's build.
The balance between using the source of a dependency and using its pre-built library from the online repository is one of personal choice and circumstance. The build system should be flexible enough to support both modes, and make it painless to switch back and forth.
Each project depends on a specific version of a dependency. However, when projects are aggregated into an integration project such as the Application Server, they must be compatible with the same version of a library. Since the projects are compiled seperately, there is a risk that when they are combined into a single classpath, there will be runtime LinkageErrors.
Let's take an example of JBossCache and JMS, two projects which both depend on JGroups. Both are built and versioned seperately. A release from each will need to be included in the application server. At this point JBossCache and JMS will need to be compatible with the JGroups library included in the application server.
There is a potential that the JGroups JAR provided by the application server will have binary or functional incompatibilities with JBossCache or JMS because of the version discrepancy. To mitigate this, these projects will need to be continuously integrated with the Application Server to ensure compatibility. This includes compiling and testing these projects against the library versions dictated by the Application Server.
Therefore, the build system should support the overriding of dependency versions so that projects can be compiled and tested with the versions specified by another project. This could be in the form of allowing the library version number to be set as a property. One could then override the properties for all library versions when running a build.
Projects need the ability to modularize source code into components. The build system should provide projects with the ability to integrate component builds and aggregate their artifacts into project releases.
Individual projects need the ability to customize the behavior of the build system to specific needs. The build system should allow for ad-hoc extensions specific to a project to be introduced by clearly defined extension points.
The build system should optimize the code, compile and test cycle for the developer. At the very least, this means that dependency checking should be used to ensure that a given invocation of the build performs the least amount of work.
It should be possible for projects to declare minimum JDK requirements for a given artifact. If the JDK performing the build is at a lower release than an artifact specifies, the artifact will be excluded from the build. For example, a artifact might require JDK 1.5 to build. When the project is built using JDK 1.4, that particular component will not be compiled.
A developer wishes to checkout the AS. The initial checkout will not include any libraries or components. Once the build is invoked, the unsatisfied dependencies will be resolved from the online repository.
cd $WORK
cvs co jboss-as
:
cd jboss-as
ant build
Since this is a clean checkout, none of the libraries will be locally available. The build system will detect the unresolved dependencies and will download them from the repository. Since none of the dependencies (I.E., aop, microcontainer) are available in source form, their artifacts will be downloaded from the repository.
After checking out the jboss-as project, the developer wishes to make a change to the jboss-common dependency. To accomplish this, the developer checks out the dependency from CVS into the working directory. When jboss-as is rebuilt, the build for jboss-common will be invoked. The artifact from the jboss-common project will override the jboss-common library which had previously been downloaded from the online repository.
cd $WORK
cvs co jboss-common
:
cd jboss-as
ant rebuild
If the developer wishes to only build the jboss-common component he can change to that directory and invoke the build. Only the jboss-common project will be built.
cd $WORK
cd jboss-comon
ant rebuild
This use case demonstrates how a developer might test a bug fix in JBossCache with the Application Server. For the purposes of this example, the work directory is empty.
cd $WORK
cvs co -r JBoss_Cache_1_1 jboss-cache
cvs co -r JBoss_AS_5_1 jboss-as
cd jboss-as
ant tests
In this example, the jboss-as build would see the jboss-cache project in the work directory and would build it before building the AS and then running the testsuite.
Developer A commits a valid change to jboss-common, CruiseControl detects the modification and performs a clean build of all affected projects. The build succeeds, and the snapshot artifacts are published to the repository. Later, Developer B performs a build of jboss-as without a source copy of jboss-common. The build system downloads the snapshot of the jboss-common jars and successfully builds the project.
Developer A commits a change to jboss-common which breaks an interface used by jboss-system. Developer A did not build jboss-system against the changed interface and so does not commit any fixes for jboss-system.
Cruisecontrol detects the change in jboss-common and intiates a build of all projects which depend on the given branch of jboss-common. The build fails, and a failure email is issued. No artifacts are published to the repository, so development against the existing snapshot can proceed.