Title | Productizing Process for JEMS |
Author | Andrig (Andy) Miller |
Creation Date | February 9, 2006 |
Status | Final |
Revision | 1.0.1 |
Filename | Productizing Process for JEMS |
Date | Revision | Status | Author | Description |
|---|---|---|---|---|
February 9, 2006 | 0.1 | Draft | Andrig (Andy) Miller | Initial version. |
February 20, 2006 | 0.2 | Draft | Andrig (Andy) Miller | Incorporated feedback from Pierre Fricke. |
February 21, 2006 | 0.3 | Draft | Andrig (Andy) Miller | Incorporated feedback from Shaun Connolly. |
February 22, 2006 | 0.4 | Draft | Andrig (Andy) Miller | Incorporated feedback from Ryan Campbell. |
February 28, 2006 | 0.6 | Draft | Andrig (Andy) Miller | Incorporated feedback from Scott Stark. |
February 28, 2006 | 0.8 | Draft | Andrig (Andy) Miller | Incorporated feedback from Sacha Labourey. |
March 7, 2006 | 0.9 | Draft | Andrig (Andy) Miller | Incorporated feedback from Ivelin Ivanov. |
March 7, 2006 | 0.9.1 | Draft | Andrig (Andy) Miller | Incorporated feedback from Pierre Fricke. |
March 20, 2006 | 0.9.9 | Release Candidate | Andrig (Andy) Miller | Incorporated feedback from Adrian Brock, Andrew Oliver, Manik Surtani, Ben Wang, and Ben Sabrin. |
March 24,2006 | 1.0.0 | Final | Andrig (Andy) Miller | Incorporated feedback from Andy Oliver, Rich Friedman, Sacha Labourey, and Scott Stark. |
April 11, 2006 | 1.0.1 | Final | Andrig (Andy) Miller | Incorporated Pierre's and Shaun's web related checklist for releasing projects, that came from our lessons learned discussion with JBoss Messaging. |
This document will define the process to turn a release of an open source project into a revenue generating product for JBoss, Inc. It is NOT an all encompassing document in that regard though. This document focuses only on those aspects that are led by the development/engineering organization. It does not delve into what product management and services processes are for productizing a JBoss project. However, there are tasks described in this document that do involve product management and services as contributors to the development process in regards to productizing our projects.
This document will not try to dictate process within the development life-cycle of each project, but instead concentrate on the steps that are not directly related to development, but to product.
Below is an illustration of the development life-cycle of our projects:
The above illustration is just one way to visualize the development life-cycle of a project. Certainly, it is not a true spiral or circle, since all of the tasks above can and do happen in parallel.
At a given point in time, this life-cycle above stops to release something that is considered more than just a work in-progress, but something that contains a feature set, and a level of quality that the lead developer(s) are happy with. This is considered the “stable” release. This release is the one that we will focus on in terms of becoming a product.
The above illustration shows that point-in-time when a “stable” release is dropped. Normally, this is solely decided by the lead developer(s) based on the feature set and quality that they deem fit for the label of “stable” release. In this transition from developing code, testing, community testing and feedback, to a stable release the process for turning this into a product for JBoss, Inc. must run in parallel.
In this transition from community releases to a stable release, or a final release, that is ready for customer consumption, the developer cannot make the decision in isolation about when that stable or final release will be. This must be done in conjunction with all of the stakeholders identified in the following sections. If the productizing steps have not been completed, yet the software is ready, it will not be officially released. We can call it a release candidate, but not stable or final until all the productizing steps, that have been agreed to, are complete.
There are seven main areas that I would like to focus on in regards to taking software that is being developed in this life-cycle, and getting it to a point where it is ready to be released as a product.
Product Road Map Creation and Maintenance
Features
Productizing Tasks
Known Bugs
Improvements to Existing Features
Reference documentation
API Reference
Administration Guide
User Guide
On-line education
Trailblazers
Demonstrations
Training materials
Internal training materials
Support organization
System Engineers
Consultants
External training materials
Customers
Partners
Quality Assurance
Performance testing
Scalability testing
Soak testing
Integration testing
Availability testing
Certification testing
Development Tooling
JBoss IDE support for developers
Release the stable or final release
Community announcements
In the following sections, the “Who Does It?” column describes roles that are played. This is not meant to dictate that those tasks are done by non-development resources. The exception to this is product management and services. All other roles can be performed by the project developers, whether the are JBoss employees or outside contributors, if they so choose to do so. We would like the project to perform their work in as flexible a manner as possible. What we care about is delivering a high-quality product as quickly as possible, not who specifically does the work.
A product road map should be developed and maintained for each release of the project. It should contain at least the following:
List of planned new features for the release
The productizing tasks that are needed for the release
Address the high priority know bugs or issues with the previous release(s)
Improvements to existing features
The list of planned new features for the release should be discussed in the forums, and with developers from dependent projects and with product management. The interdependencies of many of our projects makes this critical. Feedback from our customers, in the form of surveys, support cases that are feature requests, feature requests from the forums, etc., should all be incorporated to come up with this list. The finalized road map should be in sync with product management’s product plans.
The productizing tasks that are discussed throughout this document should be incorporated into the road map. For example, let’s say that you don’t currently have a soak test, and that needs to be developed, tested, and executed to complete the productizing tasks for the next release. Then you would add that to the road map, and that would turn into JIRA tasks.
Where the known bugs and issues are concerned any fixes from previous releases that have to be reconciled due to overlapping development needs to be addressed. The road map shouldn’t necessarily address each one of those individually, but just make sure that the overall task is taken into account in the plan. Also, some of these issues may not be bugs, but merely that code needs to be re-factored or that performance enhancements have been identified in particular areas from either customers or our own testing, etc.
Improvements to existing features can take many forms, depending on the project. It may involve making a particular feature easier to use, it may involve making a feature easier to manage, etc.
A good example of a product road map is the following from the Portal project:
Some questions that are good to ask yourself as you prepare the road map are:
What do you plan to do for this release?
Have you prioritized the work?
Is the work specified?
Have you discussed it with others to validate the ideas?
Have you used feedback from your users?
What do you need from other projects?
Who uses your project? How will they be affected?
What do other projects want from you?
Do you have tasks for all the productizing that needs to be done?
Have you scheduled time for the productizing work, i.e. taken it into account, when estimating, what can be done for a release?
Other potential issues:
What do you plan to deprecate?
What do you plan to remove or retire?
Should you really be doing that in your project?
Is the work already done elsewhere? Don't fall into the "Not Invented Here" trap.
What third-party dependencies do you plan to introduce?
How will that third-party software be supported?
What is the license for that third-party software?
One thing that I would like to stress, is that we want to create releases of reasonable size. The product road map shouldn’t contain every possible feature, fix, issue, etc. Use your best judgment in what can be delivered in a reasonable amount of time and prioritize accordingly. Remember, release early and often is the goal. There are no rules of thumb for how often releases should be made, because that is highly dependent on the project, its maturity, and the market demands.
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Road map creation. | Define the features, fixes, issues, productizing tasks, and improvements to be made for a given release of the project. | Development with feedback from other projects, the community, customers and product management. | Delivered before work starts on the release. |
Publish road map. | Post the road map on the jboss.org website (jboss wiki is a good tool for this), and create the JIRA release with associated tasks. | Development. | Delivered before work starts on the release. |
Reference documentation is produced in parallel with developing the code, and should evolve through the community releases, such as Alpha, Beta and Release Candidate releases. Reference documentation is an area that we do quite well, as illustrated by the examples below:
The JBoss 4 Application Server Guide
HIBERNATE - Relational Persistence for Idiomatic Java
JBoss Portal 2.2 Reference Guide
JBoss jBPM 3.1 Workflow and BPM made practical
TreeCache: a Tree Structured Replicated Transactional Cache
JBoss Microcontainer Reference
SEAM - Contextual Components A Framework for Java EE 5
JBoss EJB 3.0 Reference Documentation
These examples, all go over the public API of the software at a minimum. Installation, configuration and on-going administration, if applicable should also be covered. They are almost all published on the docs.jboss.org site, and they should all be accessible from there, even if they are not directly hosted there. Hibernate and Apache Tomcat documents are examples of this. The documents should be published in HTML, for easy on-line viewing, PDF for printing purposes, and if there are any examples, then the source code for those examples should be provided in an archive format such as zip.
Each project will certainly have different levels of documentation that is needed, but a plan for what the minimum for each project should be put together in conjunction with product management. Also, the documentation team, should be involved in creating the documentation, to make it navigable, presentable, and published in the two different formats required. A recommended set of documentation would be:
API Reference
The API reference should contain a definition of the public API. This public API will be backward compatible between minor releases. Private APIs can change between all releases, with the exception that they need to preserve binary compatibility. There is a more complete statement on API stability and compatibility between releases in the JBoss Product Versioning Wiki.
Administration Guide
User Guides
The user guide should contain, if applicable, a list of unsupported or experimental features that may be present, but are not recommended for production use.
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Minimum Content Definition. | Define the minimum documentation set that should be produced for the project (e.g. API reference, administration guide, user guide). | Product management and development. | Delivered prior to the first alpha release[a]. |
Documentation Creation. | Create the minimum defined documentation set for the project. | Development and the documentation team. | Delivered with each release, and iteratively updated as the project progresses through alphas, betas, release candidates through to a stable release. |
[a] This may only be applicable to new projects that haven’t had a stable release yet. Of course, a project may go through a significant structural change to warrant a change in the content definition. | |||
Note: The definition for alpha, beta, and release candidate are in the JBoss Product Versioning Wiki page. It is under the heading, “Current Qualifier Conventions (Post 2006-03-01)”. Here is the link:
On-line education primarily consists of two elements. Trailblazers and demonstrations. The Trailblazers and demonstrations should be produced in parallel with the code and should evolve through the community releases, such as Alpha, Beta and Release Candidate releases. Examples of these are as follows:
JBoss Seam DVD Store Demonstration
Overall page for Trailblazers and Demonstrations
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Needs assessment for trailblazer and demonstration. | Determine whether trailblazers and demos are needed to help market the project, and to help with adoption. | Product management and development. | Delivered prior to the first alpha release[a]. |
Trailblazer Creation. | Create the minimum defined documentation set for the project. | Development and the documentation team. | Delivered with each release, and iteratively updated as the project progresses through alphas, betas, release candidates through to a stable release. |
Demonstration Creation. | Write an application or record a demo of the usage of the project, whichever is appropriate. | Development and the documentation team. | Delivered with each release, and iteratively updated as the project progresses through alphas, betas, release candidates through to a stable release. |
[a] This may only be applicable to new projects that haven’t had a stable release yet. Of course, a project may go through a significant structural change to warrant a change in the content definition. | |||
There are five distinct audiences for training materials. First, and foremost is our customers. This training targets the developers and administrators that will be using our technology to develop applications and support applications respectively. Second, is the support organization. In order for them to be able to be as self-sufficient as possible, they need training. This training needs to be detailed enough that it helps them be able to troubleshoot issues that customers have. Third is our consultants. They need the same level of training as support, in that they will not only be helping to develop solutions in concert with our customers, but they will be the first line of support in solving development related issues (troubleshooting ability is key). Fourth is our system engineers. They need training similar to the consultants, in that they will be in front of prospective customers, and may have to delve into technical details during pre-sales activities. And fifth, is our partners. The training for our customers is what is, and will still be, used for their training.
The training materials for developers, can certainly fill part of the training needs for support, consulting and our partners. It will need to be augmented with training that is helpful for troubleshooting customer problems. This training material should have instructions on where errors are logged, and typical reasons that exceptions are thrown, etc.
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Define structure of training materials for customers. | Define the number and types of classes that the training materials need to support (e.g. Beginner, advanced, etc.). | Development with input from services. | The definition should be defined prior the first beta release[a]. |
Needs assessment for support, consulting and sales engineers. | Do a troubleshooting assessment for support/consulting, so that additional training materials, or training sessions determined. At a minimum this should contain a triage list for first line support that identifies what information needs to be collected for problem resolution. | Development and product management with services. | The needs assessment should be complete prior to the first release candidate. |
Develop training materials. | Develop the identified training materials for customers and from the needs assessment for support. | Development and documentation team. | The training materials should be delivered for testing purposes at the same time as the first release candidate[b]. |
Test training materials. | Test the training materials in a real classroom setting to make sure that the training materials are accurate, and that the labs actually work. | Services with feedback to development[c]. | Testing of training materials should be done at the time of the first release candidate. |
[a] This may only be applicable to new projects that haven’t had a stable release yet. Of course, a project may go through a significant structural change to warrant a change in the current structure. [b] This may have exceptions for newer technology or projects that we don’t anticipate will have significant uptake in the market on their initial stable release. These exceptions will be made on a case-by-case basis by product management. [c] The feedback should take the form of JIRA issue for the project, that would be defined as a blocking issue for a stable release. | |||
Note: The definition for alpha, beta, and release candidate are in the JBoss Product Versioning Wiki page. It is under the heading, “Current Qualifier Conventions (Post 2006-03-01)”. Here is the link:
All JEMS projects should be going through a standard quality assurance process[1]. There are five areas that need to addressed for projects where quality assurance is concerned.
The first area is performance testing. Every JEMS product should have a performance test that measures straight line performance of the product (single virtual user/client). The second area is scalability testing. Every JEMS product should have a scalability test that measures performance under high concurrent usage scenarios (many virtual users, clients, nodes, etc). This will be different depending on the JEMS product you are considering, but we should be able to sustain straight line performance levels (or at least not degrade very much) with high concurrency. The third area is soak testing. Every JEMS product should have a soak test that demonstrates sustained high performance with high concurrency over a long duration of time (catch issues like unintended object retention, leaked file descriptors, garbage collection issues, etc.). This test will run for a minimum of 24 hours. The fourth area is integration testing. All of the JEMS components that can be used in conjunction with each other for an application that our customers may develop or deploy, should be tested under scenarios that have them work together. The fifth area is availability testing. In this test, we should use our scalability scenarios and create fault conditions, so that we have a system under high concurrent usage, and are able to measure the ability to have failures and continue running.
These are all important quality aspects that our customers will expect to have nailed with each and every release of JEMS products. The testing described above is not meant to replace the existing unit test suites that each project already executes through their build process, or is it meant to replace any performance testing that each project may already have in place. What is described could very well leverage existing tests that projects already have.
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Define performance test scenarios. | For each JEMS project, there should be a performance test(s) scenarios defined with a goal for what the straight line performance should be (this could be relative to a baseline release, or relative to a competitors number, etc.) | QA and development. | Complete prior to the first release candidate[a]. |
Define scalability test scenarios. | For each JEMS project, there should defined what constitutes high concurrency for that given project, and what levels of concurrency should be the goal. | QA and development. | Complete prior to the first release candidate[b]. |
Define integration test scenarios. | For JEMS as a whole, scenarios should be defined that cross all of the integration points of the JEMS (e.g. Application that has a web tier that uses JSF/Seam, it has a middle tier that uses Stateless and Stateful session beans, and implements a workflow through jBPM, persists through EJB3/Hibernate, etc.) | QA and development. | Complete prior to the first release candidate of the application server (this is where most of the integration comes into play)[c]. |
Define availability test scenarios. | For each project, and for JEMS as a whole, test scenarios that inject failures in a high-availability configuration should be defined. Fault injection can be done through many techniques. Some as simple as unplug the network cable from a system, to some as sophisticated as having an aspect that is deployed that inject exceptions into the running application. | QA and development. | Complete prior to the first release candidate[d]. |
Build performance test. | Develop the appropriate test scripts to automate the performance test. | QA and development. | Complete and ready to execute by the time we offer silver support. |
Build scalability test. | Develop the appropriate test scripts to automate the scalability test. | QA and development. | Complete and ready to execute by the time we offer silver support. |
Build integration test. | Develop the appropriate test scripts to automate the integration test. | QA and development. | Complete and ready to execute by the time we offer silver support. |
Build availability test. | Develop the appropriate test scripts to automate the availability tests[e]. | QA and development. | Complete and ready to execute by the time we offer silver support. |
Execute performance test. | Run the test. | QA and development. | Should be complete prior to moving project to support levels above silver. |
Execute scalability test. | Run the test. | QA and development. | Should be complete prior to moving project to support levels above silver. |
Execute soak test. | Run the test. | QA and development. | Should be complete prior to moving project to support levels above silver. |
Execute integration test. | Run the test. | QA and development. | Should be complete prior to moving project to support levels above silver. |
Execute availability test. | Run the test. | QA and development. | Should be complete prior to moving project to support levels above silver. |
[a] This may not be needed for each and every release, as once they are developed the scenarios may not change. However, the goals of the runtime may change with each release. Therefore, at a minimum we should evaluate the goals with each release. [b] Same as seven. [c] This will probably only have to be done when there is a new JEMS component, once it is done the first time. [d] Same as seven. [e] This may be manual, unless it is identified that aspects are needed to be developed that will inject exceptions into the running application. | |||
Note: The definition for alpha, beta, and release candidate are in the JBoss Product Versioning Wiki page. It is under the heading, “Current Qualifier Conventions (Post 2006-03-01)”. Here is the link:
Many of the JEMS projects have development and management tooling requirements. Developers in our target customer base, are typically not the highest skilled developers[2], and certainly administrators need information and tools to make them more productive, especially in large deployments. Therefore, there is a need to make our platform as accessible to developers and administrators as possible through appropriate tooling.
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Define development tool needs. | Define what developers need to be productive developing against a specific project. | Product management and development. | Complete by the first alpha release. |
Develop tools. | Build the tools that have been defined for developers. | Development. | Complete by the time the release is considered stable[a]. |
Define management needs. | Define what administrators need to be productive managing a production deployment. | Product management and development. | Complete by the first alpha release. |
Develop management tools. | Build the tools, and the features within the product to expose management information for administrators. | Development. | Complete by the time the project moves to support levels above silver. |
[a] This may or may not be a hard requirement depending on the project, and the market adoption rate. Exceptions to this should be approved by product management. | |||
Note: The definition for alpha, beta, and release candidate are in the JBoss Product Versioning Wiki page. It is under the heading, “Current Qualifier Conventions (Post 2006-03-01)”. Here is the link:
Again, to reiterate what has been said already. The designation of the final release should never be done in a vacuum. This should be coordinated through product management. No one within JBoss should read about a release of our software without knowing about it in advance. This allows us to coordinate all public relations activities, as well as do a final check on whether sales and services are truly ready to go.
Task | Description | Who Does It? | When Is It Delivered? |
|---|---|---|---|
Review release content and timing. | Review the plan for what will be released, in terms of web site content, and exactly what the timing will be. | Product management with development. | Approximately four weeks prior to final release. |
Review product data sheet. | Review the product data sheet with product management, to make sure that it is accurate. | Product management with development. | Approximately one to two weeks in advance of the final release. |
Review press release. | Review the press release with product management for accuracy of information. | Development with product management. | Approximately one week in advance of the final release. |
Verify that agreed to productizing steps are complete. | Make sure that all the agreed and applicable productizing steps from sections one through five have been completed. | QA led with project management and development. | Just before final release. |
Verify license conformance for distribution. | Verify that the final distribution conforms to the license that it uses, and that third-party libraries licenses are being complied to. A statement of all licenses involved in the distribution and what they apply to. This should included a source code header check for all the files. | QA led with development[a]. | Just before final release. |
Create/Update JBoss.org product pages. | Create or update the Jboss.org product pages, documentation, and downloads with new release. | Development. | On release day. |
Test links on website. | Test all of the links from the new release information, downloads, etc., to make sure that everything is functional. | Development. | On release day. |
Internal communication. | Development informs QA that the release is complete, and tags the source repository appropriately. | Development. | On release day. |
Announcement. | Certification of the release, and communication is made to internal stakeholders and community forums. | Development and QA with review from product management[b]. | On release day. |
Official corporate announcement. | Marketing collateral, PR, JBoss ON, etc. | Product management. | Determined by product management. |
[a] This could be generated by the build process, and not necessarily have to be done manually. [b] It is important that release announcements are reviewed by product management to make sure that our messaging is in sync across all communication channels. | |||
Note: The verify license conformance task is a part of producing the final build and distribution, and is not intended to be the time where all license issues are addressed. Issues around whether the licenses are compatible, whether we can use the code within our projects, etc., are a part of due diligence before productizing processes begin. There is a formal license policy in development, that will be linked to here, when it is complete.
Contact Name | Role | E-Mail Address | Related Mailing List |
|---|---|---|---|
Andrig (Andy) Miller | Process owner. | ||
Ivelin Ivanov | Development management. | ||
Shaun Connolly | Product management. | ||
Ryan Campbell | Quality Assurance. | ||
Norman Richards | Documentation. | ||
Damon Sicore | JBoss.org website. |
[1] The probable exception to this rule is JBoss IDE, as it is a development tool, and really doesn’t fit the profile for these types of tests. Of course, JBoss IDE should do some form of performance testing around UI responsiveness and memory footprint.
[2] If you have every heard the presentation given by Dave Thomas of The Pragmatic Programmers” titled “Herding Race Horses, and Racing Sheep” you have seen empirical evidence of the fact that most developers are either Novices or Advanced Beginners, and they are NOT competent.