Code coverage describes the degree to which the source code of a program has been tested.
The basic information about code coverage can be found at http://www.bullseye.com/coverage.html .
Code coverage runs in most frameworks in these phases :
- instrumentation changes the bytecode of a classes and inject custom calls to the program
- run of tests and collection of metrics reported by the injected custom calls
- processing of the metrics
- reporting of results
The main java projects which are working or worked on code coverage are :
- JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, Quilt, Codecover, Coverclipse, Jester, Cobertura, Emma, Clover
The criteria for selection of right tool for our projects are:
- Maven integration
- Hudson integration
- Reporting in HTML
- Fast processing
- Good documentation
- Progress on project support and development
The mentioned tools will be divided into there groups :
a) Tools that don't fullfill criteria
- dead tools
-JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, quilt
- different testing method
b) Tools that fullfils criteria
- Dead tools
|Description|| Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test.
Optimized for use with the JUnit unit test package
|Last release||20 October 2003|
|License||Apache License, Artistic license|
|Description||The popular tools included with GroboUtils include multi-threaded tests, hierarchial unit tests, and a code coverage tool.|
|Technology||logging statements for tracking coverage|
|Reports||XML, xsl-t transformations, HTML|
|Description|| Gretel is a test coverage monitoring tool for Java programs. The current version provides statement
coverage monitoring (identifying which lines of Java have been executed, and which have not been touched by testing).
|Start||GUI, command line|
|Last realease||June 2002|
|Description|| Hansel is an extension to JUnit that adds code coverage testing to the testing framework.
Much of the source code is taken from Gretel a "Residual Test Coverage Monitoring"-Tool developed at the University of Oregon.
|Description||NoUnit allows you to see how good your JUnit tests are. It generates a report from your code to graphically show you how many of your project's methods are being tested , and how well|
|Reports||XML, xsl, HTML|
|Addition information||dont need source codes|
- Different technology
|Description|| Jester doesn't watch which lines of code have been executed.
Instead Jester changes the source code, recompiles it, and runs the test suite to see if anything breaks.
|Last release||jester 2005-04-02, simple-jester 2009-11-24|
|License||open license (http://jester.sourceforge.net/license.html)|
|IDE support|| http://www.dcs.shef.ac.uk/intranet/teaching/projects/archive/ug2005/abs/u2sl.htm
Undergraduate project dissertation
there is another eclipse plugin ? http://jester.cvs.sourceforge.net/viewvc/jester/jester-eclipse-plugin/
|Description||Coverlipse is an Eclipse plugin that visualizes the code coverage of JUnit Tests.|
|Reports||Eclipse, java editor|
|Comments||only eclipse support|
|Start||Eclipse, Command line, ant|
|Technology||CodeCover measures statement, branch, loop, term coverage (subsumes MC/DC), question mark operator coverage, and synchronized coverage|
|Reports||template engine Velocity, html, csv|
|Licence||Eclipse Public Licence (EPL)|
|Comments|| - there is no maven support
- nice reports
- good documentation
|Description||Clover delivers actionable Java code coverage metrics to assess the impact of your tests. It goes beyond raw test coverage to help you focus your limited testing resources where they will be most effective.|
|Start||Eclipse, Netbeans, IntelliJ Command line, ant, maven|
|Technology||Instrumentation of source codes|
|Reports||html, xml, pdf, json|
|Licence||Commercial, sale for acamedics, free for non-profit and open source projects|
|Comments|| - Most complete reports and statistics
- integration with other atlassian products
- good documentation
- Distributed Test Coverage Aggregation
- Extensibility via java api
- Hudson plugin
|Description|| Cobertura is a free Java tool that calculates the percentage of code accessed by tests.
It can be used to identify which parts of your Java program are lacking test coverage.
It is based on jcoverage.
|Start||Ant,Command line, Maven|
|Technology||McCabe cyclomatic code complexity of each class (http://en.wikipedia.org/wiki/Cyclomatic_complexity)|
|Reports||Can generate reports in HTML or XML.|
|Last release||last version 22.214.171.124 released 2010-03-03|
|License||GNU GPL 2|
|IDE support|| http://ecobertura.johoop.de/
written in scala,
supported by one man
added 3,2005 modified sep 13 2006
|Description|| EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other
tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual
developer's work fast and iterative.
|Start||ANT , EMMA maven plugin ( The maven EMMA plugin simply wraps the EMMA Ant tasks, so it behaves in a very similar way),|
|Reports||plain text, HTML (source code linking), XML|
|Support||No write transaction on CVS during last year|
|IDE support||support for eclipse EclEmma (http://sourceforge.net/projects/eclemma/http://sourceforge.net/projects/eclemma/)|
From mentioned frameworks Clover provides the best services. Clover has the best reports, good documentation and easy to use configuration.
As a open source project we can use the product for free. To get the free license the project must be registred and approved by atlassian.
There is also term in license that code coverage reports must be publicly accessible.
Emma provides good enough functionality but the last release was at 2005-07-27 and there is no progress on the project anymore.
Cobertura provides all functions which are needed for code coverage. The main difference between Clover and Cobertura is in reports.
Cobertura report task provides the same basic data but clover offers much more statistics about projects and coverage.
Cobertura maven plugin has also problems with coverage of integration tests, but there is a workaround for this problem.
I vote for Cobertura because it is open source project and provides sufficient data about code coverage of our project.
(Interesting tool is also Jester which uses mutation testing, but it is not what we are looking for right now.)