Skip to end of metadata
Go to start of metadata

Overview

Code coverage describes the degree to which the source code of a program has been tested.
The basic information about code coverage can be found at http://www.bullseye.com/coverage.html .
Code coverage runs in most frameworks in these phases :

  • instrumentation changes the bytecode of a classes and inject custom calls to the program
  • run of tests and collection of metrics reported by the injected custom calls
  • processing of the metrics
  • reporting of results

The main java projects which are working or worked on code coverage are :

  • JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, Quilt, Codecover, Coverclipse, Jester, Cobertura, Emma, Clover

Selection

The criteria for selection of right tool for our projects are:

  • Maven integration
  • Hudson integration
  • Reporting in HTML
  • Fast processing
  • Good documentation
  • Progress on project support and development

The mentioned tools will be divided into there groups : 
a) Tools that don't fullfill criteria

  • dead tools
    -JVMDI, JBLANKET, InsectJ, noUnit, Hansel, gretel, Groboutils, quilt
  • different testing method
    -jester
  • other
    -codecover, coverclipse
  • licensed
    Jcoverage
    Clover

b) Tools that fullfils criteria

  • Clover
  • Cobertura
  • Emma

a) Tools that don't fullfill criteria

  • Dead tools

QUILT

Source http://quilt.sourceforge.net/
Description Quilt is a Java software development tool that measures coverage , the extent to which unit testing exercises the software under test.
Optimized for use with the JUnit unit test package
Start Ant, Maven
Technology statement coverage
Frameworks BCEL
Reports  
Report example  
Last release 20 October 2003
License Apache License, Artistic license
IDE support No

GROBOUTILS

Source http://groboutils.sourceforge.net/
Description The popular tools included with GroboUtils include multi-threaded tests, hierarchial unit tests, and a code coverage tool.
Start Ant
Technology logging statements for tracking coverage
Frameworks BCEL
Reports XML, xsl-t transformations, HTML
Reports example http://groboutils.sourceforge.net/codecoverage/CoverageReport-LineCount.html
Last release 2004-11-05
Licence MIT licence
Additional information
  • well documented
  • instruments ear files, war files, zip files within zip file
  • slower performance

GRETEL

Source http://www.cs.uoregon.edu/research/perpetual/dasada/Software/Gretel/
Description Gretel is a test coverage monitoring tool for Java programs. The current version provides statement
coverage monitoring (identifying which lines of Java have been executed, and which have not been touched by testing).
Start GUI, command line
Technology  
Frameworks BCEL
Reports GUI
Last realease June 2002
License BSD

HANSEL

Source http://hansel.sourceforge.net/
Description Hansel is an extension to JUnit that adds code coverage testing to the testing framework.
Much of the source code is taken from Gretel a "Residual Test Coverage Monitoring"-Tool developed at the University of Oregon.
Start ANT
Technology  
Frameworks ASM
Reports  
Last release 2006-10-04
License BSD
Aditional Information

NOUNIT

Source http://nounit.sourceforge.net/
Description NoUnit allows you to see how good your JUnit tests are. It generates a report from your code to graphically show you how many of your project's methods are being tested , and how well
Start command line
Reports XML, xsl, HTML
Report example http://nounit.sourceforge.net/sample.html
Last release 2003-12-15
License GNU GPL
IDE support  
Addition information dont need source codes
  • Different technology

JESTER

Source http://jester.sourceforge.net/
Description Jester doesn't watch which lines of code have been executed.
Instead Jester changes the source code, recompiles it, and runs the test suite to see if anything breaks.
Start  
Technology mutation testing
Frameworks  
Reports easy HTML
Report example http://jester.sourceforge.net/jester.html
Last release jester 2005-04-02, simple-jester 2009-11-24
License open license (http://jester.sourceforge.net/license.html)
IDE support http://www.dcs.shef.ac.uk/intranet/teaching/projects/archive/ug2005/abs/u2sl.htm
Undergraduate project dissertation
there is another eclipse plugin ? http://jester.cvs.sourceforge.net/viewvc/jester/jester-eclipse-plugin/
Additional information
  • very poor documentation, no comments in code
  • there is also Pester for Python PyUnit tests
  • Other

COVERLIPSE

Source http://coverlipse.sourceforge.net/
Description Coverlipse is an Eclipse plugin that visualizes the code coverage of JUnit Tests.
Start Eclipse
Technology Statement Coverage
Frameworks  
Reports Eclipse, java editor
Reports example http://coverlipse.sourceforge.net/screenshots.php
License CPL
Last release 02-14-2009
Comments only eclipse support

CODECOVER

Source http://codecover.org/
Description  
Start Eclipse, Command line, ant
Technology CodeCover measures statement, branch, loop, term coverage (subsumes MC/DC), question mark operator coverage, and synchronized coverage
Frameworks  
Reports template engine Velocity, html, csv
Report example http://codecover.org/documentation/tutorials/SimpleJavaAppReport.html
Last release 19.04.2010
Licence Eclipse Public Licence (EPL)
Comments - there is no maven support
- nice reports
- good documentation
  • Licensed
    JCoverage
    Clover

b) Tools that fullfills criteria

CLOVER

Source http://www.atlassian.com/software/clover/
Description Clover delivers actionable Java code coverage metrics to assess the impact of your tests. It goes beyond raw test coverage to help you focus your limited testing resources where they will be most effective.
Start Eclipse, Netbeans, IntelliJ Command line, ant, maven
Technology Instrumentation of source codes
Frameworks  
Reports html, xml, pdf, json
Report example http://www.atlassian.com/software/clover/sample-reports.jsp
Last release 9-September-2009
Licence Commercial, sale for acamedics, free for non-profit and open source projects
Comments - Most complete reports and statistics
- integration with other atlassian products
- good documentation
- Distributed Test Coverage Aggregation
- Extensibility via java api
- Hudson plugin

COBERTURA

Source http://cobertura.sourceforge.net/
Description Cobertura is a free Java tool that calculates the percentage of code accessed by tests.
It can be used to identify which parts of your Java program are lacking test coverage.
It is based on jcoverage.
Start Ant,Command line, Maven
Technology McCabe cyclomatic code complexity of each class (http://en.wikipedia.org/wiki/Cyclomatic_complexity)
Frameworks ASM
Reports Can generate reports in HTML or XML.
Report example http://cobertura.sourceforge.net/sample/
Last release last version 1.9.4.1 released 2010-03-03
License GNU GPL 2
IDE support http://ecobertura.johoop.de/
written in scala,
supported by one man
http://www.eclipse-plugins.info/eclipse/plugin_details.jsp?id=1151
added 3,2005 modified sep 13 2006
Additional Information
  • cobertura plugin for hudson http://wiki.hudson-ci.org/display/HUDSON/Cobertura+Plugin
  • is a fork of jcoverage started at 2005
  • When instrumenting, you can now specify a zip, jar, war,ear or sar file and Cobertura will instrument any classes inside of the archive and archives within archives
  • Ability to instrument classes on a given classpath instead of specifying filesets
  • Can now use multiple filesets in the cobertura-instrument task
  • support for classes compiled with AspectJ
  • Support for instrumenting classes written in Groovy
  • thread safe
  • data obtained in different instrumentation or test runs can be merged together
Comment  

EMMA

Source http://emma.sourceforge.net/http://emma.sourceforge.net/
Description EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other
tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual
developer's work fast and iterative.
Start ANT , EMMA maven plugin ( The maven EMMA plugin simply wraps the EMMA Ant tasks, so it behaves in a very similar way),
Technology  
Reports plain text, HTML (source code linking), XML
Report example http://emma.sourceforge.net/coverage_sample_a/index.htmlhttp://emma.sourceforge.net/coverage_sample_a/index.html
Support No write transaction on CVS during last year
License CPL
IDE support support for eclipse EclEmma (http://sourceforge.net/projects/eclemma/http://sourceforge.net/projects/eclemma/)
Comments
  • should be fast
  • there is a hudson plugin
  • eclipse support
  • Coverage data obtained in different instrumentation or test runs can be merged together.

Summary

From mentioned frameworks Clover provides the best services. Clover has the best reports, good documentation and easy to use configuration.
As a open source project we can use the product for free. To get the free license the project must be registred and approved by atlassian.
There is also term in license that code coverage reports must be publicly accessible.

Emma provides good enough functionality but the last release was at 2005-07-27 and there is no progress on the project anymore.

Cobertura provides all functions which are needed for code coverage. The main difference between Clover and Cobertura is in reports.
Cobertura report task provides the same basic data but clover offers much more statistics about projects and coverage.
Cobertura maven plugin has also problems with coverage of integration tests, but there is a workaround for this problem.

I vote for Cobertura because it is open source project and provides sufficient data about code coverage of our project.

(Interesting tool is also Jester which uses mutation testing, but it is not what we are looking for right now.)

Labels:
None
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.