Java Code Coverage: Cobertura vs. Emma vs Clover

The last time I looked for free java code coverage tools, I found EclEmma for Eclipse and Cobertura for automated builds through ant or maven. They’re both good tools, but they come up with slightly different numbers. Maybe I just missed it before, but there’s now an Eclipse plugin for Cobertura and a maven plugin for Emma. Both have Hudson and even Sonar plugins, so I thought I’d spend a little more time comparing them. Just for kicks, I also looked at Atlassian’s Clover even though it’s fairly expensive (but free for open-source projects).

Units of coverage

Cobertura focuses on lines and branches (ie. if and case statements); EclEmma concentrates on bytecode instructions, though you can also get line metrics. Emma has a good write-up of their methodology, but I didn’t see as much for Cobertura. Clover looks at statements, branches, and methods to come up with a total score. All these approaches result in fairly similar coverage results (unless your code does things like put multiple statements on the same line separated by semi-colons) but they’ll almost always be a little different.

Eclipse plugin

All the tools will highlight covered and uncovered code in Eclipse for you. The Emma and Clover plugins have a nice graphic display and let you drill down.

Emma:

Clover:

Cobertura’s display in Eclipse is a bit weak, no drill-down into classes and no visual graph of coverage:

Full report

Probably more important than the Eclipse view are the reports that each tool can create, eg. in an automated build. I didn’t take much time to explore different configuration options here, just took the defaults for each.

Emma’s report actually has less visual appeal than its Eclipse plugin. If you invest some time, you can probably make it look better but this is what you get out of the box:

Cobertura, on the other hand, has some nice graphics. If you just configure report output as xml, and turn on the Hudson plugin, it looks like:

And here’s Clover:

The Emma and Cobertura reports include metrics on classes, lines, and methods. Cobertura also has conditional statements and files while Emma includes blocks. Conditionals could be interesting, I guess, if you notice a big coverage gap. Clover’s ‘elements’ appears to be lines and methods. Cobertura has nice drill-downs and Clover has not just drill-downs but tree map and word cloud charts as well.

Aside from graphics, I also looked a little closer at how some specific code elements are handled.

Static nested classes

The code under test for the result page below has an internal builder class. If you’ve never noticed, the compiler creates an extra $1 class for you behind the scenes. Emma reports the SampleBuilder$1 class as uncovered:

So somehow you have 100% coverage of blocks, lines, and methods, but only 67% coverage of classes… Cobertura excludes SampleBuilder$1 in its class coverage to give you 100%, but if you drill-down, you see the class with N/A in its line and method coverage columns, which seems like a good solution. Clover does not show the $1 class at all, which is also fine.

Enums

Emma dings you if your tests don’t explicitly test the built-in enum valueOf and values() methods- they show up as 0% coverage. Like with the inner classes, Cobertura will show those methods but mark them as N/A and not include them in your method coverage percentage. Clover will not show them at all.

Annotations

Cobertura will show you an annotation (an @interface that you’ve created) with N/A. It won’t appear at all in Emma. It also won’t appear in Clover, but curiosly, I did notice it shows up in a word cloud chart of top project risks.

Default no-arg constructor

If your class doesn’t define its own constructor, but a test calls the implicit no-arg constructor, does it count in the coverage? For Emma, yes. It lists the constructor as classname(), eg. Sample() and gives you credit for covering one method and one line. Cobertura will show the constructor as <init>() but give it an N/A for coverage. Clover won’t show an undeclared default constructor at all.

Conclusion

These are all good tools. In Eclipse, I’m mostly concerned with finding untested lines and getting a general sense of how much code is covered so I think I’ll keep using EclEmma. For a full report, Cobertura has a nicer look and fewer coverage oddities to note. Also, even though new plugins have come out recently for Emma, it doesn’t appear that the main project is still being actively developed. As far as speed goes, we try to keep our unit tests very fast anyway, so it was not a big concern and I didn’t bother investigating it.

Clover looks goods visually, has lots of options, and I like how it handles results. If money were no object, I’d definitely look more closely at it, but for our situation, the free tools work just fine.

This entry was posted in java, software and tagged , . Bookmark the permalink.

10 Responses to Java Code Coverage: Cobertura vs. Emma vs Clover

  1. Sudha Kannan says:

    Another important thing to note is that Emma detects partial coverage..in the sense that, a conditional statement like , if(A equals B) OR (B equals C), would be reported as covered by most tools if one of the condition passes. However Emma reports this as 50% covered. Besides this, the size of the instrumented classes for Emma is much lesser than what it is for Cobertura. Using Emma, on a project of size 1.58MB made the instrumented code 1.79MB, whereas Cobertura made it upto 2.14MB. That gives Emma an added advantage

  2. Pingback: Pedro Newsletter 26.10.2010 « Pragmatic Programmer Issues – pietrowski.info

  3. neo says:

    cool got this
    i’m a basic beginner of java looking for it

  4. Keith says:

    Nice article! I was looking for this exact comparison. I also wanted to note that Google’s CodePro suite of tools uses EMMA on the backend.

  5. Raymond Naseef says:

    Hi, excellent article, great quick reference for comparing the tools ;)

    Annotations states Clover 2x, and misses Cobertura:
    Clover will show you an annotation (an @interface that you’ve created) with N/A.
    It also won’t appear in Clover…

    Is one of those 2 supposed to be Cobertura?

  6. johnstosh says:

    I use NetBeans because it is so awesome and easy to install. I see that Cobertura has plugins for NetBeans; perhaps Emma does as well.

  7. Applebee says:

    what is the Eclipse plugin for Emma, is it EclEmma or Emma have its own seperate plugin?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>