DEVEL::COVER – ARE YOU COVERED?

The Product Management team recently built a new Catalyst Perl application and we wanted to ensure that we would maintain a high level of code coverage in our test suite. This seemed like an ideal opportunity to use Devel::Cover; we currently use Jenkins to build all our branches, and Devel::Cover allows us to integrate easily with Jenkins to run the coverage report after each build.

What is Devel::Cover?

Devel::Cover is a tool to measure code coverage in Perl. It measures how the tests you have written exercise the code in your application. It helps you identify areas of code that are untested and shows you where you need to concentrate on providing better test coverage.

One of the features it provides is to generate a coverage report, the cover program allows you to customise these reports (HTML, textual, etc). This provides clarity and a meaningful display of the test coverage by creating a graphical representation of the test coverage. The main report we use is the HTML output that we have configured with Jenkins so that at the end of a successful build of a feature branch we get the option to view the coverage summary, which allows us to see whether any test additions/changes have improved or diminished the test coverage.

Coverage Thresholds

We have chosen a threshold such that, if the test coverage falls below this threshold, we will automatically fail the build. This ensures we maintain a high level of code coverage within our tests. If you’re thinking of integrating Devel::Cover within your legacy applications that have large test suites, then you may decide to reduce the threshold to a lower value, increasing it as and when your test coverage improves.

You can think of the coverage threshold as a percentage of the code parsed by Devel::Cover. We set it in Jenkins as follows:

COVERAGE_THRESHOLD=95 HARNESS_PERL_SWITCHES=-MDevel::Cover prove -lrv --timer --harness=TAP::Harness::JUnit t/

You can also generate reports in development using Dist::Zilla, e.g. dzil cover -report html, will provide you with the HTML report.

It is important to note that the coverage level shouldn’t be relied upon as the sole measure of quality. It’s easy to reach high coverage levels with low-quality testing. Devel::Cover will help you identify areas you may want to improve, and with the help of the HTML coverage report mentioned above you can focus on testing your untested code.

Coverage Report

The report shows a list of all your files including test and template files, along with various columns for different types of coverage (statement, branch, conditionals, subs, pod and time taken to run). Each of these columns has a total value across all files covered in the report. You can customize what files you want Devel::Cover to omit from the report as it may skew the total coverage if you have spreadsheets, text files, etc. within your code checkout. To do this, you can use the -ignore flag with a regex to match the files you want to omit.

HTML Report – Columns Explained

Paul Johnson’s Devel::Cover::Tutorial gives an introduction of the different types of coverage and what they cover. This proves useful when trying to understand the HTML report and all its columns. I’ve attempted to provide a brief one-liner for each of these; for more information, see the Devel::Cover::Tutorial and Devel::Cover on the CPAN.

Statement coverage checks if a statement is executed. If so, then it’s covered. It is not a comprehensive measure of coverage; even with 100% statement coverage there may still be problems with your code.

Branch coverage checks all the places in your code where you have conditionals. In the case of an if () {} else {}, the branch coverage checks whether your test covers both branches. In other words, you have tested for both the positive and negative conditions.

Conditionals coverage checks for uncoverable conditions. Devel::Cover gives you the option to tell it of known uncoverable conditions, and it will not count these in your report if specified using a special comment:

# uncoverable [% statement details %]

Subroutines coverage checks a subroutine is being tested. It marks the sub as uncoverable if the first statement is marked as uncoverable.

Pod coverage checks you have suitable pod documentation for all your subs within your modules.

Conclusion: Stop the obsessing, they’re just numbers!

Devel::Cover is a starting point to help you develop applications with a high level of code coverage, giving you more confidence in your application. The coverage report is a powerful tool but you can easily start obsessing over the numbers in the report. In your journey to the promised land of 100% test coverage, you may waste a lot of time and effort for very little gain. There is value in tracking coverage over time by setting a threshold and having your test builds fail if the coverage falls below that threshold. Devel::Cover is also good at highlighting untested code within your application but it should not be seen as a tool for checking test quality. The generated report allows you, as a development team, to focus on areas with low test coverage and you can run the report as often as you like.

Print Friendly

Comments are closed.