Don’t Use Code Coverage

I got asked in a seminar recently how to perform code coverage for T-SQL. There aren’t a lot of options, but there is a project from Ed Elliot called SQLCover that looks to examine how much of your code gets called from a test. There may be other methods, but my question back was why does code coverage matter? There were a few reasons given by various people, but ultimately the original questioner said that their boss wanted a report.

That’s fine, and it’s a valid reason to try and measure code coverage, but is that an effective use of developer time? I don’t think so, especially as trying to meet some code coverage goal is fraught with all sorts of issues. Apart from various good points on Stack Overflow, no shortage of people have blogged about the problems of relying heavily on a code coverage metric. I won’t repeat all the arguments, but I’ll give a few of my own.

First, someone pointed out that you want to be sure all your code is tested. I disagree with that, especially in many  long dev cycle applications. There are all sorts of enhancements I’ve seen in many applications that never get used. While you should certainly check them for security issues, if the features aren’t being used, remove them. Since most of you don’t implement tests for security issues, I’m not sure there’s value in telling you to write a unit test for logic or functionality.

The other issue with code coverage is that anyone other than a developer using this metric will put pressure on the developers to write more tests. Some developers may write good tests, but I wouldn’t be surprised to have tests written that always pass and aren’t checked for correctness. Tests like these are misleading and potentially create more bugs since anyone refactoring code might think their changes haven’t caused an issue if the test passes. Bad tests, IMHO, are worse than no tests.

I do think that code coverage can be valuable. If a developer is looking to ensure that some part of their code is being checked, especially in long stored procedures, then code coverage can guide them to a place that might need  testing. Since most code needs more than one test, covering more than one case, code coverage doesn’t help. I’d prefer a developer spent more time thinking about how to build better tests (or experimenting) than looking at coverage reports.

And if you need a code coverage report, try my technique for building one

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

About way0utwest

Editor, SQLServerCentral
This entry was posted in Editorial and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s