Voice of the DBA

Tag: software testing

  • Don’t Use Code Coverage

    I got asked in a seminar recently how to perform code coverage for T-SQL. There aren’t a lot of options, but there is a project from Ed Elliot called SQLCover that looks to examine how much of your code gets called from a test. There may be other methods, but my question back was why does code coverage matter? There were a few reasons given by various people, but ultimately the original questioner said that their boss wanted a report.

    That’s fine, and it’s a valid reason to try and measure code coverage, but is that an effective use of developer time? I don’t think so, especially as trying to meet some code coverage goal is fraught with all sorts of issues. Apart from various good points on Stack Overflow, no shortage of people have blogged about the problems of relying heavily on a code coverage metric. I won’t repeat all the arguments, but I’ll give a few of my own.

    First, someone pointed out that you want to be sure all your code is tested. I disagree with that, especially in many  long dev cycle applications. There are all sorts of enhancements I’ve seen in many applications that never get used. While you should certainly check them for security issues, if the features aren’t being used, remove them. Since most of you don’t implement tests for security issues, I’m not sure there’s value in telling you to write a unit test for logic or functionality.

    The other issue with code coverage is that anyone other than a developer using this metric will put pressure on the developers to write more tests. Some developers may write good tests, but I wouldn’t be surprised to have tests written that always pass and aren’t checked for correctness. Tests like these are misleading and potentially create more bugs since anyone refactoring code might think their changes haven’t caused an issue if the test passes. Bad tests, IMHO, are worse than no tests.

    I do think that code coverage can be valuable. If a developer is looking to ensure that some part of their code is being checked, especially in long stored procedures, then code coverage can guide them to a place that might need  testing. Since most code needs more than one test, covering more than one case, code coverage doesn’t help. I’d prefer a developer spent more time thinking about how to build better tests (or experimenting) than looking at coverage reports.

    And if you need a code coverage report, try my technique for building one

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Why Don’t You Unit Test Code?

    First, I’ll apologize to those that do use formal, automated tests in advance. I’m not really writing for you, and I’m sure you can teach me more about testing. Perhaps you can comment about how and why you do this, or even write some articles to help others.

    For everyone else, why don’t you test in a repeatable fashion? Let me know with a comment today.

    One thing I’ve noticed as I’ve talked DevOps and testing for a number of years is that so many developers agree testing is a good idea, but claim they have no time, no support from management, or no need to repeat tests once code is written. Sometimes I hear all three excuses at the same time. The same people will also admit that they have problems with code at times, and that quality could be improved.

    Perhaps it’s me, but I’ve never had my boss sit there and watch me code. He or she doesn’t know how I might write code. I certainly write a query, run it, and look at a bunch of rows and see if I think the code is correct. Perhaps I need to run other queries to verify values in the results. However, I could also build a quick test with something like tSQLt, like the one I set up for checking COALESCE that does the same thing. With my unit test, I can run it over and over, modifying the test as I discover new edge cases to validate. Which way do you want to write code?

    Even if I have a set of data I need to assemble, it doesn’t take much longer for me to write a test, especially when I need to also solve the problem. I find that putting a test together forces me to slow down and think about the rules for my code. Since I need to come up with some result set, building that into a test of some sort is fairly easy and let’s me double check how I think data will change. I can then repeat the test over and over as I modify code. I can even grab sample data from production and use that as the basis for my unit test if I need to check a specific entity’s values.

    I know many things I write in SQL might not be worth a test. Often the simple CRUD queries and basic aggregations aren’t things I’d test, but when my query becomes complex, includes APPLY or outer joins, and logic to decide how to filter or format data, wouldn’t a unit test make sense? What about if you want to be sure you’re processing the correct rows from a large CUBE or ROLLUP operator? Certainly if I’m fixing a bug, writing tests makes sense, at least I think so.

    Let me know what you think today.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Less QA?

    This editorial was originally published on Jul 31, 2013. It is being re-run as Steve is out of the office.

    Throughout most of my career, people have complained that software is never tested enough, not tested well, and certainly not enough time is allocated in project plans. I’d also argue that all too often we don’t have people building software with experience in how to perform extensive testing. Most developers only have a rudimentary knowledge of software testing and that part of their skill set doesn’t receive constant training. However I’ve found plenty of QA people that were in the same situation. Some educate themselves and learn how to test better, but many go through the motions. They don’t take pride in their role as a software tester.

    There have been lots of advances in automated testing and TDD methodologies over the years, which means that developers have been taking more of the responsibility, and effort, for testing code. It’s hard to know if this has resulted in higher quality code, lower quality, or a mix. Overall I think that the latter is likely. Code quality is all over the board, probably based more on the skill and talent of the individual developer than on any process.

    That’s good for the companies that have hired strong developers, but not so great for our industry as a whole. We still produce lots of software that takes too long to develope, costs too much, and often has too many bugs. Overall we are sloppy and inconsistent in how we build software. Some of that is the nature of our business, one with such a low bar of entry that anyone can give software development a try.

    I don’t know how we improve the quality of software, though I do think continuing to educate and train people on what works well is a good start. However I also think that we still need QA groups. We need people that will use software in ways that automated tests won’t. People that will press buttons, enter values, and push software in ways that developers would never consider using it. Most of all, we need QA and testing to be treated as a valuable part of our industry. We need QA people that view testing as a valid career path, not a stepping stone for junior developers that look to move on as soon as possible.

    Steve Jones

  • Test Coverage

    This editorial was originally published on Nov 12, 2013. It is being re-run as Steve is on holiday.

    I’ve never had to work full time in a QA group, but I have had to perform software testing of an application and it wasn’t fun. Even as I worked through the various features, looking for edge cases, common mistakes, etc., I realized that running tests was something that really required more discipline and process than I was giving it at the time. Too often I ‘d realize that my regressions weren’t 100% duplicates of previous executions because I’d allowed too much “human effort” into my process.

    These days most testing of software is automated. I know QA groups still exist, and they need to handle some of the manual checks that are very hard to automate. However more and more testing is being pushed back onto developers to handle, with frameworks like NUnit and JUnit. In the database world, we haven’t done a great job of including testing into the code we write, but there’s a great testing framework we can use.

    TSQLT is a framework written by Sebastian Meine and Dennis Lloyd and it’s free. It’s been developed to help you write tests that can exercise your T-SQL code to determine if it’s doing what you expect. I’ve used it a little, and while I see the potential, I also realize that this will take some practice to learn how to more easily write tests that can cover the various potential places where bugs can be introduced.

    However, if you use version control, and you should, then you can easily spread the load of writing tests to all of your developers. Since the tests are stored procedures, they can be included in your VCS project and shared by all your developers. With a little effort from each member of your team, you might be surprised at the code coverage you can achieve with this framework.

    Testing is important, as we see over and over again as software is released and doesn’t work as expected. I think much of that is our fault, as software developers, for not improving our testing skills and discipline.

    Steve Jones