Tag: software testing

I am a big advocate of testing your code, including your database code. I like repeatable testing, especially unit testing. I think this has contributed to the increase in quality over the last ten years as more developers have incorporated unit testing into their work. That, along with the increase in standard frameworks, seems to have resulted in less crashes and instability in much of the software I used today.

That being said, the client ultimately decides if software works as expected. Certainly there can be logic errors, but there also could be errors in how specifications and requests are interpreted. This is one reason we need humans to do some QA testing and clients to ensure there is some user acceptance review.

Apparently that didn’t happen with one election machine. There is a hash that is supposed to be used to verify that the correct version of software is installed. However, if that reference hash isn’t there, the machine still reports things are fine. Certainly an issue that is a problem, though not necessarily one that users would detect. We would expect someone that purchases, updates, or administers these machines to check for the correct version.

That isn’t an issue here because the acceptance testing was done by the vendor. While I am a big advocate of developers checking their work, there needs to be independent evaluation by a CI process, and there ought to be some QA review by another group. Certainly a client ought to be able to double check the software as well, and some client should have done acceptance testing here.

As the world moves more towards DevOps type software development, we need better testing, and that likely needs to include some independent testing outside of the developers and testers. I certainly could see the need for clients to submit some tests and get some verification that those tests passed. Automated CI/CD systems can do this, and provide detailed logs of what happened.

Ultimately we may still have bugs, either because we don’t have enough testing, or we don’t quite write the code that does what client expects. We may also have performance issues, but that’s another testing issue.

We can get better, but we have to work to do so, learning from our mistakes, ensuring we are always improving testing, and listening to feedback.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

One of the areas where machine learning and artificial intelligence have had lots of success is with image work. Whether identifying people in pictures or helping cars stay on the road and out of each other’s way, this capability of computing has worked well. It’s not perfect, and not necessarily as accurate as most humans, but it works well. At least well enough. Sometimes it’s even better than humans.

There are issues, however, and I think some of them are because of poor data sets. Last year when the pandemic hit, education was challenged with how to conduct remote exams. While there are some solutions, they don’t always work well. Sometimes the algorithms don’t recognize people, especially non-Caucasians.

The issues raised reminded me of the issues with some bathroom gadgets. I have fairly dark skin, and I’ve always wondered why some sinks and soap dispensers wouldn’t work for me. I hadn’t thought much about it until I saw a few reports like the one listed above.

I don’t think there is anything malicious here, but I do think that often we find teams work on a happy path when building some new tool. They test it often themselves, but they don’t think widely about how a variety of customers will use things. While I’ve seen many personas, I often don’t see anyone creating personas that might consider something like skin color, or even a different culture. We often consider roles, without deeply examining how those roles are implemented.

We need to work with representative data in whatever area we work, but data that does include some of the edge or corner cases that might come up. Our dev and test areas can start with small data sets, including those that we build, but at some point we need representative data. Whether we’re building OLTP software, sensors, or image recognition, our data should be well rounded.

While systems don’t need to solve every issue, we ought to consider a large percentage. In the case of imaging, certainly understanding the wide variety of type of people that can use products would seem to be important. Hopefully future teams won’t make the mistake of assuming that most of their customers look exactly like them.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

This seems like something out of a thriller that Hollywood made for a spy. Someone switches two cables in a rocket and then it crashes. However, it’s what happened recently, not from a spy, but an employee somewhere connected two cables to the wrong connectors, resulting in reverse commands being sent to the rocket.

That sounds crazy. Who would plug cables in backwards, and how is this not caught? Surely we wouldn’t run into this in software would we? With all the unit testing? Well, we did once have an English->metric error.

Actually, I’ve seen this in software. I’ve seen someone pass parameters in backwards to a method or stored procedure. I think I’ve done this before as well. While I hate typing out named parameters, I do appreciate SQL Prompt for ensuring I connect the right value to the right parameter by spelling them out.

Do you type out parameter names, as in EXEC myProc @Param1 = @value1, @param2 = @value2? It’s a good habit to get into to prevent simple mistakes. In fact, this is where some simple, quick tests can catch errors. Not only will this catch some weird changes, this can ensure that your procedures are backwards compatible. That’s often a concern in refactoring, where someone might not think a parameter is being used, or wants to replace it and then breaks other code.

Testing is something that has improved in software dramatically over the years, though not as much in databases as I’d like. I urge you to use more procedures, but also add more tests and ensure that you don’t make simple errors that should be caught.

Steve Jones

Listen to the podcast at Libsyn, Stitcher or iTunes.

Performance tuning is a challenge for many data professionals. Often, it’s also a task that we struggle to find time to perform in many environments. Developers have new features to build and DBAs have plenty of other work. As a result, performance testing is usually done in a crisis.

Netflix is always looking at performance, since their customers and audience are very sensitive to delays. They wrote a post on examining the performance of a potential technology change, which was a good look at different ways they test the effects of something new. In his case, they talk about the changes they were considering for networking stack. There is some technical discussion of why, but the interesting part of the piece, for me, was the A/B testing section.

They planned an experiment, conducted it, and then measured the results. This wasn’t just a test for a developer workstation, which I’ve seen most people do. This was a test with half a million users. Netflix has over 150mm users, so this isn’t a significant number, but it’s also not a tiny number. It’s enough to look for potential issues, though I’d hope they’d expand this to a 2-5% of users to verify their results before they deploy to everyone.

Too often I see developers assuming a test on their workstation of a very small set of data determines if their approach makes sense. I do believe that is a good place to start, but before getting too far along deploying your changes, some significant scale test ought to occur, with repeatable measurements. If you want to ensure your auditing trigger or update code or anything else that might impact lots of users will cause issues, test on a larger set of data. Then repeat the test and verify that your results make sense.

Testing is a skill. A bit of an art, but very much a science. You should make sure you use test harnesses that are large once you think your code works, just to be sure it does. If you want ideas on how, Jeff Moden and Dwain Camps have a few articles that might help.

Steve Jones

Listen to the podcast at Libsyn, Stitcher or iTunes.