Voice of the DBA

Tag: DevOps

  • The Change Failure Rate

    One of the measurements used in DevOps to determine if your team is improving is the change failure rate. This is the number of times that there is a problem with a deployment as a ratio of the total number of deployments. The idea here is to determine the reliability, and to some extent, the risk of deploying changes in your environment.

    In the past, I’ve been very successful with deployments as a DBA or developer. Often I’ve completed database changes within the change window, with success that allowed our applications to run after changes. That doesn’t mean I pressed a button and the deployment worked. On a regular basis, my expertise with SQL was needed to fix a script or re-run a process, or make some other “development change” in the production environment to ensure the entire deployment completed. While still a minority of times, this wasn’t an uncommon experience.

    Many people have had the same experience, as the State of DevOps report has shown for years. In both the application and database worlds, doing anything other than pressing a button or running a single script is a deployment failure if the steps aren’t clearly documented and run to completion without any alteration. Not a catastrophic failure, and certainly one that many of us can recover from, but still a failure from the standpoint of being ready to deploy and having a reliable process.

    Quite a few of us have made a career of cleaning up other people’s messes in deployments, with our ability to get changes deployed and applications back up and running being a testament to our expertise and skill. That’s not a reliable process, especially when an organization is forced to depend on a Steve or a Brent or a Kendra to ensure smooth deployments. That’s a recipe for disaster, especially as all of us want to go on holiday, undisturbed by some software “emergency”.

    I think this is one of the more telling metrics for a strong software development process and a reliable deployment process. If this number isn’t extremely high, in the 95-99% range, then our organizations are spending resources, especially time, on items that don’t add value. Instead, we ought to invest in moving to a DevOps style process that allows our expertise to be used solving new problems, not cleaning up the mess of the poor development practices of others.

    Steve Jones

    Listen to the podcast at Libsyn

  • Really Important Software

    I remember reading about the Boeing 787 (the Dreamliner) being designed. I was fascinated by the process and excited to fly one. I finally did and enjoyed it. Since then, I try to fly on those when I can as the spacious ceiling and other changes seem to make for a better flight. I impressed with the work they did and the scale of the changes, both in manufacturing and software. There’s a short timeline here and if you ever get the chance to take a tour of the factory, do it. It’s very cool.

    I fly a lot. Last year I took around 50 flights on airplanes, some of them Boeing, some Airbus, some 737s, 777s, 787s, A350s, and a few others. I haven’t flown on a Boeing 737-Max 8, though I did just recently fly a 737-800. When a few of those planes crashed, I somewhat chalked this up to pilot training and a lack of options on the planes. I’m not exonerating Boeing or the airlines, but I wasn’t personally concerned. I saw an interesting piece at Vox, but was still curious about the issues. After all, I’m in software, and I fly a lot.

    There’s an analysis of the crash from a software developer at the IEEE site that’s worth reading. It’s a look at the issues, the potential ways things could have been addressed and, perhaps most importantly, the idea that our software is not necessarily going to fix problems. Software is a way of reading data and then affecting changes in the hardware, but software isn’t perfect. One of the quotes in the article puts this in perspective for me: ” Every time a software update gets pushed to my Tesla, to the Garmin flight computers in my Cessna, to my Nest thermostat, and to the TVs in my house, I’m reminded that none of those things were complete when they left the factory—because their builders realized they didn’t have to be complete.”

    Data matters, but as we all know, data is sometimes incorrect, jumbled, or otherwise problematic. In this case, not only does it appear the data could be incorrect, but the plane software isn’t looking at all the data, and it certainly doesn’t take all the inputs from the pilots. While humans make plenty of mistakes, they need to have inputs and overrides in software systems.

    I don’t want to get caught up in the issue with these planes, but rather look more generally at our systems. DevOps looks at the idea that we don’t know everything about how to build our software when we start. We definitely know it is not complete and will need more work when it leaves our factory, but we also know that this is the case. We know it’s not complete. We take feedback, learn, and adjust what we do. We constantly drive quality up, or at least, that’s the goal. Money and politics can get in the way, but the more they do, the less you’re doing DevOps.

    Steve Jones

    Listen to the podcast.

  • More Software Quality with DevOps and CD

    Tomorrow is the SQL in the City Summit in London. Right now I’m in a Heathrow airport hotel, finalizing some prep for the event. I have to run a panel as well as give a talk, Building quality into your continuous delivery process, that will cover some of the ways that we can improve quality by moving to DevOps.

    That’s a passion of mine because I want better software quality. I know it’s hard, believe me, after the SQLServerCentral migration, I remember how hard it is. I also know that this was a few steps forward with a few backward, and across time, this is a big move forward.

    That’s how DevOps will go, especially as you implement it. That was some of what we did, with a complete change in underlying architecture, a software pipeline, and new methods for deploying changes. It’s painful, and we likely went too soon, but we also had external constraints that forced some change.

    Hopefully I’ll see some of you tomorrow and have some advice for you on how to improve quality within your organization. Culture is the big item, and I talked a bit about this at SQL in the City Streamed. Watch that and let me know what you think.

  • Enter our Database #DevOps Contest

    Share your story and you could win. Enter as often as you like before Mar 20.

    https://www.red-gate.com/hub/entrypage/competition