Voice of the DBA

Tag: DevOps

  • Microsoft, Think DevOps First

    This post from Melissa Coates is a good example of not thinking through your product architecture early on. The short version of the post is that Power Bi Desktop is a better place, perhaps really the only place, to author your Power BI Reports. Melissa covers a few potential issues while using the Power BI online editor, but the big item is that you cannot easily prevent conflicts and track versions there.

    Power BI started as the online service, with the Power BI Desktop tool seeming like a bit of an afterthought. At least from the marketing perspective as all of the early demos and media from Microsoft was about the online report service. Since then it’s evolved to become the primary way we will build reports in SQL Server moving forward, which I think is a good move.

    While I can understand the developers at Microsoft not really thinking that the service would on-premises and maybe not considering the need to provide some text format for the reports, I can’t understand why they didn’t learn from the Integration Services team and realize that binary versions of programmable items don’t make sense. We need a format that can be easily versioned, and maybe more importantly, stored in a VCS and diff’d in a way humans can understand.

    When building a format for storing code, please consider the need to work in a team and version the changes made. This means any format should not be a) binary and b) difficult to decode. Separate visual elements from logical elements and ensure a text version of this can be examined by developers. You can use XML, JSON, YAML, or any other text based format, but choose something that makes sense. Even if you add your own extension, ensure that standard tools that work with code can use this.

    I do know the PBIX format is a ZIP file, but zip files don’t easily integrate with a VCS. We could use hooks to extract/rebuild out files on commit/checkout, but that’s cumbersome and silly. I’d rather that the PBIX were a folder with the files inside. Users, including my Mom, can zip a folder and email that if needed. To me, that would have been a better structure from the beginning.

    Microsoft is supposed to be a company providing platforms that we build upon and use in our work. The decisions for the Power BI service seem to be poorly thought through with that in mind. I’d urge them to create a baseline set of rules for future products that consider DevOps, teams, and the need to track code.

    Steve Jones

  • Practice Those Scripts

    I’ve written lots of scripts that were deployed to production. I’ve often had another set of eyes look them over, and still, we made mistakes. In fact, a recent Salesforce outage was blamed on a poorly written database script that gave users more rights than they should have gotten. There wasn’t an actual outage caused by the script, but since customers might have been able to see data and change from other customers, Salesforce took its own service down to prevent anyone from doing so.

    I’m a big fan of DevOps, and certainly including the database in a DevOps process to build a better software development flow. Part of that is ensuring that you can deploy by practicing the act multiple times. In a database world, this would mean that we run a script not just on a development server, but on a QA server, on a staging server, on any other environment we can find to practice and test the deployment. At that point, we should be confident of execution on the production system without issues.

    Good in theory, but sometimes you can’t easily test scripts in intermediate environments. I think changing security is a place where it can be hard to actually test things, especially if specific accounts are referenced that might not need or have access in that environment. Certainly some data changes might be easily be tested in intermediate environments, especially when these refer to configuration differences, like email or messaging systems.

    In this case, I suspect the “access changes” were data changes that updated values in certain tables in the Salesforce application. In that case, why wasn’t this tested? A restore of production to a staging environment would allow developers to test their script. It’s not multiple executions on intermediate servers, but it is better than nothing.

    I’m sure many of you have had the need to execute scripts to change data, alter permissions, or something else in production. Could the same thing that happened to Salesforce happen to you? What precautions do you take, or what would you recommend to prevent this type of issue. Let us know today.

    Steve Jones

    Listen to the podcast at Libsyn.

  • The Change Failure Rate

    One of the measurements used in DevOps to determine if your team is improving is the change failure rate. This is the number of times that there is a problem with a deployment as a ratio of the total number of deployments. The idea here is to determine the reliability, and to some extent, the risk of deploying changes in your environment.

    In the past, I’ve been very successful with deployments as a DBA or developer. Often I’ve completed database changes within the change window, with success that allowed our applications to run after changes. That doesn’t mean I pressed a button and the deployment worked. On a regular basis, my expertise with SQL was needed to fix a script or re-run a process, or make some other “development change” in the production environment to ensure the entire deployment completed. While still a minority of times, this wasn’t an uncommon experience.

    Many people have had the same experience, as the State of DevOps report has shown for years. In both the application and database worlds, doing anything other than pressing a button or running a single script is a deployment failure if the steps aren’t clearly documented and run to completion without any alteration. Not a catastrophic failure, and certainly one that many of us can recover from, but still a failure from the standpoint of being ready to deploy and having a reliable process.

    Quite a few of us have made a career of cleaning up other people’s messes in deployments, with our ability to get changes deployed and applications back up and running being a testament to our expertise and skill. That’s not a reliable process, especially when an organization is forced to depend on a Steve or a Brent or a Kendra to ensure smooth deployments. That’s a recipe for disaster, especially as all of us want to go on holiday, undisturbed by some software “emergency”.

    I think this is one of the more telling metrics for a strong software development process and a reliable deployment process. If this number isn’t extremely high, in the 95-99% range, then our organizations are spending resources, especially time, on items that don’t add value. Instead, we ought to invest in moving to a DevOps style process that allows our expertise to be used solving new problems, not cleaning up the mess of the poor development practices of others.

    Steve Jones

    Listen to the podcast at Libsyn

  • Really Important Software

    I remember reading about the Boeing 787 (the Dreamliner) being designed. I was fascinated by the process and excited to fly one. I finally did and enjoyed it. Since then, I try to fly on those when I can as the spacious ceiling and other changes seem to make for a better flight. I impressed with the work they did and the scale of the changes, both in manufacturing and software. There’s a short timeline here and if you ever get the chance to take a tour of the factory, do it. It’s very cool.

    I fly a lot. Last year I took around 50 flights on airplanes, some of them Boeing, some Airbus, some 737s, 777s, 787s, A350s, and a few others. I haven’t flown on a Boeing 737-Max 8, though I did just recently fly a 737-800. When a few of those planes crashed, I somewhat chalked this up to pilot training and a lack of options on the planes. I’m not exonerating Boeing or the airlines, but I wasn’t personally concerned. I saw an interesting piece at Vox, but was still curious about the issues. After all, I’m in software, and I fly a lot.

    There’s an analysis of the crash from a software developer at the IEEE site that’s worth reading. It’s a look at the issues, the potential ways things could have been addressed and, perhaps most importantly, the idea that our software is not necessarily going to fix problems. Software is a way of reading data and then affecting changes in the hardware, but software isn’t perfect. One of the quotes in the article puts this in perspective for me: ” Every time a software update gets pushed to my Tesla, to the Garmin flight computers in my Cessna, to my Nest thermostat, and to the TVs in my house, I’m reminded that none of those things were complete when they left the factory—because their builders realized they didn’t have to be complete.”

    Data matters, but as we all know, data is sometimes incorrect, jumbled, or otherwise problematic. In this case, not only does it appear the data could be incorrect, but the plane software isn’t looking at all the data, and it certainly doesn’t take all the inputs from the pilots. While humans make plenty of mistakes, they need to have inputs and overrides in software systems.

    I don’t want to get caught up in the issue with these planes, but rather look more generally at our systems. DevOps looks at the idea that we don’t know everything about how to build our software when we start. We definitely know it is not complete and will need more work when it leaves our factory, but we also know that this is the case. We know it’s not complete. We take feedback, learn, and adjust what we do. We constantly drive quality up, or at least, that’s the goal. Money and politics can get in the way, but the more they do, the less you’re doing DevOps.

    Steve Jones

    Listen to the podcast.