Voice of the DBA

Tag: DevOps

  • Are Indexes Actually Changes to the System?

    I haven’t thought about this in some time, but Brent Ozar raises an interesting question: should index changes require change control? I’ve worked in an organization that didn’t consider an index change to be an item that was presented as an official change. They were seen somewhat like adding new users or changing a firewall rule. These were ticket requests that could be triaged and made by an individual group without notifying the rest of IT.

    I’ve also worked in organisations where an index change was viewed as a deployment and subject to change control procedures. Not that the network or security people cared about the details of the index, though the help desk might. These just wanted to be aware as any deployment change could affect their departments.

    In today’s world where many companies seek to adapt faster to changing customer demands or market opportunities, I think the idea of meeting every week to discuss any upgrades or changes to applications is somewhat silly. Certainly large changes, like major application upgrades to ERP systems, new security hardware, and alteration of core infrastructure ought to be debated in a larger group. Deploying changes to an application isn’t one I’d think deserves the debate. At least not if your organization is trying to become more efficient.

    I do think a way to avoid issues is to work more closely with the development group if you are in operations or vice versa. I’d be sure to let other app teams know about index changes? Certainly Brent lists a number of potential problems, many of which could occur. A new feature might break and prevent inserts, exports could use more disk space, imports might eat up data and backup space, queries could cause slowdowns or blocking. These are all possible, maybe even probable.

    I agree with Brent in that any changes need to be tested. I don’t know about any sort of formal change control, but there ought to be a process that evaluates the changes and mimics them in an environment that allows you to determine the production impact. Using a known, automated process with testing and well documented deployment actions will reduce the risk, but perhaps not eliminate it. However, when you find issues, if you have an automated process, it’s easy to add tests or alter the process to ensure that it will succeed in the future. Especially if you examine the effects of these changes and learn to avoid problematic development or deployment patterns in the future.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Azure DevOps

    I’ve been a fan of Visualstudio.com and VSTS for some time. I moved most of my demos to this platform a couple years ago and I’ve been pretty happy with it since then. There is tremendous flexibility in how you can use automation and dashboards to build software and coordinate the activities of your team.

    The VSTS system underwent a rebranding and reorgnaization renently. The various pieces of the system were renamed a part of Azure DevOps, with the different parts being given new monikers such as Azure Pipelines, Azure Boards, and more. This was combined with an initiative from Microsoft to better support open source projects by giving them unlimited build minutes for public Github repositories on a variety of platforms such as Windows, OSX, and Linux.

    I’m not a bit fan of name changes, as I think that if the software performs well and provides value, it will succeed. However, marketing people need work, too, and management inside a company often rearranges things to put their own mark on a project. I’m actually glad the marketing effort was ramped up as I think the Microsoft platform based on TFS for tracking work, version control, builds, and releases has become a fantastic platform for anyone building software. That’s not to take away from some other products like Bamboo and Octopus Deploy, which might work better for you. If they do, they plug into the Azure DevOps platform easily.

    If you haven’t tried Azure DevOps, I’d urge you to give it a try. There’s an all day recording of various parts of the system being used to produce software that will show you how to get started and use the system. I’m sure there will be more information and talks this week from Ignite. There aren’t a ton of database tools, but there are some add-ons from various companies to help you build and deploy databases alongside your application software.

    While there are challenges with databases, I’d argue that incorporating a known process will increase reliability and lower risk for making changes. This won’t help you build better code. That’s something you still need to ensure your developers are doing. This system just helps ensure simple, silly mistakes aren’t made and everyone knows exactly how your changes will be deployed to your production environment.

    Give Azure DevOps a try today and see how you can build a smooth, repeatable, reliable process for your software.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Rollbacks Are Normal

    Do any of you think that a rollback of a version for your application is easy? Most people struggle, and when I talk DevOps and improving your process, the number one question is about rollbacks. In fact, recently I had a few people that struggled to even listen to the early parts of DevOps because all they could think about were previous failed releases and the need to roll back. They kept saying, what about rolling back. Hopefully you can stick with me a bit longer.

    I’d argue that rollbacks are the process that needs DevOps more than ever, with smaller, more regular releases and practice at making changes. Whether forward or backward, we ought to be able to rev our software easily. I ran across a piece from the cloud platform blog at Google called Reliable releases and rollback – CRE life lessons. The title is an interesting one, but suspend some of your database skepticism until the end.

    It’s easy to consider rolling back in the early parts of the article and say “it’s way easier to roll back your application”, and it is. Applications just stomp down new (or old) versions on top of what’s there, which is often an easy thing to do. As they say at Google, “rollbacks are normal”, which certainly seems to fit with the application paradigm.

    In fact, they recommend rolling back a good release. After all, it’s much easier to practice this sort of thing when you have a working new version of software. When the release breaks your system, as mentioned in the piece, everyone’s stress level rises and the fixes often aren’t well built. Even when they work, which isn’t anywhere near all the time, there are often problems later. The idea should be to roll back and ensure everyone knows how to undo a version change. They can document the reasons for rollback and get the previous state of the application running. I hadn’t thought about this, but it makes sense. Practice in advance and be prepared. You can always re-deploy the working version.

    What about databases? They have a solution, and I like it. They want the app developers to build two versions of the application. One pre-schema change, and post. That way you deploy the first version, then the schema change(s). Then you deploy the second version. If there’s an issue, you rollback to the first version and undo the database changes. This sounds hard, but once you get into the swing of building code that survives additive changes to the database, this is easy.

    This doesn’t solve any destructive changes to the database, like dropping objects or manipulating data. I would suggest that drops are a completely separate release, and have a full backup (or snapshot) taken and saved for awhile. For data manipulation, save off the previous state of data, just in case you need to reload things.

    Becoming better at not only delivering changes to the customer, but also removing them when issues are detected is a valuable skill, and since we’re likely to have a bad release at some point, this might be a way to even further reduce the risk of deploying database changes.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Discussing Database DevOps with Gene Kim

    I’m really excited for this webinar. Register today for the Sept 25 webinar  to learn more about the 2018 State of DevOps report.

    I read The Phoenix Project a few years back and I’ve been following Gene Kim as one of the main researchers and advocates of DevOps ever since. I think his work is amazing and I’ve enjoyed watching him present online during conferences and webinars. I never thought I would get the chance to talk one on one, much less host a webinar.

    This year’s State of DevOps report, from DevOps Research and Assessment (DORA), sponsored by Google Cloud, is near and dear to my heart. For the first time, there is a section on databases. Redgate helped sponsor the report as well and was able to get Gene Kim to do a webinar to discuss the entire report, but with a focus on databases.

    At least, that’s my focus. I’ll get the chance to ask Gene questions as well, mostly focused on the data aspect of software delivery, which I think is too often left out. Too many software developers are worried about the database and haven’t included it in the their DevOps world. That’s changing, and join me to see how.

    Don’t forget to register and I’ll see you on the 25th.