Voice of the DBA

Tag: deployment

  • Rollback vs. Roll Forward

    The Advocates at Redgate Software had an interesting discussion about deployments in databases and how you go forward or back from the point at which you discover a problem. You can watch the episode, but a few things occurred to me while we were having our discussion.

    The first thing is we all agree data makes things hard. A database is a stateful object, and dealing with stateful objects is hard. That is one of the things I’ve internalized the last few years that has tremendously changed how I work with Redgate customers. The more I consider state, the more I am able to work with the challenges that databases bring.

    The second interesting thing from the episode for me was that each of us had a tendency for what to do. Do you tend to aim to get to a previous state or move to a new state? Each of us had a preference.

    I think lots of us aim for a new state, mostly because we’re optimistic about our ability to “fix” the broken thing. I also don’t think this is a DBA, engineer, or technology thing. I find lots of people in the real world making mistakes and thinking they can do a new thing to fix the situation. Mechanics, lawyers, doctors, plumbers, they all think they can roll forward to a new state.

    What’s occurred to me is that the people I know who are very much in demand for their skills and expertise are often the ones with a tendency to roll back to the previous state. They acknowledge the mistake and undo it. I’ve seen people in construction and other professionals also try to go back to the previous state when they realize they are in a situation where their fix didn’t work, and they abandon that plan.

    In the DBA world, we might prefer this even when our deployment caused data changes. We often will save data, roll back, and then decide what to do. This often means reconciling data, which isn’t a fun task, but a necessary one.

    Watch the episode and decide what your tendency is for adjusting for changes. Do you think about patching the issue and rolling forward? Or do you want to roll back to a known state and re-test our changes in a lower environment.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Rolling Back a Broken Release

    We had an interesting discussion about deployments in databases and how you go forward or back from the point when you discover a problem. You can watch the episode and see what you think, but one thing that Pat asked was about rolling back a broken release.

    I’ve seen a few broken releases that were rolled back immediately in my career. However, in a lot of cases, I’ve also been a part of semi-failed releases where we had to roll things forward.

    I learned early on to smoke test the system post-deployment. Either get an account, or run known queries after applying a patch to ensure things worked BEFORE I let someone know the deployment was complete.

    In one case, we applied a patch, restarted the application and started receiving errors immediately. We knew then that whatever things had changed, those changes were not in sync with the application. In this case, the application code and the database code had a small typo in a name, but late at night, we didn’t realize it was this simple.

    In the days of outage windows, we couldn’t debug for long, so I decided immediately that getting back online was important. We replaced the new .exe with the old one and I looked at each of the database commands and wrote a reversing one to reset the database. Since the system was down, there weren’t any data changes.

    In recent years, a few clients have had an easier time as they use feature flags to enable new functionality. When they’ve had an issue like the wrong name in code, they just flip the toggle to disable the feature. This rolls back the code.

    I highly recommend using feature flags to anyone working with database software changes. Coupled with zero-downtime architectures for database changes, this lets us rollback things quickly.

  • What is a Failed Deployment?

    When talking about DevOps, the goal is to produce better software over time. Both better quality as well as a smoother process of getting bits to your clients. There are a number of metrics typically used to measure how well a software team is performing, and one of the things is Change fail percentage. This is the percentage of deployments that causes a failure in production, which means a hotfix or rollback is needed. Essentially we need to fail forward or roll back to get things working.

    For most people, a failed deployment means downtime. I’ve caused a service to be down (or a page or an app) because of a code change I made. This includes the database, as a schema change could cause the application to fail. Maybe we’ve renamed something (always a bad idea) and the app hasn’t updated. Maybe we added a new column to a table and some other code has an insert statement without a column list that won’t run. There are any number of database changes that might require a hotfix or rollback and could be considered a failure.

    However, some people see an expanded definition. If a service is degraded (slower), is that a failure? Some people think so. If we change code in a database (or indexes) and see performance slow down. In that case, is this a failed deployment? Customers would think so. Developers might not like this idea, at least not without some sort of SLA that might allow for some things to be a little slower. After all, slow is still working, right?

    What if I don’t notice a problem? Imagine I add a new table/column, and the app starts accepting data and storing it. What if we are supposed to use this data downstream, and we don’t notice it is being aggregated incorrectly by a process until many days later. Perhaps we’ve performed some manipulation or calculation on our data and the result isn’t what we wanted. It might not be incorrect, but maybe it’s ignoring NULLs when we want NULLs treated as 0s.

    Is that a failure? If I deploy today and Bob or Sue notices next week that the data isn’t correct, that’s a failure. I don’t know I’d count downtime from today until next week, but from when Bob/Sue files a ticket, the clock starts on calculating the MTTR (mean time to recovery).

    I don’t often see database deployments failing from the “will it compile on the production server” standpoint. Most code gets tested on at least one other system, and with any sort of process, we catch those simple errors. More often than not, we find performance slowdowns or misunderstood requirements/specifications. In those cases, some of you might consider this a failure and some may not. I suppose it depends on whether these issues get triaged as important enough to fix.

    While I might have a wide definition of deployment failures for most coding problems, I don’t for a performance slowdown. Far too few people really pay attention to code performance and are happy to let bad code live in their production systems for years.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • SQL Saturday South Florida 2024 Powerpoint Decks

    Thanks to everyone who attended my sessions today at SQL Saturday South Florida. Here are the resources from today.

    Blogging for the Tech Professional

    Deck: BloggingfortheTechProfessional.pptx

    Get started with writing today. Use Word or Open Live Writer, save 10 posts offline, then set up a blog.

    Architecting Zero Downtime Deployments

    I always enjoy this talk and I hope you did. Here are the things you might need: