Voice of the DBA

Tag: DevOps

  • Rolling Back a Broken Release

    We had an interesting discussion about deployments in databases and how you go forward or back from the point when you discover a problem. You can watch the episode and see what you think, but one thing that Pat asked was about rolling back a broken release.

    I’ve seen a few broken releases that were rolled back immediately in my career. However, in a lot of cases, I’ve also been a part of semi-failed releases where we had to roll things forward.

    I learned early on to smoke test the system post-deployment. Either get an account, or run known queries after applying a patch to ensure things worked BEFORE I let someone know the deployment was complete.

    In one case, we applied a patch, restarted the application and started receiving errors immediately. We knew then that whatever things had changed, those changes were not in sync with the application. In this case, the application code and the database code had a small typo in a name, but late at night, we didn’t realize it was this simple.

    In the days of outage windows, we couldn’t debug for long, so I decided immediately that getting back online was important. We replaced the new .exe with the old one and I looked at each of the database commands and wrote a reversing one to reset the database. Since the system was down, there weren’t any data changes.

    In recent years, a few clients have had an easier time as they use feature flags to enable new functionality. When they’ve had an issue like the wrong name in code, they just flip the toggle to disable the feature. This rolls back the code.

    I highly recommend using feature flags to anyone working with database software changes. Coupled with zero-downtime architectures for database changes, this lets us rollback things quickly.

  • DevOps is DevOps

    DevOps can mean a lot of things, but I find in practice that this results in a team using Continuous Integration and Continuous Deployment/Delivery using automation to check and evaluate your software in some way. This should result in quicker delivery of updates and changes to customers, better agility, and higher quality of code.

    That last one only comes if you use testing and try to ensure your code is well-written. It’s easy to just use DevOps to throw out more poorly written code that doesn’t perform well.

    Recently, I went to a presentation on Fabric CI/CD, looking to see if I could learn something about how you might handle Fabric in a DevOps software development flow. What I came away with was the idea that DevOps is really just DevOps. In this case, Azure DevOps was being used to orchestrate the CI/CD flow, and the pipeline looked the same way as one I had set up for a relational database.

    There are differences, such as the need to have multiple Fabric workspaces and a need to understand how the code is stored and managed in Git, but the idea is the same. We use a build step to verify that any code changes can be successfully compiled and that those changes can be deployed to a new location. A workspace instead of a database, which means the automation doing the work is slightly different, but the process is the same.

    Testing? You want tests to verify that the code does what is expected. A lot of people skip testing with data, but I’d argue it’s as important in Power BI or SQL Server as it is in C#. We ought to include tests, not because we don’t know how to write code, but because others might maintain/extend/refactor our code, and we want their changes to pass our tests. Again, the details are different, but including a step to run tests is still there. The same idea.

    Using a flow to approve and deploy changes between environments? This is no different, and having an automated process reduces the effort, time, and risk of mistakes. This is again the same, though the details are different specifics because the automation is different. Deploying a semantic model is different than pushing out a new .exe or running database changes, but the idea of an automated deployment is the same.

    We, as an industry, have learned over time how to better move code through a software development life cycle. The ideas work for data-based projects as well, and we ought to be learning and adapting these to ensure we can be just as agile as application software developers.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Requiring Technical Debt Payments

    I was working with a customer recently that is trying to improve their processes. This was a large company, over 100,000 employees, though most of them aren’t in the technology area. However, across many divisions and groups, there are a lot of developers and operations personnel who have tended to work in silos, managing their own applications and systems in disparate ways.

    In other words, doing software development the way most companies do it.

    I had been working with one group to streamline and standardize some of their software practices to implement more of a DevOps flow to smoothly build, operate, and update their systems. They’ve had some success and other groups noticed that this set of teams is very efficient. They aren’t DevOps like a lot of the articles you read. They still have development and operations, but the groups work closely to ensure efficiency.

    They started to get requests to onboard other teams into their flow as the management of this group has been advertising their success. Other groups want to implement Continuous Integration, get database unit testing and static code analysis setup, implement gates for approval, and more. The Operations team manages most of this and is happy to help other groups.

    But

    They require some things to be in place, some of which are cleaning up technical debt. Not all debt, but certain things that create additional risk or instability. Before they onboard anyone, they don’t want to take on a codebase that is difficult to manage. A lot of this debt isn’t difficult, but they want some good coding practices implemented. They require integrated security or a waiver from InfoSec. They want explicit index names, not system-generated ones. They want permissions granted to roles, not users. Not big things, but little items that make a system less maintainable and understandable.

    The same things a lot of us let creep into our codebase over time.

    On one hand, I thought this was an idea that would slow adoption and allow many groups to continue to operate inefficiently. They won’t clean up code. On the other hand, this might be the lever that helps create a better run environment across the organization. This might help them smooth their upgrade cycles, let staff change between projects, and more importantly, reduce the overhead of communication and work between teams.

    I don’t know how this will work over time, but I am interested to see what happens.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • What is a Failed Deployment?

    When talking about DevOps, the goal is to produce better software over time. Both better quality as well as a smoother process of getting bits to your clients. There are a number of metrics typically used to measure how well a software team is performing, and one of the things is Change fail percentage. This is the percentage of deployments that causes a failure in production, which means a hotfix or rollback is needed. Essentially we need to fail forward or roll back to get things working.

    For most people, a failed deployment means downtime. I’ve caused a service to be down (or a page or an app) because of a code change I made. This includes the database, as a schema change could cause the application to fail. Maybe we’ve renamed something (always a bad idea) and the app hasn’t updated. Maybe we added a new column to a table and some other code has an insert statement without a column list that won’t run. There are any number of database changes that might require a hotfix or rollback and could be considered a failure.

    However, some people see an expanded definition. If a service is degraded (slower), is that a failure? Some people think so. If we change code in a database (or indexes) and see performance slow down. In that case, is this a failed deployment? Customers would think so. Developers might not like this idea, at least not without some sort of SLA that might allow for some things to be a little slower. After all, slow is still working, right?

    What if I don’t notice a problem? Imagine I add a new table/column, and the app starts accepting data and storing it. What if we are supposed to use this data downstream, and we don’t notice it is being aggregated incorrectly by a process until many days later. Perhaps we’ve performed some manipulation or calculation on our data and the result isn’t what we wanted. It might not be incorrect, but maybe it’s ignoring NULLs when we want NULLs treated as 0s.

    Is that a failure? If I deploy today and Bob or Sue notices next week that the data isn’t correct, that’s a failure. I don’t know I’d count downtime from today until next week, but from when Bob/Sue files a ticket, the clock starts on calculating the MTTR (mean time to recovery).

    I don’t often see database deployments failing from the “will it compile on the production server” standpoint. Most code gets tested on at least one other system, and with any sort of process, we catch those simple errors. More often than not, we find performance slowdowns or misunderstood requirements/specifications. In those cases, some of you might consider this a failure and some may not. I suppose it depends on whether these issues get triaged as important enough to fix.

    While I might have a wide definition of deployment failures for most coding problems, I don’t for a performance slowdown. Far too few people really pay attention to code performance and are happy to let bad code live in their production systems for years.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.