I delivered my talk on Architecting Zero Downtime Deployments yesterday at VS Live Las Vegas 2023. It went fairly well, even though I ran some incorrect code somewhere. Apologies for that, but glad I could fix things.
The code for the database and the C# app is in this Github repo: https://github.com/way0utwest/ZeroDowntime
The PPT is also in the repo, updated today.
If you find issues, or an improve my C#, please feel free to open an issue or submit a PR.
Many organizations have been trying to find better ways to build and deploy software for their customers. Whether they deal with the general public or internal customers, we know that delivering software that customers use can be a competitive advantage. That’s the goal of DevOps.
While most developers and management want to do this, they sometimes forget what the goal is. Instead, they want to continue to work in a similar manner themselves while giving lip service to actual change. They often do this while pushing others to somehow produce more and better software inside the same system. I see this over and over inside various companies.
To become better, many of us use metrics and measurement of data to help guide us in determining how to move forward. In the area of software, there are a number of research reports showing which metrics are indicative of organizations that do a good job of delivering value to their customers. There are four main metrics: deploy frequency, lead time, change fail percentage, and mean time to repair. These are highlighted, though there are plenty of other things to track in your software process.
However, aiming to just improve their metrics as the primary goal isn’t going to make your software better. The goal is to deliver software that meets your customers’ needs. Quicker, better quality, more features, and all those things that customers use are what is important. These metrics are there to help guide you, not to be the targets of efforts. There’s a good article that talks about some of the downsides of just trying to improve these metrics.
The goal is the continuous delivery of value to customers. The way we do this is by experimenting with code, getting rapid feedback from customers, adjusting and improving the code, and repeating the process, learning from our efforts. We drive automation to make this smooth and easy while enabling us to get our software to customers at the pace that suits our situation. It sounds vague and amorphous, and it is.
There is a bit of an art to developing a process that efficiently builds software. It depends highly on the people involved, and on two other things. First, guiding them to improve their process and skills with references to practices that have worked well. Second, giving them the freedom and support to experiment and learn from their efforts. In doing those two things, it’s important to remember that while you can measure how well things are changing, aiming to improve the measurements often doesn’t help you improve the goal: building better software.
It’s good to measure things, but keep in mind that the measures are not the goal.
Steve Jones
Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.
I’ll be at VS Live in Las Vegas this March to discuss zero downtime deployments. If you want to come and join me for this session, or any of the other great ones, register today and save $500 with the promo code “Jones”. You can use this link to register.
To simulate the effects of deployments on a workload, I built a small client. It’s nothing great, and likely some of you will laugh at my C# skills, but it works well enough. It’s a simple Windows form application that writes to a text box. However, it’s valuable to determine if there are any issues when you’ve made a deployment.
This post looks at the rough design of the client. Code is available in this repo: https://github.com/way0utwest/ZeroDowntime
I wrote a small app a few years ago to test and present on Always Encrypted. This was a basic WPF app that added the proper values to the connection string for Always Encrypted and let you query encrypted data (or not).
Like all mediocre developers, I copied and pasted that project into a new folder and set about modifying it. In this case, I set up a loop that continues to run and execute some database code, essentially using this loop:
while (iRunQuery > 0)
{
I set this value to 0 initially, and when a button is clicked, it’s 1. This then runs a bunch of lines to decide which DB code to run. I’ve mostly made this stored procedures to make it easier to adjust demos without touching C# code.
It’s not pretty.
At the bottom, I have this (outside the loop)
Application.DoEvents();
System.Threading.Thread.Sleep(100);
This is designed to catch me clicking a “stop” button that will set the variable back to 0. I added the delay because otherwise this runs a bit fast.
I have a few option buttons that adjust what code I’m calling, so I can simulate toggling feature flags on and off. I also log results to a window so you can see them, and I catch errors and log those. Errors are also counted, so we can see the impact of “non zero-downtime” changes.
It’s not a great example of software, but it does work.
Many of us work inside an organization that has a process for building and deploying software. We may find our org doing this well, or we may feel our process is poor with lots of room for improvement.
A lot of the discussion around how to be better at building software in the last ten years has been around the philosophy of DevOps. This concept doesn’t really prescribe how to build software, but give you goals to aim for. That means you still need to take the ideas of flow, feedback, and learning and decide how you implement them with your staff. What practices do you follow to ensure you can deploy quickly anytime your code is done? These can include ensuring you’ve tested code, getting feedback from customers, and more.
I ran across a post on recommended software engineering practices for an organization. The list includes seven things you should do:
This is a set of things I often preach to customers as well, especially 2, 3, 6, and 7. I often focus on the database and having curated test data, migrations you can count on and easy setup is important. And, of course, with SQL Prompt, you don’t need formatting ;). Just kidding, that’s important, too.
These are solid practices, and none of them are that hard to set up, but they do require some discipline and willingness to work as a team and maintain your process across time. Each of these items needs some care and feeding across time to remain relevant and helpful to your staff.
Do you have good software engineering practices? Are you proud of them and would you bring them to a new position? Or perhaps you wish your team would adopt better habits and a different mentality towards building software.
Steve Jones
Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.
Voice of the DBA 