Tag: DevOps

Do any of you think that a rollback of a version for your application is easy? Most people struggle, and when I talk DevOps and improving your process, the number one question is about rollbacks. In fact, recently I had a few people that struggled to even listen to the early parts of DevOps because all they could think about were previous failed releases and the need to roll back. They kept saying, what about rolling back. Hopefully you can stick with me a bit longer.

I’d argue that rollbacks are the process that needs DevOps more than ever, with smaller, more regular releases and practice at making changes. Whether forward or backward, we ought to be able to rev our software easily. I ran across a piece from the cloud platform blog at Google called Reliable releases and rollback – CRE life lessons. The title is an interesting one, but suspend some of your database skepticism until the end.

It’s easy to consider rolling back in the early parts of the article and say “it’s way easier to roll back your application”, and it is. Applications just stomp down new (or old) versions on top of what’s there, which is often an easy thing to do. As they say at Google, “rollbacks are normal”, which certainly seems to fit with the application paradigm.

In fact, they recommend rolling back a good release. After all, it’s much easier to practice this sort of thing when you have a working new version of software. When the release breaks your system, as mentioned in the piece, everyone’s stress level rises and the fixes often aren’t well built. Even when they work, which isn’t anywhere near all the time, there are often problems later. The idea should be to roll back and ensure everyone knows how to undo a version change. They can document the reasons for rollback and get the previous state of the application running. I hadn’t thought about this, but it makes sense. Practice in advance and be prepared. You can always re-deploy the working version.

What about databases? They have a solution, and I like it. They want the app developers to build two versions of the application. One pre-schema change, and post. That way you deploy the first version, then the schema change(s). Then you deploy the second version. If there’s an issue, you rollback to the first version and undo the database changes. This sounds hard, but once you get into the swing of building code that survives additive changes to the database, this is easy.

This doesn’t solve any destructive changes to the database, like dropping objects or manipulating data. I would suggest that drops are a completely separate release, and have a full backup (or snapshot) taken and saved for awhile. For data manipulation, save off the previous state of data, just in case you need to reload things.

Becoming better at not only delivering changes to the customer, but also removing them when issues are detected is a valuable skill, and since we’re likely to have a bad release at some point, this might be a way to even further reduce the risk of deploying database changes.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

A few weeks ago I gave a presentation at the 24 Hours of Pass Summit Preview. During my session, a demo broke and I had to ignore it. Later I found the issue and blogged about it. What was neat for me is that I didn’t need to keep my code up or apps open, but I still found the issue quickly. I found it because I have a DevOps process that instruments and tracks everything.

This happened to me a few years ago. I was at a SQL Saturday and demoing one of the Redgate tools. There was an error and I couldn’t move changes to a downstream database, which threw me off. That was the point of the talk. I decided to debug on stage (or behind the podium in this case), looking through the logs. Within a few minutes, I had found the issue and solved it. A real life DevOps story in action.

The idea of DevOps is what most talented developers and DBAs do. They ensure that they don’t just make changes on a whim, and they don’t depend on SSMS to be kept open. If they find an error, they save it, or they ensure their process captures all relevant logging. I’ve built systems like this, but many of the modern tools we use in DevOps automatically do this. Build servers and release servers make it a point of capturing all logging, so when that PoSh or CLI system runs, the output is saved and available for solving problems. And the data is available for improving the process, which is the goal.

At various times in my career, I haven’t followed a set process, I’ve “tried” things in SSMS, a configuration dialog, or a command window that haven’t worked. I’ve sometimes remembered to undo them, sometimes not. I’ve lost track of what I’ve run often, just because I was working in an ad hoc manner. That led me to learn to do better, which led me to embrace and follow many of the principles and ideas that people call DevOps today. These are things I learned to do those things 15 years ago.

Lots of DevOps is about using automation and tools, and those really do help. Having a pipeline for software changes, and one for configuration changes, reduces the chance of mistakes and problems. It also lets you instrument and track everything that happens. However, these are the easy things. These are the things that really talented professionals already do in their daily work.

The hard part of building software is building a culture that learns to work together instead of separately, or worse, adversarially. It is truly hard to work together and concern yourself with the customer first, and your job last. Or put the QA person, the DBA, the developer, or someone else ahead of what makes your day easier. It’s also hard for management to build a framework where we don’t have incentives to put our own interests ahead of others. If we can do that, and build a DevOps style process, things work smoother, and our software will be better. Maybe more importantly, we’ll enjoy our time at work.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.