We had an interesting discussion about deployments in databases and how you go forward or back from the point when you discover a problem. You can watch the episode and see what you think, but one thing that Pat asked was about rolling back a broken release.
I’ve seen a few broken releases that were rolled back immediately in my career. However, in a lot of cases, I’ve also been a part of semi-failed releases where we had to roll things forward.
I learned early on to smoke test the system post-deployment. Either get an account, or run known queries after applying a patch to ensure things worked BEFORE I let someone know the deployment was complete.
In one case, we applied a patch, restarted the application and started receiving errors immediately. We knew then that whatever things had changed, those changes were not in sync with the application. In this case, the application code and the database code had a small typo in a name, but late at night, we didn’t realize it was this simple.
In the days of outage windows, we couldn’t debug for long, so I decided immediately that getting back online was important. We replaced the new .exe with the old one and I looked at each of the database commands and wrote a reversing one to reset the database. Since the system was down, there weren’t any data changes.
In recent years, a few clients have had an easier time as they use feature flags to enable new functionality. When they’ve had an issue like the wrong name in code, they just flip the toggle to disable the feature. This rolls back the code.
I highly recommend using feature flags to anyone working with database software changes. Coupled with zero-downtime architectures for database changes, this lets us rollback things quickly.
Voice of the DBA 
I don’t miss having to deal with releases. That said, your post reminds me of a company in Ann Arbor Michigan, that my parent company acquired. This was in the early 2000s.
They had a VERY robust rollback procedure. They had to, because they had no effective QA process! So they’d do releases at 12:01 AM on Saturday mornings and have everyone in the company log in and hit the application as hard as they could!
I recall sitting in the Toledo Amtrak station, waiting for my train home, watching the chat on line as they did a release one night. It was interesting. It certainly as no way to run releases though.
Later, at a different company, we only had one release that scared the daylights out of me. Given the scope of the release and the number of updates it made, it was determined that rolling it back was pretty much not going to be possible short of an entire restore of the database. I actually slept in a chair in my apartment waking up every 30-60 minutes to monitor it. Fortunately it worked perfectly (in part because at this company we had a robust QA department) and I was able to catch the train to Williamsburg to meet my family for vacation.
LikeLike
I don’t miss late night releases. Talking a customer recently, they are in that spot, with a limited window. The reality is they don’t have the 3 or 4 hours to release. They really have an hour, because if there are issues, they need time to diagnose and then rollback.
My worst windows were once a quarter, Sat midnight to Sun 4a.
LikeLike