Voice of the DBA

Tag: disaster recovery

  • Incident Review

    Something will always go wrong with our software systems. Issues might happen regularly or rarely, but something is going to go wrong at some point and a client will complain. Often the way we handle these situations determines whether our customers continue to do business with us or look for an alternative.

    In my career, I’ve been a part of many incidents, lots of which were service outages for customers. I’ve attended some incidents as a technical person working to diagnose the issue. In other incidents, I’ve been a developer that has to fix code. During many, my role was as a manager trying to ensure information moves smoothly between resources, ensuring people double-checked that our “fix” wouldn’t cause another problem. After a problem, I’ve also had to take part in a post-incident review. Unfortunately, this has happened far less often than it should.

    When I read this description of a post-incident review, it is unlike many of the after-action meetings I’ve attended. In my experience, there is one meeting and someone is being blamed. Senior management is often there, putting pressure on everyone in attendance to “never let this happen again.” I haven’t known anyone that wanted to go through another outage or a post-incident meeting after an incident, but with complex systems, and humans managing them, something is bound to go wrong. We should expect to have incidents in the future.

    What we want is this same type of incident to not get repeated, which comes about if we learn from our mistakes and design better protocols that help us catch mistakes.  Often to ensure a human doesn’t make a simple error. We should accept that mistakes will happen and try to find ways to detect the problem quickly, limit the scope of impact, and provide a way to share this knowledge with other workers. Depending on humans to be more perfect in the future isn’t likely to be successful.

    These days I read post-incident reviews that my company will publish after an outage, I find them fact-based, focusing on what things went wrong without blaming a person. They include analysis not only of the actual issue but the conditions that led to the hardware/software failure or decision that was made. There are learnings about how we might have prevented something with a time machine, while still assuming that humans would make mistakes or a component might fail. There are also suggestions for improvements in hardware, software, training, or monitoring that might assist in quicker recovery in the future.

    Coming out of an incident with a positive mindset is the best way to try and prevent a repeat of the same incident in the future. This requires that we not only avoid blaming someone for an error but that we also take steps to limit the potential for future errors. If the issue is someone clicking the wrong selection in a drop-down or pressing “OK” when they meant to press “Cancel”, there are limited ways to prevent those issues. However, we can adopt the mindset an outage is a team failure and build a habit of double checking each other. That’s much better than blaming one person, giving the job to another human, who might easily make the same mistake.

    Many humans struggle to avoid placing blame on others and just accepting that some mistakes will happen. A DevOps mindset, with blameless reviews instead focuses on how we can do better as a group, rather than how we failed as an individual. This little change helps us build a better team, one that often performs better in the future.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • How Do You Decide to Rollback?

    One of the constant questions I get from Redgate customers is about rolling back database changes. We build software to help you deploy changes, but what about rollbacks? I think I’ve heard this in almost every conversation I’ve had on this topic in the last 5 years. It’s one reason that Flyway has undo, though with some caveats. Please don’t expect this works magically like a restore operation.

    These is a complex topic, and one that can be way harder than a rollback for application changes. In fact, everyone wants it to be simple, but I don’t think it ever will be. There is a big reason for that: data. Undoing data changes can be impractical, especially in if there are lots of changes.

    The question I’d like you to ask yourself, and give me thoughts about, is when do you decide to rollback. How can you make that decision in a production system? Are there criteria or guidelines you use? Do you ask someone else? Maybe another aspect of this situation is how you decide to roll forward instead of rolling back.

    I was listening to some MVPs discuss this awhile back and one of the main criteria that one person brought up was data changes. This individual said if no data had changed, they just undid everything. However, once data changed, they were likely in a roll forward scenario where they needed to fix code quickly.

    That’s often the big factor for me. Once data is changed (or added), then we often can struggle to rollback. If we have an application that needs a new column, and we add it, if there software has a problem, we may not be able to roll back because the old software doesn’t work with the new column. We don’t want to delete the column, at least not until we save the data. These are the problems.

    The easy rollbacks are when the deployment fails and transactions undo the work, or we just quickly undo everything we did. That’s an easy decision, but I rarely find this sort of issue. Instead, usually we find certain data breaks our application or the logic is improperly implemented.

    I look forward to your ideas and thoughts here, but I do have some advice. One thing I always tell customers is that you want to deploy often, so that you are ready to fix something you broke. That’s important. The other thing that simplifies life is to never add and delete objects in the same deployment. This goes for everything. If I add new first and last name columns, but keep the old fullname column, if I’ve broken something, I still have the old columns and data. I can rollback easier. If everything works, I can always delete the fullname column later. Following these concepts has made my life much easier when I made database changes.

    Steve Jones

  • The Tech Blame Game

    Last year Solarwinds was hacked and blamed an intern for a security lapse. When Equifax was hacked, in testimony to the US Congress, the former CEO blamed a specific, though unnamed, person for not patching a system. British Airways blamed their USD$200+mm IT issue on an engineer that rebooted a system too quickly.

    I don’t know that any large company from my younger days, say before 1990, would have blamed a massive failure on a single person. While any single person can influence more systems in the age of technology, no one should have the power to cause such a massive failure. If they do, I think I’d look towards poor system design, rather than individuals.

    These aren’t the only examples of management trying to scapegoat an IT worker, and I suspect we’ll see more examples in the future. However, I hope that governments and shareholders start to demand better management from management. If you don’t understand how IT works, get auditors or consultants to evaluate things and explain them to you. If you don’t think that your systems are well put together without single points of failure, address that. If you worry about security, make that a priority. Microsoft did after the Slammer worm, and arguably they have a difficult job where most employees want to control their laptops and workstations entirely and run them in their individual manner. Microsoft built better controls into infrastructure and software development, and everyone else should as well. Management needs to own their responsibility for failures.

    We should expect mistakes in security, in design, in coding, and more. We should also be placing guardrails, tests, and limits inside our environments to ensure that we catch most of the issues. Software development and system design have improved dramatically the last decade to help us improve quality and security, but we have to embrace the knowledge that’s been gained, as well as ensure we have circuit breakers to prevent runaway failures. If a sysadmin can alter a Chef script to set the max memory in SQL Server to 1MB, this shouldn’t get deployed to all instances. Moreover, we ought to be testing for all sorts of potential changes that can cause issues.

    To me, this is the area that DevOps, GitOps, anything Ops, automated, or at scale, needs to mature. We need to allow for, expect, and assume mistakes and failures will happen and build in controls to our build and test systems. Once we start to better understand how someone can make simple mistakes, we can attach more checks and balances to ensure that we continue to improve quality, without sacrificing speed, or lowering security.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Nothing on My Laptop

    I heard this quote at the recent Redgate Summit: “If I drop my laptop in water, I can get everything back.” The context of the quote is that all your work, your IP, your code, etc. is saved in some common or duplicate location. Everything essentially has a backup.

    I learned the lesson early in my career that keeping one copy around of things was bad. On an old Apple II, I formatted the wrong disk numerous times, necessitating me recreating homework multiple times on a very tight schedule. Over the years, I often made sure I had some backup service, but things really changed for me about ten years ago.

    I had a laptop die on me while on the road. I had copies of things in different places, and a backup service, but I had to buy a new machine and get things set up quickly. It was a pain, but I started to adopt the DevOps idea of improving my system and learning what worked. Today’s cloud services, as well as package managers (Chocolatey) helped me. A few years ago my laptop blue screened and I had to reformat the drive and rebuild everything for a presentation the next day.

    It took my about 3 hours, thanks to code, documents, and more stored and available in different services. These days, I think I was up and running on a new laptop in tens of minutes, and had most of my bits available in Explorer or inside an application in a little over an hour.

    These are great habits to ensure I can continue to work, and good for home where I don’t want to lose photos or other digital assets.  These are also good ideas for shared work inside an organization, whether on workstations or servers. Don’t operate without a way to rebuild systems, including configuration and data, if something fails. Cultivate cattle, not pets, as the DevOps people say.

    Steve Jones