Voice of the DBA

Tag: disaster recovery

  • Advanced Incident Response

    Early in my career, I worked at a few smaller companies where a problem with the computer systems meant I went into the office and stayed until things were working. As I went to a few larger companies, I wasn’t alone when a system went down, but the process was mostly the same. We figured out what was wrong and found a way to fix or replace things, occasionally with help from a vendor. Those were the (not necessarily) good days before our internal networks were connected to a public Internet.

    Companies developed formal incident response plans to deal with various issues, whether these were problems we caused ourselves or failures of an application. I had the fortune, or misfortune, to be involved in more than a few issues and learned a great deal in how to solve problems as well as how to manage the impact to a large number of employees.

    As email and Internet use grew, so did the attacks with viruses and other sorts of malware. Antivirus software helped a great deal, but these days ransomware seems to be a common problem that isn’t as preventable as I would have hoped. Quite a few friends have dealt with ransomware issues, most of which have not been widely reported in the news.

    I saw an article about a few things that you might want to consider adding to your incident response plan. While some of these items might be unique to the ransomware threat, the thing that struck me was that there is a need to react quickly, in real-time, in response to any detection of an issue. I can only assume this means that there needs to be some advanced monitoring of nodes to detect issues, and I’m not sure how many organizations would adopt this, but in today’s world where we want systems available 24/7, perhaps they will.

    Being on call is a part of working in many IT departments. Having a response plan, even the general outline of one, helps to coordinate resources and ensure that we use people effectively. Tools are important, especially in today’s complex world, and it is important that one of those tools is a simple backup, preferably air-gapped from your main systems. If you don’t have these things in place, you might suggest someone start assembling them. These days it seems it’s not if you will get attacked, but when.

    Lastly, I don’t often see this addressed in plans, but make sure you have spelled out some guidelines on rotating staff and getting rest. Far too many companies want “all hands on deck” and forget that normal business still needs to occur. Any incident could last longer than a day, and you want to ensure that some of your staff is fresh and ready to take over from those that do need rest. Don’t be afraid to send some people home, or better yet, don’t call them in the first place.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Incident Review

    Something will always go wrong with our software systems. Issues might happen regularly or rarely, but something is going to go wrong at some point and a client will complain. Often the way we handle these situations determines whether our customers continue to do business with us or look for an alternative.

    In my career, I’ve been a part of many incidents, lots of which were service outages for customers. I’ve attended some incidents as a technical person working to diagnose the issue. In other incidents, I’ve been a developer that has to fix code. During many, my role was as a manager trying to ensure information moves smoothly between resources, ensuring people double-checked that our “fix” wouldn’t cause another problem. After a problem, I’ve also had to take part in a post-incident review. Unfortunately, this has happened far less often than it should.

    When I read this description of a post-incident review, it is unlike many of the after-action meetings I’ve attended. In my experience, there is one meeting and someone is being blamed. Senior management is often there, putting pressure on everyone in attendance to “never let this happen again.” I haven’t known anyone that wanted to go through another outage or a post-incident meeting after an incident, but with complex systems, and humans managing them, something is bound to go wrong. We should expect to have incidents in the future.

    What we want is this same type of incident to not get repeated, which comes about if we learn from our mistakes and design better protocols that help us catch mistakes.  Often to ensure a human doesn’t make a simple error. We should accept that mistakes will happen and try to find ways to detect the problem quickly, limit the scope of impact, and provide a way to share this knowledge with other workers. Depending on humans to be more perfect in the future isn’t likely to be successful.

    These days I read post-incident reviews that my company will publish after an outage, I find them fact-based, focusing on what things went wrong without blaming a person. They include analysis not only of the actual issue but the conditions that led to the hardware/software failure or decision that was made. There are learnings about how we might have prevented something with a time machine, while still assuming that humans would make mistakes or a component might fail. There are also suggestions for improvements in hardware, software, training, or monitoring that might assist in quicker recovery in the future.

    Coming out of an incident with a positive mindset is the best way to try and prevent a repeat of the same incident in the future. This requires that we not only avoid blaming someone for an error but that we also take steps to limit the potential for future errors. If the issue is someone clicking the wrong selection in a drop-down or pressing “OK” when they meant to press “Cancel”, there are limited ways to prevent those issues. However, we can adopt the mindset an outage is a team failure and build a habit of double checking each other. That’s much better than blaming one person, giving the job to another human, who might easily make the same mistake.

    Many humans struggle to avoid placing blame on others and just accepting that some mistakes will happen. A DevOps mindset, with blameless reviews instead focuses on how we can do better as a group, rather than how we failed as an individual. This little change helps us build a better team, one that often performs better in the future.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • How Do You Decide to Rollback?

    One of the constant questions I get from Redgate customers is about rolling back database changes. We build software to help you deploy changes, but what about rollbacks? I think I’ve heard this in almost every conversation I’ve had on this topic in the last 5 years. It’s one reason that Flyway has undo, though with some caveats. Please don’t expect this works magically like a restore operation.

    These is a complex topic, and one that can be way harder than a rollback for application changes. In fact, everyone wants it to be simple, but I don’t think it ever will be. There is a big reason for that: data. Undoing data changes can be impractical, especially in if there are lots of changes.

    The question I’d like you to ask yourself, and give me thoughts about, is when do you decide to rollback. How can you make that decision in a production system? Are there criteria or guidelines you use? Do you ask someone else? Maybe another aspect of this situation is how you decide to roll forward instead of rolling back.

    I was listening to some MVPs discuss this awhile back and one of the main criteria that one person brought up was data changes. This individual said if no data had changed, they just undid everything. However, once data changed, they were likely in a roll forward scenario where they needed to fix code quickly.

    That’s often the big factor for me. Once data is changed (or added), then we often can struggle to rollback. If we have an application that needs a new column, and we add it, if there software has a problem, we may not be able to roll back because the old software doesn’t work with the new column. We don’t want to delete the column, at least not until we save the data. These are the problems.

    The easy rollbacks are when the deployment fails and transactions undo the work, or we just quickly undo everything we did. That’s an easy decision, but I rarely find this sort of issue. Instead, usually we find certain data breaks our application or the logic is improperly implemented.

    I look forward to your ideas and thoughts here, but I do have some advice. One thing I always tell customers is that you want to deploy often, so that you are ready to fix something you broke. That’s important. The other thing that simplifies life is to never add and delete objects in the same deployment. This goes for everything. If I add new first and last name columns, but keep the old fullname column, if I’ve broken something, I still have the old columns and data. I can rollback easier. If everything works, I can always delete the fullname column later. Following these concepts has made my life much easier when I made database changes.

    Steve Jones

  • The Tech Blame Game

    Last year Solarwinds was hacked and blamed an intern for a security lapse. When Equifax was hacked, in testimony to the US Congress, the former CEO blamed a specific, though unnamed, person for not patching a system. British Airways blamed their USD$200+mm IT issue on an engineer that rebooted a system too quickly.

    I don’t know that any large company from my younger days, say before 1990, would have blamed a massive failure on a single person. While any single person can influence more systems in the age of technology, no one should have the power to cause such a massive failure. If they do, I think I’d look towards poor system design, rather than individuals.

    These aren’t the only examples of management trying to scapegoat an IT worker, and I suspect we’ll see more examples in the future. However, I hope that governments and shareholders start to demand better management from management. If you don’t understand how IT works, get auditors or consultants to evaluate things and explain them to you. If you don’t think that your systems are well put together without single points of failure, address that. If you worry about security, make that a priority. Microsoft did after the Slammer worm, and arguably they have a difficult job where most employees want to control their laptops and workstations entirely and run them in their individual manner. Microsoft built better controls into infrastructure and software development, and everyone else should as well. Management needs to own their responsibility for failures.

    We should expect mistakes in security, in design, in coding, and more. We should also be placing guardrails, tests, and limits inside our environments to ensure that we catch most of the issues. Software development and system design have improved dramatically the last decade to help us improve quality and security, but we have to embrace the knowledge that’s been gained, as well as ensure we have circuit breakers to prevent runaway failures. If a sysadmin can alter a Chef script to set the max memory in SQL Server to 1MB, this shouldn’t get deployed to all instances. Moreover, we ought to be testing for all sorts of potential changes that can cause issues.

    To me, this is the area that DevOps, GitOps, anything Ops, automated, or at scale, needs to mature. We need to allow for, expect, and assume mistakes and failures will happen and build in controls to our build and test systems. Once we start to better understand how someone can make simple mistakes, we can attach more checks and balances to ensure that we continue to improve quality, without sacrificing speed, or lowering security.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.