Voice of the DBA

Tag: disaster recovery

  • DR as a Service

    It’s not the first task when I start a new job, but often as a DBA or developer, I usually ask about Disaster Recovery (DR) plans sometime within the first six months. If I’m a DBA, of course I need a plan. If I’m a developer, however, I still need to understand how this might work as it can affect how I build the software and prepare for networking, machine changes, etc. Even if I don’t concern myself with production DR, I usually do want to make sure the VCS repos are being protected, which is something I’ve found isn’t always being handled.

    I have had to build and test DR plans as a member of an Operations team in the past. While my plans and practice are nothing like Google’s large exercises, they often reveal some issues, even when we duplicate the service without touching production. I’ve usually found the brainstorming and debating of the various ways to build a plan to be fun. Arguing for money and then actually implementing plans is less interesting, but the testing is a great challenge. I’ve had some fun days offsite where we try to recover systems and find all the little things that we take for granted in our production environment.

    These days there are companies offering DR as a Service (DRaaS), which is an interesting concept. I found an article from Michael Otey that talks about the features you might want to look for if you contract with a vendor. In the past, I would never consider this, but the more we advance in the world with cloud infrastructures and even full service co-location vendors, the more I think DRaaS makes sense.

    I wouldn’t necessarily take anyone’s word that their service meets my needs, so thinking about the requirements, and then working through a few PoCs (proof of concept) is likely very important. We do a lot of PoC work at Redgate to help customers evaluate whether Compliant Database DevOps is a good fit. I think this is important for software development, but even more important for DR plans. After all, downtime is expensive, and the last thing you want to find out when troubles arise is that some critical piece of infrastructure can’t be easily duplicated.

    I’ve used DR companies in the past, with their own physical facilities. They have impressive capabilities and marketing, but the mixing of their skills with my systems has often been rocky and lead to changes in our plans, contracts for new or fewer services, and often updated documentation for junior staff. After all, I usually expect DR situations to occur when I’m on vacation, so I plan for that.

    You never know when you’ll need to execute DR plans. It pays to think about this ahead of time and periodically test yourself and your staff.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • What’s the Bus Factor?

    I was on a webinar recently and someone mentioned the bus factor, which is a phrase I’d heard before, but not in a long time. Essentially this is the number of people, or maybe a certain specific group of people, who can get hit by a bus before your organization has serious issues.

    Do you explicitly know who is important in your organization? I’m sure most executives think their personal factor is 1, meaning they can’t disappear without issues. I’m not sure I agree, though certainly we can’t lose all executives at once. In one small company I worked in, we never put more than two execs in a plane or car for a trip out of town, just in case something happened.

    This week, I wonder what your bus factor is in technology. If you think about your projects, your applications and infrastructure, do you know who the critical people are and how many are needed to keep things going? Or could everyone quit and new hires would take over. In other words:

    How many people can get hit by a bus before your org fails?

    I think that in Redgate, we don’t have many people that are irreplaceable. We have four advocates, and while we have different strengths and abilities, we can fill in for each other. Likely our advocate factor is 4. We’d struggle slightly if all four of us left, though I think our sales engineers and some developers could fill in for us.

    In development, we move people around regularly, and while they are valuable and great contributors, I think we could work around a large number of them leaving. I have no idea how many would cripple the company, but it would have to be a number much greater than 10.

    It’s worth recognizing when you have people in critical positions that are hard to replace. Do you have a single COBOL programmer still keeping something going, or maybe just one long time employee that knows how the BizTalk server runs and no one else makes changes. Think about it today, and let us know if you think there are single points of failure in your organization.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Defending Against Ransomware

    I don’t manage production systems at Redgate, and I’ve not seen us have any disruption in the access to systems across the last few years. At a number of events, however, I’ve spoken to friends that have told me about the problems they’ve had with ransomware. I’ve been quite surprised how often this happens, as I see headlines, but none of the half dozen or so people I spoke with have worked at any company in the headlines. This is likely a bigger problem than I thought.

    I don’t know if you’ve been attacked, but you should be prepared. Denny Cherry has had to deal with this at at least one client, and he wrote a few things down in this article. He doesn’t necessarily recommend you pay, though that’s certainly an option. If you do, you should be prepared for future attacks or other issues. Anyone that wants to hold your data hostage once might be inclined to do it again.

    Apart from backups, air gaps, and other preventative measures you might take, including limited access between some workstations and servers, you ought to be sure you can rebuild systems that work in a similar manner to the current ones. Do you know what the specs and settings are for all your systems now? So many people assume they could just go get that information from the system itself, but what if you VMs and VM host are compromised? What if your monitoring system is encrypted? Do you really know all the settings that have been changed from defaults? If your DR plan somewhere outside the network and can you get to it?

    This is a good reason why DevOps and infrastructure as code (IaC) are important in a modern organizations. In these cases, a VCS can hold the data you need to rebuild systems. This is a good reason by using GitHub or some remote VCS might be better than self-hosting your code. If you routinely rebuild systems for dev/test, you’ll have some idea of how to deploy, and if you think about IaC as a part of your DR process, you should be able to redeploy on new hardware (or existing hardware that’s wiped clean).

    Ransomware is a part of the new world we live in, and we ought to be prepared to defend against it. Ensure you’re ready today, because there will likely be a ransomeware or other types of attack in the future. The more prepared you are with better practice now, the more likely you’ll be ready to adapt to some other security need in the future.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Expect the Unexpected with DiRT

    Disaster recovery is one of the core tasks that many DBAs think about on a regular basis. Ensuring that we can get our data back online, available, accessible, and intact is important. More than a few DBAs that haven’t been able to recover systems, find themselves seeking new employment.

    That’s not to say that most DBAs perform perfectly under pressure. Plenty make mistakes, and there may be times when they can’t recover all data. There does seem to be a correlation between how often DBAs practice recovery skills and how well they perform in an actual emergency. I know that at a few companies, we scheduled regular disaster tests, though often with simulated recovery of a systems that didn’t expect to actually take over a workload. Arguably not a good test, but better than nothing.

    Google takes things a step further. They have annual, company wide, multi-day DiRT (Disaster Recovery Testing) events. These are across many departments and can be substantial in terms of the disruption that the these events cause to their infrastructure. This is a way for the various individuals responsible for infrastructure to actually evaluate if they are prepared for potential issues.

    If you read the article, you find that Google started small with these and progressed them to larger, more inclusive tests, like taking down a data center. They also whitelist some servers, knowing they cannot pass a test, so there is no reason to actually take them down. After all, business still needs to work.

    It’s good to have tests and walk through actual events, like call lists and bridges to be sure that communication and documentation work. This might be especially important when teams often expect that all their written procedures are available. I went through an audit with one company, where we failed immediately when all our DR plans were on a network share. In this simulation, we had experienced a network failure and servers had crashed. We were supposed to bring up the systems on spare hardware, but some critical documentation wasn’t available without a network. We started printing things out right away so that we could continue on with the simulation (as well as have this in a binder in our office).

    Not everyone can schedule large scale tests, and certainly many managers don’t see the point. They’ll often want to gamble that staff will “figure things out” if there is an incident. That doesn’t mean that DBAs and sysadmins can afford to wait for a disaster to practice some skills. Be sure that everyone on your team can recover databases, they know where backups are (or how to determine this), and multiple people have access to resources. The last thing you want is a disaster to occur during your vacation and have managers calling you to cut short your holiday because you’re the only one that knows where something is or has the authority to access a resource.

    Think about this ahead of time and prepare.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.