Voice of the DBA

Tag: disaster recovery

  • Nothing on My Laptop

    I heard this quote at the recent Redgate Summit: “If I drop my laptop in water, I can get everything back.” The context of the quote is that all your work, your IP, your code, etc. is saved in some common or duplicate location. Everything essentially has a backup.

    I learned the lesson early in my career that keeping one copy around of things was bad. On an old Apple II, I formatted the wrong disk numerous times, necessitating me recreating homework multiple times on a very tight schedule. Over the years, I often made sure I had some backup service, but things really changed for me about ten years ago.

    I had a laptop die on me while on the road. I had copies of things in different places, and a backup service, but I had to buy a new machine and get things set up quickly. It was a pain, but I started to adopt the DevOps idea of improving my system and learning what worked. Today’s cloud services, as well as package managers (Chocolatey) helped me. A few years ago my laptop blue screened and I had to reformat the drive and rebuild everything for a presentation the next day.

    It took my about 3 hours, thanks to code, documents, and more stored and available in different services. These days, I think I was up and running on a new laptop in tens of minutes, and had most of my bits available in Explorer or inside an application in a little over an hour.

    These are great habits to ensure I can continue to work, and good for home where I don’t want to lose photos or other digital assets.  These are also good ideas for shared work inside an organization, whether on workstations or servers. Don’t operate without a way to rebuild systems, including configuration and data, if something fails. Cultivate cattle, not pets, as the DevOps people say.

    Steve Jones

  • DR as a Service

    It’s not the first task when I start a new job, but often as a DBA or developer, I usually ask about Disaster Recovery (DR) plans sometime within the first six months. If I’m a DBA, of course I need a plan. If I’m a developer, however, I still need to understand how this might work as it can affect how I build the software and prepare for networking, machine changes, etc. Even if I don’t concern myself with production DR, I usually do want to make sure the VCS repos are being protected, which is something I’ve found isn’t always being handled.

    I have had to build and test DR plans as a member of an Operations team in the past. While my plans and practice are nothing like Google’s large exercises, they often reveal some issues, even when we duplicate the service without touching production. I’ve usually found the brainstorming and debating of the various ways to build a plan to be fun. Arguing for money and then actually implementing plans is less interesting, but the testing is a great challenge. I’ve had some fun days offsite where we try to recover systems and find all the little things that we take for granted in our production environment.

    These days there are companies offering DR as a Service (DRaaS), which is an interesting concept. I found an article from Michael Otey that talks about the features you might want to look for if you contract with a vendor. In the past, I would never consider this, but the more we advance in the world with cloud infrastructures and even full service co-location vendors, the more I think DRaaS makes sense.

    I wouldn’t necessarily take anyone’s word that their service meets my needs, so thinking about the requirements, and then working through a few PoCs (proof of concept) is likely very important. We do a lot of PoC work at Redgate to help customers evaluate whether Compliant Database DevOps is a good fit. I think this is important for software development, but even more important for DR plans. After all, downtime is expensive, and the last thing you want to find out when troubles arise is that some critical piece of infrastructure can’t be easily duplicated.

    I’ve used DR companies in the past, with their own physical facilities. They have impressive capabilities and marketing, but the mixing of their skills with my systems has often been rocky and lead to changes in our plans, contracts for new or fewer services, and often updated documentation for junior staff. After all, I usually expect DR situations to occur when I’m on vacation, so I plan for that.

    You never know when you’ll need to execute DR plans. It pays to think about this ahead of time and periodically test yourself and your staff.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • What’s the Bus Factor?

    I was on a webinar recently and someone mentioned the bus factor, which is a phrase I’d heard before, but not in a long time. Essentially this is the number of people, or maybe a certain specific group of people, who can get hit by a bus before your organization has serious issues.

    Do you explicitly know who is important in your organization? I’m sure most executives think their personal factor is 1, meaning they can’t disappear without issues. I’m not sure I agree, though certainly we can’t lose all executives at once. In one small company I worked in, we never put more than two execs in a plane or car for a trip out of town, just in case something happened.

    This week, I wonder what your bus factor is in technology. If you think about your projects, your applications and infrastructure, do you know who the critical people are and how many are needed to keep things going? Or could everyone quit and new hires would take over. In other words:

    How many people can get hit by a bus before your org fails?

    I think that in Redgate, we don’t have many people that are irreplaceable. We have four advocates, and while we have different strengths and abilities, we can fill in for each other. Likely our advocate factor is 4. We’d struggle slightly if all four of us left, though I think our sales engineers and some developers could fill in for us.

    In development, we move people around regularly, and while they are valuable and great contributors, I think we could work around a large number of them leaving. I have no idea how many would cripple the company, but it would have to be a number much greater than 10.

    It’s worth recognizing when you have people in critical positions that are hard to replace. Do you have a single COBOL programmer still keeping something going, or maybe just one long time employee that knows how the BizTalk server runs and no one else makes changes. Think about it today, and let us know if you think there are single points of failure in your organization.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Defending Against Ransomware

    I don’t manage production systems at Redgate, and I’ve not seen us have any disruption in the access to systems across the last few years. At a number of events, however, I’ve spoken to friends that have told me about the problems they’ve had with ransomware. I’ve been quite surprised how often this happens, as I see headlines, but none of the half dozen or so people I spoke with have worked at any company in the headlines. This is likely a bigger problem than I thought.

    I don’t know if you’ve been attacked, but you should be prepared. Denny Cherry has had to deal with this at at least one client, and he wrote a few things down in this article. He doesn’t necessarily recommend you pay, though that’s certainly an option. If you do, you should be prepared for future attacks or other issues. Anyone that wants to hold your data hostage once might be inclined to do it again.

    Apart from backups, air gaps, and other preventative measures you might take, including limited access between some workstations and servers, you ought to be sure you can rebuild systems that work in a similar manner to the current ones. Do you know what the specs and settings are for all your systems now? So many people assume they could just go get that information from the system itself, but what if you VMs and VM host are compromised? What if your monitoring system is encrypted? Do you really know all the settings that have been changed from defaults? If your DR plan somewhere outside the network and can you get to it?

    This is a good reason why DevOps and infrastructure as code (IaC) are important in a modern organizations. In these cases, a VCS can hold the data you need to rebuild systems. This is a good reason by using GitHub or some remote VCS might be better than self-hosting your code. If you routinely rebuild systems for dev/test, you’ll have some idea of how to deploy, and if you think about IaC as a part of your DR process, you should be able to redeploy on new hardware (or existing hardware that’s wiped clean).

    Ransomware is a part of the new world we live in, and we ought to be prepared to defend against it. Ensure you’re ready today, because there will likely be a ransomeware or other types of attack in the future. The more prepared you are with better practice now, the more likely you’ll be ready to adapt to some other security need in the future.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.