I’ve sent my slide decks to the organizers, but I know they don’t always get posted. So I’m putting a few links up here for my talks.
If you check the page for my Preparation for Disaster talk, you’ll find I have links for the two recent events to the decks. I often version the decks after talks, and so I’ve decided to keep all versions live.
I’ve also uploaded the decks from PASS if you’re interested.
Disaster recovery can be a huge project at any company. Considering the ways in which you build a plan that covers all the infrastructure, and it can quickly become a full time job for someone. The details, and the scope of the project can be overwhelming when you try to address your environment.
There’s a way to make this easier: buckets. If you can group your systems into a few gross buckets that define those systems that have similar needs, you can make life simpler.
In the past, I have typically built four buckets of systems for DR purposes, though I think you could go with three. These are the buckets I’ve had:
Critical Systems – high priority systems that typically must be running the majority of the business day. Note that you may have a 24 hour business day in which case downtime must be minimized to minutes.
Low priority – systems that we can function without for a day or two if we need to. Often development systems fall into this area.
Everything else – Medium levels of priority.
Not recovered – This is an optional level, but it’s almost always been a set of test systems, maybe some development systems that aren’t important enough to worry about.
The fourth bucket, the “Not Recovered” bucket doesn’t mean you abandon those systems. In the event of an issue, you would make an attempt to recover them, but you might not spend time or resources practicing or preparing for the recovery effort.
Bucketing your systems is the best way to easily manage your preparations for disaster, and also set some gross priorities. You might end up recovering systems within the bucket in different orders, depending on what happened, but this gives you an easy way to allocate resources, both in a disaster, and in preparation.
One of the sayings that I’ve heard a few times from professional atheletics is this:
“Amateurs practice until they get it right. Professionals practice until they don’t get it wrong.”
Most of us working in technology would like to think that we’re professionals, but do we really practice restores, practice recovering systems, practice reconfiguring things until we don’t get it wrong?
I don’t think most of us do. We give it a try, and if it doesn’t work, we may examine why, but we often don’t go through the exercise enough to ensure that we really can do it well when there is pressure.
You can’t practice constantly, that’s not practical, but you ought to take a regular look at different systems, different problems, and try to reset them up every quarter or so in order to be prepared. Even if it’s something you do informally, you ought to ensure that you understand how your scripts work to automate restores, reconfigure systems, restore databases and logs, etc. With the advances in virtualization, you can even do this on workstations, setting up complex, multi-server environments on ordinary hardware.
It’s good practice for your skills and your career, and it might make you shine when there is an issue.
It’s hurricane season on the East Coast of the US right now, and after Hurricane Irene, I’m sure there are more than a few IT workers that are worried about their disaster recovery (DR) plans. Irene didn’t cause much damage, but it was a reminder that there are situations which are completely out of our control, but could easily end up causing us problems with our infrastructure.
SQL Server gives us numerous ways to prepare for disasters with different technologies like database mirroring, log shipping and more. The new Always On features in Denali will give us even more options, and may be worth the cost of an upgrade. While DBAs seem to regularly practice backups and restores, recovering a full application often requires more than a SQL Server restore. This Friday, I wanted to ask how often you practice your recovery.
How often do you perform a full DR test for an application?
I think it’s very unlikely that you would lose all your servers at once. It’s possible, especially with an event like a hurricane, but it’s very unlikely that you would have to recover all your systems at once. However that doesn’t mean that you shouldn’t be prepared to recover a complete system. If new hardware arrived, could you rebuild your Windows hosts, your SQL Server instance and application or web servers? I’ve found out the hard way this isn’t as easy as it sounds.
I used to perform a complete test of our one critical system once a year. We set aside a day, sending the IT staff needed to a secondary location with media, backups, and unformatted hardware to perform a test. If we couldn’t recovery the system in a day, we usually reviewed the issues and tried again in a month. Unfortunately, if we failed that test, we didn’t usually get another chance that year. Not exactly great disaster recovery planning.
These days my part of the infrastructure, SQLServerCentral, lives on virtual servers. While I don’t have the responsibility for these servers, we do have images of them, as well as regular backups, and I suspect we could recover them fairly quickly. Maybe I should ask for a test someday soon….