Tag: disaster recovery

The other day we had a blizzard in Colorado and we weren’t quite prepared. My wife and I were away on a trip and watching the weather. We were concerned about flights and about the conditions at the ranch with someone else in charge of horses. We were lucky in that the weather didn’t come as quickly as predicted and we made it home before the snow started.

Depending on moisture and temperature, we sometimes put blankets on horses, which is a chore that I’m not very good at completing. Fortunately the kids were around and at 8pm, we all dressed in warm clothes and headed out to the barn. The horses depend on us to provide for some of their needs, so we each had jobs to do. One kid secured a temporary holding area to keep horses nearby. One gathered 14 or 15 blankets from storage and laid them out. My wife made grain to facilitate attracting the horses and giving us a way to hold them while blankets went on.

Me? I had the envious task of double checking electricity and tank water headers. Our employee had thought that one was shorting out and at night, with a flashlight and plug tester, i got to debug 4 water heaters, including one that’s semi-remote from the barn. Testing for shorts involves touching water to see if there is any flowing power (this is very low amperage), which isn’t terribly dangerous, but is a little daunting. We managed to get everything done, and felt like we were prepped for a 4-10″ snowfall, 50mph winds, and a 0-5F temperatures.

Later I lay in bed, trying to relax with a little TV before calling it a night. While the wind was howling, I heard a pop and the power went out. We have a generator, and I wasn’t too worried, but I did want to ensure it came on. I walked downstairs and across the house to listen for the engine noise. I heard cranking, but the engine didn’t catch. It was then I remembered the tank was low the last time I checked, and with a weekly diagnostic auto run for 5 minutes, I was likely out of propane. Since we needed power to heat water for horses, I had to get up around 11pm and take care of things.

Fortunately I keep a spare bottle of propane and I went outside in the blowing snow to change it. A true work-at-home-techie, I did this without pants and got things running. Back to bed, though worried a bit about how to get more propane in the morning, just in case of an extended outage. The power came back on in the morning, but I still needed to get more propane just in case.

Extended disasters sometimes cause problems with our plans because many of us focus on the immediate reactions. Longer term, things like additional fuel, food for humans, even shelter and child care are issues that I’ve had to deal with in teams that no one had planned for. We’ve seen this over and over again in the world, especially when supply chains break down quickly in disaster situations.

I could have prevented some of the issues and stress if I’d prepped things better. I should have filled propane tanks earlier, knowing that bad weather can come at any time, and likely will at some time. Double checking heaters and power earlier might have made the blanketing process go quicker. There are always things we miss in prep, often ones we don’t consider to be important, but may be difficult to deal with in the moment. Take a few minutes and think about the little things you might dismiss as not important. Imagine how hard some of them might be to deal with under the pressure and stress of a DR event. If you can do a little more prep, some maintenance, or get some work done early, now might be the time to do it.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 5.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

Data professionals know that protecting and ensuring our data is safe is one of our primary jobs. Many of us worry that if we were unable to recover a database, our job might be jeopardy, which makes sense. If we lose data, an organization might make the decision that we aren’t trustworthy enough to manage a database system. This leads many of us to plan and prepare for different types of disasters, with procedures and systems in place to ensure that we can quickly get things running after an issue.

Natural disasters, or other large scale disruptions of business sometimes exceed the plans we’ve put into place. Fortunately they’re rare, but I ran across an article that talks about a few items that we might not consider when making plans. Even if you can fail your database over with an Availability Group running in another location, there are still some points you might want to think about.

Perhaps the biggest one for me is that testing is not optional. I’ve seen some amazing plans put into place, but never tested because no one wanted to disrupt ongoing business activities. Testing is really critical, perhaps because of the second more important item I see cause issues: failback. This isn’t always as simple as we’d like, even with some of the amazing work that Microsoft has done with SQL Server. We want to ensure that we do know that if something happens to our primary systems and we move them, we can come back. After all, these are the primary systems for a reason.

While other items such as making DR planning something your organization cares about can matter, I think the viability of this depends on how likely these disasters are. If you’re in a seasonal hurricane path, or some other disaster, you likely need to have better plans than if you’re in a stable region. Not that you can avoid any plans, but most of the time a disaster is a rare event and it’s really IT’s job to ensure that systems are resilient.

I do think it is worth noting that regulatory compliance isn’t optional. While your auditor might have sympathy if they are likewise affected by the disaster, that sympathy dissipates over time. A disaster next month might not factor into an audit review in 20 months. More laws are on the books and more are coming than most of us have dealt with in the past. Design your system and process to be compliant, even in a disaster.

Lastly, the people side of business can’t be emphasized enough. While some of us might be expected to shoulder a greater workload during a disaster, keep in mind that we would still need support outside of work, perhaps even a replacement worker to handle some of our work tasks or even help with personal items. The longer our lives are disrupted, the less likely we are to work as hard, or remain focused. Part of any plan should be how can our organization help support those that are struggling themselves. After all, the effects of a localized disaster on business could get compounded if staff stops coming to work.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 4.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

Disaster Recovery is both an exciting challenge for many DBAs and also a dreaded event that many hope they never experience. When a true disaster befalls your system, there is a tremendous amount of stress as we work to get things running again so clients can access data. Even the best laid plans will often have a glitch, and while it’s a great technical challenge, most of us would rather keep practicing and gaming possibilities than actually experience a disaster.

The majority of our system disasters are localized to the actual machine handling our workloads. We often don’t have outside disasters, like hurricanes, with disruption and damage to other parts of our infrastructure and ever our lives. When that happens on top of a down system, it’s a level of stress that can affect our health. I’ve been lucky in that I’ve experienced both kinds of disasters, but never at the same time. I hope you can say the same thing.

Let’s assume some local disaster defalls your system. Hardware, software, it doesn’t matter, but you end up with a corrupted database of some sort. This week I’m wondering if you have an idea of what the recovery time would be? How long before clients are up and running? Maybe more important, how long before you have system rebuilt?

If you have some sort of High Availability (HA) plan, then you might be back up for clients quickly, in minutes or even seconds. The disaster really isn’t over, however, since you are now running on less hardware than you planned. Until you can rebuild the downed node, you’re still in disaster mode. If you’re like most of the organizations that have employed me, you’ll also be stressed as your formerly well-designed HA setup is now a single point of failure and you’re scrambling to get the main node rebuilt.

We often think of a disaster as the time we’re down because of some event. Once clients are being served again, we tend to relax and think the disaster is over. It’s not, because until you replace the affected systems and bring them back online, you’re even more vulnerable than you were previously to another Murphy’s Law incident.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and Libsyn.