Voice of the DBA

Tag: disaster recovery

  • Someone Lost My Data

    It was bound to happen. I’m sure it’s happened before, but this event was interesting to me as it wasn’t a customer issue, but a cloud vendor problem. There was an outage in Azure on Jan 29, which happens, but in this case data was lost. There was problem internal code in the Azure cloud that dropped some customer databases using the Azure Key Vault with TDE encryption. That was slightly disconcerting for me as I was setting up and testing Azure Key Vault this week.

    There are frequent snapshots and Microsoft was able to restore the databases from one that was about five minutes old. Microsoft acknowledges that five minutes of data loss might be an issue and is asking customers that lost business or were affected by the drops to raise a support ticket. I have found Azure support to be pretty good about crediting my account when issues occur, and I hope they do the same here, though I wonder if they’ll compensate anything beyond the charges normally assessed to customers. They are offering credit for the restored and original databases for a few months as well.

    Plenty of people are upset, and with good reason. There should not be Azure management code that drops databases. Or should there be? Would this be any different on premises?

    I’ve had cleanup code that removed resources after some time. I don’t work at the scale of Azure, so I usually have things removed after a month or quarter. With the scale of Azure and potential costs, they might remove resources quicker, but I certainly have seen similar home grown, if-this-then-that code that does x when y occurs. In this case, I remove encryption keys, which might be not actually be removed for month or more. At that point, there is code that trigger a drop of databases. I’ve certainly seen users in various organizations drop, or restore over, the wrong database. In some cases they don’t realize it in five minutes, and often don’t have a way to restore from a five minute ago backup. Actually, relatively few people I know have RPOs under five minutes.

    This is bad, but it isn’t necessarily out of the ordinary for complex IT environments. If this happened in an organization, the IT staff would be worried and hoping for forgiveness. Some people would want others fired, but most of the time management would understand these things happen. Perhaps not if this isn’t the first time, but usually we accept some people make mistakes. Most of us don’t have complete control of all aspects of our environment. We depend on network staff, storage people, employees that manage hardware, and more. It’s possible that any one of these people could destroy data inadvertently.

    That happened here, though I don’t want to make excuses for Microsoft. They’re supposed to hire the best people and build processes that are better than what I’d expect inside an organization. Events will cascade into different areas, and there should be circuit breakers that prevent anything that could cause data loss in those events. Protect other people’s data with more care than you think you need. It’s your responsibility to do so.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • DR Prep Can Miss the Little Things

    The other day we had a blizzard in Colorado and we weren’t quite prepared. My wife and I were away on a trip and watching the weather. We were concerned about flights and about the conditions at the ranch with someone else in charge of horses. We were lucky in that the weather didn’t come as quickly as predicted and we made it home before the snow started.

    Depending on moisture and temperature, we sometimes put blankets on horses, which is a chore that I’m not very good at completing. Fortunately the kids were around and at 8pm, we all dressed in warm clothes and headed out to the barn. The horses depend on us to provide for some of their needs, so we each had jobs to do. One kid secured a temporary holding area to keep horses nearby. One gathered 14 or 15 blankets from storage and laid them out. My wife made grain to facilitate attracting the horses and giving us a way to hold them while blankets went on.

    Me? I had the envious task of double checking electricity and tank water headers. Our employee had thought that one was shorting out and at night, with a flashlight and plug tester, i got to debug 4 water heaters, including one that’s semi-remote from the barn. Testing for shorts involves touching water to see if there is any flowing power (this is very low amperage), which isn’t terribly dangerous, but is a little daunting. We managed to get everything done, and felt like we were prepped for a 4-10″ snowfall, 50mph winds, and a 0-5F temperatures.

    Later I lay in bed, trying to relax with a little TV before calling it a night. While the wind was howling, I heard a pop and the power went out. We have a generator, and I wasn’t too worried, but I did want to ensure it came on. I walked downstairs and across the house to listen for the engine noise. I heard cranking, but the engine didn’t catch. It was then I remembered the tank was low the last time I checked, and with a weekly diagnostic auto run for 5 minutes, I was likely out of propane. Since we needed power to heat water for horses, I had to get up around 11pm and take care of things.

    Fortunately I keep a spare bottle of propane and I went outside in the blowing snow to change it. A true work-at-home-techie, I did this without pants and got things running. Back to bed, though worried a bit about how to get more propane in the morning, just in case of an extended outage. The power came back on in the morning, but I still needed to get more propane just in case.

    Extended disasters sometimes cause problems with our plans because many of us focus on the immediate reactions. Longer term, things like additional fuel, food for humans, even shelter and child care are issues that I’ve had to deal with in teams that no one had planned for. We’ve seen this over and over again in the world, especially when supply chains break down quickly in disaster situations.

    I could have prevented some of the issues and stress if I’d prepped things better. I should have filled propane tanks earlier, knowing that bad weather can come at any time, and likely will at some time. Double checking heaters and power earlier might have made the blanketing process go quicker. There are always things we miss in prep, often ones we don’t consider to be important, but may be difficult to deal with in the moment. Take a few minutes and think about the little things you might dismiss as not important. Imagine how hard some of them might be to deal with under the pressure and stress of a DR event. If you can do a little more prep, some maintenance, or get some work done early, now might be the time to do it.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • DR Planning

    Data professionals know that protecting and ensuring our data is safe is one of our primary jobs. Many of us worry that if we were unable to recover a database, our job might be jeopardy, which makes sense. If we lose data, an organization might make the decision that we aren’t trustworthy enough to manage a database system. This leads many of us to plan and prepare for different types of disasters, with procedures and systems in place to ensure that we can quickly get things running after an issue.

    Natural disasters, or other large scale disruptions of business sometimes exceed the plans we’ve put into place. Fortunately they’re rare, but I ran across an article that talks about a few items that we might not consider when making plans. Even if you can fail your database over with an Availability Group running in another location, there are still some points you might want to think about.

    Perhaps the biggest one for me is that testing is not optional. I’ve seen some amazing plans put into place, but never tested because no one wanted to disrupt ongoing business activities. Testing is really critical, perhaps because of the second more important item I see cause issues: failback. This isn’t always as simple as we’d like, even with some of the amazing work that Microsoft has done with SQL Server. We want to ensure that we do know that if something happens to our primary systems and we move them, we can come back. After all, these are the primary systems for a reason.

    While other items such as making DR planning something your organization cares about can matter, I think the viability of this depends on how likely these disasters are. If you’re in a seasonal hurricane path, or some other disaster, you likely need to have better plans than if you’re in a stable region. Not that you can avoid any plans, but most of the time a disaster is a rare event and it’s really IT’s job to ensure that systems are resilient.

    I do think it is worth noting that regulatory compliance isn’t optional. While your auditor might have sympathy if they are likewise affected by the disaster, that sympathy dissipates over time. A disaster next month might not factor into an audit review in 20 months. More laws are on the books and more are coming than most of us have dealt with in the past. Design your system and process to be compliant, even in a disaster.

    Lastly, the people side of business can’t be emphasized enough. While some of us might be expected to shoulder a greater workload during a disaster, keep in mind that we would still need support outside of work, perhaps even a replacement worker to handle some of our work tasks or even help with personal items. The longer our lives are disrupted, the less likely we are to work as hard, or remain focused. Part of any plan should be how can our organization help support those that are struggling themselves. After all, the effects of a localized disaster on business could get compounded if staff stops coming to work.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Your Recovery Time

    Disaster Recovery is both an exciting challenge for many DBAs and also a dreaded event that many hope they never experience. When a true disaster befalls your system, there is a tremendous amount of stress as we work to get things running again so clients can access data. Even the best laid plans will often have a glitch, and while it’s a great technical challenge, most of us would rather keep practicing and gaming possibilities than actually experience a disaster.

    The majority of our system disasters are localized to the actual machine handling our workloads. We often don’t have outside disasters, like hurricanes, with disruption and damage to other parts of our infrastructure and ever our lives. When that happens on top of a down system, it’s a level of stress that can affect our health. I’ve been lucky in that I’ve experienced both kinds of disasters, but never at the same time. I hope you can say the same thing.

    Let’s assume some local disaster defalls your system. Hardware, software, it doesn’t matter, but you end up with a corrupted database of some sort. This week I’m wondering if you have an idea of what the recovery time would be? How long before clients are up and running? Maybe more important, how long before you have system rebuilt?

    If you have some sort of High Availability (HA) plan, then you might be back up for clients quickly, in minutes or even seconds. The disaster really isn’t over, however, since you are now running on less hardware than you planned. Until you can rebuild the downed node, you’re still in disaster mode. If you’re like most of the organizations that have employed me, you’ll also be stressed as your formerly well-designed HA setup is now a single point of failure and you’re scrambling to get the main node rebuilt.

    We often think of a disaster as the time we’re down because of some event. Once clients are being served again, we tend to relax and think the disaster is over. It’s not, because until you replace the affected systems and bring them back online, you’re even more vulnerable than you were previously to another Murphy’s Law incident.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and Libsyn.