Data professionals know that protecting and ensuring our data is safe is one of our primary jobs. Many of us worry that if we were unable to recover a database, our job might be jeopardy, which makes sense. If we lose data, an organization might make the decision that we aren’t trustworthy enough to manage a database system. This leads many of us to plan and prepare for different types of disasters, with procedures and systems in place to ensure that we can quickly get things running after an issue.
Natural disasters, or other large scale disruptions of business sometimes exceed the plans we’ve put into place. Fortunately they’re rare, but I ran across an article that talks about a few items that we might not consider when making plans. Even if you can fail your database over with an Availability Group running in another location, there are still some points you might want to think about.
Perhaps the biggest one for me is that testing is not optional. I’ve seen some amazing plans put into place, but never tested because no one wanted to disrupt ongoing business activities. Testing is really critical, perhaps because of the second more important item I see cause issues: failback. This isn’t always as simple as we’d like, even with some of the amazing work that Microsoft has done with SQL Server. We want to ensure that we do know that if something happens to our primary systems and we move them, we can come back. After all, these are the primary systems for a reason.
While other items such as making DR planning something your organization cares about can matter, I think the viability of this depends on how likely these disasters are. If you’re in a seasonal hurricane path, or some other disaster, you likely need to have better plans than if you’re in a stable region. Not that you can avoid any plans, but most of the time a disaster is a rare event and it’s really IT’s job to ensure that systems are resilient.
I do think it is worth noting that regulatory compliance isn’t optional. While your auditor might have sympathy if they are likewise affected by the disaster, that sympathy dissipates over time. A disaster next month might not factor into an audit review in 20 months. More laws are on the books and more are coming than most of us have dealt with in the past. Design your system and process to be compliant, even in a disaster.
Lastly, the people side of business can’t be emphasized enough. While some of us might be expected to shoulder a greater workload during a disaster, keep in mind that we would still need support outside of work, perhaps even a replacement worker to handle some of our work tasks or even help with personal items. The longer our lives are disrupted, the less likely we are to work as hard, or remain focused. Part of any plan should be how can our organization help support those that are struggling themselves. After all, the effects of a localized disaster on business could get compounded if staff stops coming to work.