Voice of the DBA

Tag: disaster recovery

  • DR Failovers

    failureAlmost everyone struggles with setting up disaster recovery plans and resources. There are a few companies that take their DR seriously, but for most organizations, it’s an afterthought. It’s an insurance premium that can easily be avoided if there are not pressing problems and your past experience with disasters is minimal. After all, it’s rare that any of our data centers shuts down because of an earthquake, hurricane, fire, or other similar large scale event.

    However most of us try to have some type of disaster recovery in place. We may have cold or warm systems available. Our companies have funded an AlwaysOn Availability Group, or more likely, mirroring and/or log shipping for critical systems where data is moved to a remote location on a regular basis. We monitor these processes, and we try to keep them running, though I’m sure if they break, many people don’t give the repair top priority in their daily work.

    A DR environment is like a backup. If you don’t test it, you’re never sure if it really is something you can use in a disaster situation. You may periodically test your fail-over, like you test a restore, but do you ever really lean on the secondary system? This week, I wanted to ask you this question:

    Do you fail over to your DR system and run your business from the system for a day (or week or longer)?

    I know a few companies that consider secondary systems to be critical and will actually fail over, and then run the other system for a few months, failing back to then retest the primary system. In this case, there really isn’t a primary and secondary system, but rather two systems that can work, being alternately used throughout the year.

    This is actually moving closer to a cloud architecture model, where you don’t place high importance on any particular system, you assume any system can fail, and you have redundant systems that can pick up the load. In a cloud environment you might have more than two, relying on dozens of systems instead, any of which could fail, but with a small interruption in service.

    I’d hope that SQL Server, the version of the platform I can install in my data center, would get close to this, allowing me to serve database services to clients, but seamlessly moving those services across instances, with clients unaware when physical machines have crashed because their services just moved to another host.

  • Test Your Restores

    Test your restores: he first tip in my Hard Earned Lessons of a DBA piece.

    I can’t emphasize this enough. I’ve been called in as an employee and consultant to companies that never had tested a restore. Quite a few of those times there has been data loss.

  • Prepare for Disaster Webinar

    I’ve been asked by Red Gate to give my “Prepare for Disaster session on a webinar on July 23, 2013. It will be at 10:00am MST/12:00pm EST/ 5:00pm GMT.  This is a session that talks about some of the disaster issues I’ve faced, and you might face. I then “rewind the clock” and look at ways that we can prepare ahead of time and either prevent, or mitigate the impact of the issues.

    The registration link is here.

    For once I’m doing this from the comfort of home instead of being on the road somewhere, which is good and bad. I don’t love presenting webinars, since I really like the audience feedback. Talking to the wall isn’t something I enjoy. However I don’t have the hassles of travel, the time away from home, and I get to sleep in my own bed. All in all, it’s a win.

    If you’re interested in learning about how to get ready for disasters ahead of time, register and I’ll see you on the 23rd.

  • The Decision to Fail

    failoverWe have a number of automated technologies that we can us to seamlessly move from primary to secondary systems without human intervention. SQL Server incorporates a number of these, and many companies use them to ensure their applications are highly available. However things don’t always go as planned in a disaster and sometimes humans get involved.

    Unless you are one of the companies with a very large budget and high risk of business issues when systems failover, you probably have some sort of high availability (HA) or disaster recovery (DR) process that requires human intervention. Log shipping, for example, usually requires that some human reconfigure the application to use secondary servers. Even with Availability Groups, clustering, or database mirroring, you may need to manually fail back to primary systems.

    In those cases, it’s not always a clear decision to do so. Many of the switches are disruptive, or have the potential to be disruptive. Cluster fail-overs should not impact the application, but there is a brief period where clients may not connect. Outside of disasters, Management, and often technical people, usually want to schedule any failovers after they have prepared the end user for potential issues, however brief.

    In disaster situations, when there hasn’t been a complete failure of a system, you may not want to have unscheduled failovers right away. This week I want to know:

    How do you make the decision to fail over from one system to another?

    I’m speaking to you, the data professional or the administrator. I would guess that most of you are not the one that ultimately makes the decision to leave your primary systems. Often I’ve found that someone in management has to make the decision, but with input from the technical people. In that case, think about how you present the situation and pros and cons of the failover. Do you give hard numbers, like latency and relative CPU power in failover machines or do you attempt to quantify the effects on the business when secondary systems are in use.

    I’ve rarely had a large budget for secondary systems. Network bandwidth, CPU and memory, and more are sometimes sacrificed in secondary systems in order to align the cost of these systems with the risk of needing them. In many cases, we didn’t have automatic failover for many systems because we had to know our primary systems would be down for more than 5 or 6 hours before we would switch to the backup environment.

    If you have similar guidelines or processes in place, let us know.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.