Tag: disaster recovery

  • Disaster Recovery Docs–Offsite

    Years ago I worked at an ISO certified organization, where we took our administration and planning seriously. We ensured everything was backed up, we had backup drills, we each practiced recovering systems, loading tapes, even calling the company that rotated our tapes from an offsite location.

    One of our administrators was very conscientious and documented everything. He had screen images, copies of vendor documentation, all linked and labeled in a great folder on our network. We used this documentation constantly while restore systems from mistakes or drills and updated it regularly. It contained vendor contact information and account numbers. Each of us could access is remotely, and I had used it a few times from home while working on a problem with a system.

    One day our auditor came in and in the course of examining our details remarked on the comprehensive documentation and said it was great. Then he reached over and turned off our very detail-oriented administrator’s monitor.

    “Now show me your documentation,” he said.

    Our administrator was a little shocked. All the documentation was on the network. We’d accessed it from home. We accessed it from other corporate locations. We had restored it from tapes.

    We’d never thought it would be unavailable.

    It was a sobering lesson, one that had interns printing off copies that afternoon, dating them and binding them into books, one of which was always rotated offsite to the tape vendor’s facility.

    As for the tape vendor information itself? After overcoming his embarrassment, our administrator made wallet sized cards with the vendor company, phone number, address, and account number. He laminated those and gave us each one to keep on our person at all times.

    Make sure you have copies of the stuff you need offsite. It’s easy not with the “cloud”, though you better secure the data with encryption and limit the key access to a few people. You never know what you’ll need to recover, but you never should  find yourself in a position of not being able to recover your documentation.

  • DR Failovers

    failureAlmost everyone struggles with setting up disaster recovery plans and resources. There are a few companies that take their DR seriously, but for most organizations, it’s an afterthought. It’s an insurance premium that can easily be avoided if there are not pressing problems and your past experience with disasters is minimal. After all, it’s rare that any of our data centers shuts down because of an earthquake, hurricane, fire, or other similar large scale event.

    However most of us try to have some type of disaster recovery in place. We may have cold or warm systems available. Our companies have funded an AlwaysOn Availability Group, or more likely, mirroring and/or log shipping for critical systems where data is moved to a remote location on a regular basis. We monitor these processes, and we try to keep them running, though I’m sure if they break, many people don’t give the repair top priority in their daily work.

    A DR environment is like a backup. If you don’t test it, you’re never sure if it really is something you can use in a disaster situation. You may periodically test your fail-over, like you test a restore, but do you ever really lean on the secondary system? This week, I wanted to ask you this question:

    Do you fail over to your DR system and run your business from the system for a day (or week or longer)?

    I know a few companies that consider secondary systems to be critical and will actually fail over, and then run the other system for a few months, failing back to then retest the primary system. In this case, there really isn’t a primary and secondary system, but rather two systems that can work, being alternately used throughout the year.

    This is actually moving closer to a cloud architecture model, where you don’t place high importance on any particular system, you assume any system can fail, and you have redundant systems that can pick up the load. In a cloud environment you might have more than two, relying on dozens of systems instead, any of which could fail, but with a small interruption in service.

    I’d hope that SQL Server, the version of the platform I can install in my data center, would get close to this, allowing me to serve database services to clients, but seamlessly moving those services across instances, with clients unaware when physical machines have crashed because their services just moved to another host.

  • Test Your Restores

    Test your restores: he first tip in my Hard Earned Lessons of a DBA piece.

    I can’t emphasize this enough. I’ve been called in as an employee and consultant to companies that never had tested a restore. Quite a few of those times there has been data loss.

  • Prepare for Disaster Webinar

    I’ve been asked by Red Gate to give my “Prepare for Disaster session on a webinar on July 23, 2013. It will be at 10:00am MST/12:00pm EST/ 5:00pm GMT.  This is a session that talks about some of the disaster issues I’ve faced, and you might face. I then “rewind the clock” and look at ways that we can prepare ahead of time and either prevent, or mitigate the impact of the issues.

    The registration link is here.

    For once I’m doing this from the comfort of home instead of being on the road somewhere, which is good and bad. I don’t love presenting webinars, since I really like the audience feedback. Talking to the wall isn’t something I enjoy. However I don’t have the hassles of travel, the time away from home, and I get to sleep in my own bed. All in all, it’s a win.

    If you’re interested in learning about how to get ready for disasters ahead of time, register and I’ll see you on the 23rd.