Voice of the DBA

Tag: disaster recovery

  • 13 Disasters

    Hopefully you don’t have any server disasters because of coffee spilled on a machine.

    This editorial that was originally published on Aug 15, 2007. It is being rerun as Steve is traveling.

    This is an interesting list of disasters that can befall a production system. It comprises more than database servers, but certainly can happen to them as well.

    The list is pretty good and of these I’ve had more than a few of them happen to me. There were a few that I’ve never seen and it makes me worry a bit that this guy doesn’t have a good staff or set of vendors working with him. But one of these is very interesting.

    The second item on the list, a controller going bad and corrupting disks, is an interesting one in today’s world. What would you do if this happened on your SAN? Actually I know what you’d do. First you’d be in denial, and I don’t mean the river in Africa.

    Then you’d tell the SAN guys. They wouldn’t believe you. You’d argue and they’d check things, then they’d wonder how it could happen and why? You’d scream at them to fix it, probably using 4 letter, not 3 letter words. They’d start looking for ways to recover data and your hair would be slowly thinning as upper managers started calling down looking for answers.

    It shouldn’t happen, but it could. Some marketing VP is showing off that nice piece of SAN equipment with it’s high speed switches, dozens of drives, and lots of colored wires. He sloshes some coffee onto the system, freezes, but when the flashing red lights and siren from the Enterprise don’t go off (Red Alert!), he continues his tour while the controller scribbles on your disks.

    Restoring from backup might be easy. Of course if you have a system like some I’ve seen that shares physical disks among multiple LUNs, it might not.

    Disaster Recovery is rarely the hurricane Katrina type of issue. Usually it’s something like a disk drive, raid controller, cut wire, etc. that you have to deal with. So think about all of the minor disasters that you want to be sure you can handle and get some practice in on those. Make sure you have spare parts, you can rebuild a server (QA machines are handy for this), you know how to perform a restore.

    And most of all, be sure that you know where the backup files are stored. Preferably on different disks than the production data.

    Steve Jones

  • Always restore with NORECOVERY

    Always, always, ALWAYS restore a SQL Server database with the NORECOVERY option.

    It’s trivial to switch the database online.

    Not trivial to recover from an accidental restore with RECOVERY (the default)

    ‘nuff said.

  • The Colocated Dangers

    Titanic Poster
    It’s not “if”, but “when” you will have a disaster.

    We went down.

    It was quite a surprise for me, and something that hasn’t happened in a long time. It wasn’t for long, but a month ago, there was a period of time when SQLServerCentral was down. It’s also one of the very few times it’s happened since I was in charge of the servers. The exact reason was something to do with out hardware, and it was fixed relatively quickly by our hosting provider.

    Early on in the site’s history we had a few hiccups. I was the one that drove out to a co-location facility to replace hardware or fix something that was broken. There were a few times where all that was required was a reboot, but we didn’t have the “remote hands” service available at a few of our hosting facilities. Fortunately for me I had an understanding boss that would allow me to flex and extend my lunch time during these emergencies.

    Since Red Gate purchased SQLServerCentral, we’ve had excellent uptime. A few outages for upgrades of the site, a minor DNS issue, and a few hiccups from maintenance by our hosting facility, but all very short. This hour-or-so long outage was the first unplanned outage in years. However it reminded me of a somethings: it’s not if a disaster will occur, but when it will occur?

    Putting all your data (and the servers it lives on) in one place is the fact of life that most of us have to live with, but we ought to have some idea of a DR plan if our data center goes down, or our hosting company has an issue. It might not be a thoroughly documented plan, but it ought to be a thoroughly thought out one.

    At least if expect your enterprise to continue to function during the disaster and your employment to continue after.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • Backup and Restore

    Rainbow Six
    The team at Rainbow Six has to be prepared for a disaster anytime.

    Today’s editorial was originally published on May 13, 2007. It is being re-run as Steve is at SQL Server Connections.

    I think that it’s important to practice restoring and rebuilding your systems. You never know when you’ll encounter problems and need to restore your data onto another piece of hardware. It’s definitely something that people don’t often practice and with documentation not usually being up to par, it’s something that people should be comfortable with performing.

    I saw this article for CIOs that seemed to suggest that IT people aren’t testing their systems enough. The statistic in the article was that 89% only test once a year. Also that 67% were only minimally confident that their disaster recovery system would work as planned.

    I think it’s pretty good that 89% test their systems. I’m guessing that 11% do it more than once a year, but I’d be concerned if 89% hadn’t tested it at all! I think once a year is a pretty good plan given the hectic scheduled, heavy workloads, and the lack of buy-in by many executives for a DR plan. It’s not like we’re the guys from Rain bow Six that could be called on multiple times to perform this year. In all likelihood most of us will never encounter a disaster in our careers.

    And if you do, it won’t go as planned. I’ve had disasters and disaster tests in my career and none of them goes as planned. It’s like I’ve seen quoted: “No battle plan ever survives contact with the enemy.” We could easily paraphrase this for IT and note that no DR plan will every survive a disaster. You need people that can adapt, think in the heat of the moment, and get things working when the plan doesn’t cover every contingency.

    As DBAs I think most of us tend to practice the DR stuff on a more regular basis as part of our jobs. Many of us are constantly restoring databases for QA and development, getting practice that sysadmins don’t usually get. We also have the advantage of being separated from the hardware and we can install SQL Server and restore databases on any Windows server host.

    I think you should practice your DR skills in little bits. Maybe grab the tail of the log and do a point in time restore for your next QA cycle. But practicing a full plan more than once a year isn’t practical for most of us.