Voice of the DBA

Tag: disaster recovery

  • Expect the Unexpected with DiRT

    Disaster recovery is one of the core tasks that many DBAs think about on a regular basis. Ensuring that we can get our data back online, available, accessible, and intact is important. More than a few DBAs that haven’t been able to recover systems, find themselves seeking new employment.

    That’s not to say that most DBAs perform perfectly under pressure. Plenty make mistakes, and there may be times when they can’t recover all data. There does seem to be a correlation between how often DBAs practice recovery skills and how well they perform in an actual emergency. I know that at a few companies, we scheduled regular disaster tests, though often with simulated recovery of a systems that didn’t expect to actually take over a workload. Arguably not a good test, but better than nothing.

    Google takes things a step further. They have annual, company wide, multi-day DiRT (Disaster Recovery Testing) events. These are across many departments and can be substantial in terms of the disruption that the these events cause to their infrastructure. This is a way for the various individuals responsible for infrastructure to actually evaluate if they are prepared for potential issues.

    If you read the article, you find that Google started small with these and progressed them to larger, more inclusive tests, like taking down a data center. They also whitelist some servers, knowing they cannot pass a test, so there is no reason to actually take them down. After all, business still needs to work.

    It’s good to have tests and walk through actual events, like call lists and bridges to be sure that communication and documentation work. This might be especially important when teams often expect that all their written procedures are available. I went through an audit with one company, where we failed immediately when all our DR plans were on a network share. In this simulation, we had experienced a network failure and servers had crashed. We were supposed to bring up the systems on spare hardware, but some critical documentation wasn’t available without a network. We started printing things out right away so that we could continue on with the simulation (as well as have this in a binder in our office).

    Not everyone can schedule large scale tests, and certainly many managers don’t see the point. They’ll often want to gamble that staff will “figure things out” if there is an incident. That doesn’t mean that DBAs and sysadmins can afford to wait for a disaster to practice some skills. Be sure that everyone on your team can recover databases, they know where backups are (or how to determine this), and multiple people have access to resources. The last thing you want is a disaster to occur during your vacation and have managers calling you to cut short your holiday because you’re the only one that knows where something is or has the authority to access a resource.

    Think about this ahead of time and prepare.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • DR Priority

    Those of us that act as DBAs or sysadmins for database systems know that DR is a point of emphasis for us. We usually spend plenty of time ensuring backups are working and practicing restores. The automated scripts and processes that people use are some of the most popular and discussed topics on SQLServerCentral.

    However, we can’t ensure every system is protected at the same level. It’s not cost effective to cluster or build AGs with hot standbys, or even warm standbys, for many databases. Often our organization will ensure some are ready to go and others will have to be dealt with if there are issues.

    William Durkin noticed recently that O’Reilly hadn’t prepared well enough for their learning site. They were affected by the fires and power outages in California and since they host some of their systems in an on-premises data center, there were issues. Certainly we might think they hadn’t prepared well for DR, and perhaps this is a fair view of their service, but perhaps they made the decision not to built out an expensive DR environment for a service that can tolerate some downtime.

    This week, I wonder how some of you look at the systems you support. Perhaps you are the person that has to make decisions, or perhaps your organization doesn’t fund DR well. I’m wondering, how do you decide which systems don’t get enough DR support?

    Certainly there are inexpensive, perhaps crafty ways that some people might plan for DR. I know I’ve cobbled together systems from spare parts to use for testing restores, with the idea that the hardware might need to be an emergency DR server for a single system or two in the event of an incident. If you’ve got ideas on how to be prepared even without organizational support, let us know today.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Restoring an Expired Certificate–#SQLNewBlogger

    Another post for me that is simple and hopefully serves as an example for people trying to get blogging as #SQLNewBloggers.

    One common task that you might need to handle if you work with encryption is dealing with a certificate. Whether this is for TDE, SSL, or something else, you will want to ensure that you know how to restore a certificate.

    I’ve written about this before (restoring a certificate), but what about restoring an expired certificate? If you forget to replace one in TDE, does it still work?

    Tl;dr yes.

    Validating a Restore

    I backed up an certificate from one instance that had expired. I used this in a customer demo a few years ago, and it was on my system. Here’s what I saw.

    2019-08-23 14_35_57-SQLQuery1.sql - Plato_SQL2014.MASTER (PLATO_Steve (57))_ - Microsoft SQL Server

    I ran the backup script shown to get the expired backup of the cert.

    Next, I connected to a new instance and ran this.

    CREATE CERTIFICATE FinanceCert
    
FROM FILE = N'C:\SQL\FinanceCert.cer'
    
WITH PRIVATE KEY
    
(
         FILE = N'c:\SQL\FinanceCert.pvk',
         DECRYPTION BY PASSWORD = N'AReallyStr0ngK#y4You'
    
);

    This completed, but I get the warning that this cert is expired. It’s a warning. SQL Server will still use the certificate to decrypt anything necessary, like a DEK for TDE.

    2019-08-23 14_37_20-SQLQuery2.sql - Plato_SQL2017.master (PLATO_Steve (55))_ - Microsoft SQL Server

    This is a good reason to keep an eye on certificates, as someone might be worried about expired certs. They work, but some people (auditors) do not like to see these in use.

    To change the certificate used, see Key Rotation in TDE.

    SQLNewBlogger

    Anything DR related is a good skill to showcase. Blog about your practicing different skills. The writing will cement the skills further in your mind and employers will appreciate you showing the way you handle things.

  • Dealing with Failure

    This past week was a dichotomy for Coloradans. On Tuesday, it was 60F (15C) for most of the day, with my daughter and I going to volleyball practice in t shirts and shorts. Our phones chimed during practice, noting that schools had been cancelled for Wednesday, in anticipation of a winter storm. Plenty of people laughed about that at the time, and no shortage of parents were annoyed by the premature decision, especially with little wind and rain falling at 7am Wednesday.

    That changed with high winds and some snow all over the area. Cars were stuck on roads, first responders overwhelmed, the airport closed, and people stuck in places between home and work. We hunkered down here at the ranch, and while horse care was hard, we survived. Power flickered for me, but I was able to work Wed and Thur without worry. I also have a generator, but we never lots power longer than the 30s it takes to kick in.

    Glenn Berry wasn’t quite as lucky, though he came through the storm fine. He did lose power, and wrote about his experiences. Glenn is about 12 miles S of me, and in a rural area. I thought his thoughts on the experience were interesting, and not a lot different from the ones I had recently with another power outage. I don’t mind storing gasoline, and we have multiple generators here because of the horses, but I appreciate Glenn’s thoughts on finding a different way of storing power.

    We both realize that disaster can strike and it’s important to think through the issues ahead of time. While I depend on my phone in the event of losing our Internet connection, I could easily lose that as well. The local towers probably run off batteries and in an extended outage, they’d lose power before I would. I have a tractor and can plow out our driveway, but in this case, the county roads were closed Wednesday and Thursday due to the poor conditions and number of cars stranded. Apparently Wed night visibility was inches and cars were being abandoned in the middle of roads. I’d prepared for a few days outage, but if this were still ongoing today, I’d start feeling the pinch from a lack of supplies.

    It’s not often we encounter a major disaster at work, but it’s entirely possible that the secondary support plans we have in place won’t work. You might have a supplier for more diesel fuel, but if everyone else needs more, will you still get some? What about issues with a complex process at work? If it fails and you end up rebuilding some ETL flow, do you have the staff or are you expecting a “Brent” to help explain things and do some of the work? I think plenty of us have that dependency, one which we don’t think about until that person is on holiday and we need their expertise.

    Disasters come in all sizes, but often they seem to come in the size that is at the limit (or just beyond) our preparations. I suggest that during some down times, take a minute and think about your preparations. Think what could go wrong. Do a little brainstorming and gaming of situations. If your plan has an issue, do you have another idea of what to do? It might be good to at least have thought about the potential issues.

    Steve Jones