Voice of the DBA

Tag: disaster recovery

  • Data Loss or Downtime

    I was watching Kimberly Tripp of SQL Skills talk recently about VLDB disasters and how to recover from them. One of the first things she said in the session was getting a damaged database back online, even without all of the data, was important. Often her clients need to keep working, and it is important that they get the system back online, even without all the data. This allows business applications and business people to get back to work.

    That is interesting. I had always thought of my production OLTP databases as needing to be online, but also needing all the critical data. After Mrs. Tripp’s talk, I had to rethink that a bit and consider that a little data loss might be acceptable.

    To me this is a topic that is worth understanding. At the very least, it will help you make decisions in the event of some disaster for how you will proceed. So for this Friday’s poll:

    So what is more important to you: downtime or data loss?

    My feeling is that most of the people would really rather have the database online, even without all the data so they can continue to work. I realized that most of the time, getting the site back up, having lookup and other types of ancillary data (like products, prices, etc), was the most important thing. Recovering other data such as older orders, was secondary.

    Once the database is up, you can then work on getting other data back and merging it into the production system.

    Let us know what you think this Friday and what’s more important to your business (and why).

    Steve Jones

    Originally Published at http://www.sqlservercentral.com/articles/Editorial/72117/

  • The Institute for Backup Trauma

    This is an editorial reprinted from April 27, 2005.

    It’s viral marketing, but it’s not bad. Livevault produced a short video starring John Cleese as the director for this institute that seeks to counsel those that have had problems with tape backups. “Such an unreliable medium”, this one is straight out there. An advertisement for Livevault, a little silly, satirical, but not that bad.

    It’s an interesting concept and one that I’ve debated with various colleagues over the years. Tape v disk. Which one is better for securing and ensuring the integrity of your data, which one is better for business continuity, which one works for your budget.

    When I started in this business, it was a no brainer. Disk was way more expensive than tape and there wasn’t even a choice. Nowhere I worked could afford to backup their data to disk. On top of that, the technology for managing disk backups wasn’t great. I remember working for a small company and we couldn’t afford a VCS system. At least, I couldn’t pry money out of my boss for one, despite the fact that the business lived and died based on the code that was written. So we setup 5 folders on the network: Monday, Tuesday, Wednesday, Thursday, and you guessed it, Friday. Under each one, the other developer and I each had our own folder and a batch job that would copy data off our desktops to the appropriate folder on the appropriate day. We erased everything older than 5 days, though we still had tapes if we needed them.

    Today, with disk being nearly as cheap and much, much faster than tape, and the need to go back more than one version in a business environment being fairly rare, it makes some sense to use disk. Microsoft released their own software, that helps with data recovery on disk, so there must be either some growth or a very high profit margin in this area 🙂

    Personally, I think that the idea is a good one, but like many new technologies that change the way a user works, this one will take some time to get used to. I’m all for users being able to recover their own data without calling an admin, especially when the admin is me, but I’m sure that there will be lots of calls on how to find the files, which version, I forgot to save it, but my machine crashed and I want to go back 3 versions, annoyed calls when the last version on disk rolls off, etc.

    But I wouldn’t give up my tape. At J.D. Edwards, we backed up the databases to disk, then to tape that night, with two copies of the tapes being made. One went offsite and one was in a rotation on site. The triple protection served me well and we never had issues getting data off disk or one of the two tapes.

    Of course, that was expensive, so it’d not an option I’ve had often. Still, one disk and one tape worked out well and these days having disk be so cheap, a second copy on disk is something I’d shoot for.

    Steve Jones

  • T-SQL Tuesday #13– The Business and Disaster Recovery

    I’m hosting this month’s T-SQL Tuesday blog party, but that doesn’t mean I can’t participate. I posted the invitation last week, and hopefully we’ll have a large number of people participating today.

    The Cost of HADR

    I’ve dealt with a lot of business people over the years, and for a variety of reasons. At one job, I worked for a small company, literally about 50 people, and we had a number of systems critical to our business. We had a large fax server, receiving hundreds of faxes a day from overseas. We had a large imaging server/platform in which we stored our faxes along with quite a bit of other business correspondence that we needed a record of, and we had an inventory management/sales system running on SQL Server that people used every day to run the business.

    These were all important systems, and if they were down, business would slow to a crawl. The owner of the company wanted to take advantage of all our technology and mandated “no downtime” and “zero data loss” to my boss, the VP of sales for the company. He was, of course, happy to push this down to me as a goal for the next year.

    This was in 1997, and as I investigated our possibilities for achieving this, I realized very quickly that it wasn’t a good move for the company. Clustering in SQL Server v6.5 was shaky and hard to install and maintain. Marathon Technologies had a machine that was basically a “cluster-in-box” that would do a lot of what we wanted, but it would have been around a $30k server. Since our two SQL Servers cost about $7k together, this didn’t seem like a great solution to me.

    Instead I investigated what downtime cost us. What would it cost in terms of work and salary if we had to re-enter data that might be lost. We had been down for various periods of time due to different events, and the business had survived, even one day when we were shut completely down for 2 days.

    It quickly came to light that we’d have to be down for over a week to really cost the business more than $30k, and that was debatable. We could still conduct lots of work on paper and over the phone, so it didn’t make sense to aim for zero data loss and zero downtime. I explained it to my boss, laid out a simple case for him with a few numbers highlighted, and the provided an alternative for investment in a little redundancy, designed to minimize losses, but not prevent them.

    In the end the big boss agreed. He hadn’t really meant zero loss, after all, anyone in business knows that there are always going to be some losses. What he wanted was his business to keep running, cost effectively. That’s what we gave him at a much more reasonable price.

  • VLDB Issues at SQL Server Connections

    Kim Tripp is a great speaker, with a wealth of knowledge. This session was talking about some of the issues with VLDBs.

    What’s a VLDB? Definitely the TB range, but Kim talks about the issues that come with 100s of GBs as well.  She mentions one client with a 3TB table. Definitely time for VLDB planning.

    She also says that in the last couple of years, almost all of her clients use some type of partitioning. Granted, Kim tends to work with larger, more expensive operations, but interesting to see that partitioning is part of many clients.

    This is a DR session. When data is damaged, what do you do? The first think is think about recovery as a step by step process. Don’t freak out.

    Your first questions should be who, what, and when. Who noticed it, what is wrong, and when did it happen. Key questions to know, not necessarily for blame, but for recovery purposes. Knowing who is important for security or tracing purposes. Kim says no select/insert/update/delete on tables. Always some other mechanism to access data.

    If you don’t know who did something, then it becomes more of a detective case to investigate what might be wrong. Definitely more challenging if you don’t know who did something.

    What do you do first? It depends. You might not be able to take the database or server offline, so what do you do? It’s an issue. You can’t usually shut down the database services, so you must think carefully about what you will do.

    Recovery from a dropped table

    Interesting. A nice sales/customer/employee/product table relationship, with “Sales” being dropped. That’s the easiest one to delete with FKs, but it’s the most important table.

    A good backup is needed first. As Kim notes, if you don’t have a good backup, you can’t go very far in recovery.

    The sales table gets dropped, and one of Kim’s insert scripts starts failing. As expected, if this for real, likely your phone is going to ring.

    First thing: note the time that you first hear of something going wrong.

    Second thing, Kim looks to set the database access to “restricted access” meaning only dbos are allowed in. I have never used this, but if you really lost something big, you want to do this. If this happens, that means that your application also needs to handle this and show a good message to clients.

    Is it better to be back up or have all the data. That’s a good question. It probably will be different for each business, and maybe each database. What is more important? Might be just getting back up in many cases.

    MAKE SURE YOU GET THE TAIL of the log if you can. You don’t want to restore without having this.

    Use stopat in all restore statements. It doesn’t have an effect in RESTORE DATABASE, but allowing this means that you get in the habit of using it in all RESTORE commands. The same with the NORECOVERY option.

    Her final statement has a WITH RECOVERY, RESTRICTED_USER. That’s nice to have, and let the DBA check without users/systems connecting right away.

    Right away, Kim creates a snapshot. She can then easily go back to this point. This is also then used to compare to an investigation database that is a copy of what you restored.

    She also resets the identity seed to a higher value in this case. She has a gap, but it’s a known gap that might be useful if you recover more data.

    One of the key things that I see is Kim keeping notes of times, and what is done, as she goes through the recovery process for this demo. That’s key in disasters. Having a pad/pen or Notepad around is indispensible.

    Kim has a cool script that starts running log restores with ever incrementing STOPAT values to find out where there was an issue. In each restore, then uses tablediff to grab the differences between the snapshot and the test restore to get the missing data that might not be in the database that is now live.

    Main points:

    • contain the damage
    • decide if data loss or downtime is more important
    • recover the damaged data
    • bring offline components online
    • prevent this in the future

    A good DR list to keep handy, and think about first.

    Containment part 2

    One important thing is to contain the issue and perhaps not take the entire db or server offline.

    One thing in Enterprise Edition is to take the secondary files offline, can be useful to take an .ndf file offline. You cannot take an .mdf or .ldf offline, but you can with a .ndf, so for large dbs, separating tables into separate filegroups is a good thing.

    Kim shows an online index rebuild that moves a table across multiple files and it’s pretty cool. While users are in the system, the table is moved to a new filegroup/files. This partitions the table to get better control of disasters.

    Now we lose a file, and the file needs to go offline. This may or may not affect your application. The connections are killed to the db, but a good application will retry and handle this as a transient error. Users that access the partition that is offline will get a trappable, level 16 error.

    Restoring a single file with the move command. That’s cool. Never had to do that myself. Not sure you could do this in SQL 2000 when I had a few disasters. Not sure I had Enterprise Edition in most of my jobs.

    Very cool session, highly recommended if you have to plan for, or execute, a DR plan.