Voice of the DBA

Tag: disaster recovery

  • Cloud Safe

    I saw a question recently from an individual that was trying to decide if it made sense to move their databases to the cloud. In this case, management wanted to move, but the technical staff had concerns about Disaster Recovery in the cloud. These are very valid concerns for any system, as technical staff is often responsible for data, regardless of who decides on the architecture of a system.

    The concerns got me thinking a bit. Is the cloud safer for DR? It’s certainly an “it depends” question, especially as the “cloud” isn’t necessarily the same thing for each of us. Some of us would use IaaS, with VMs in the cloud. Others might choose PaaS, with RDS or Azure SQL Database as a platform. Still others might want something like Managed SQL Instances, which is like IaaS+, or maybe PaaS#. I’m not completely sure how to classify this.

    In any case, your choice of cloud architecture can mean better or worse DR. The closer you are to IaaS, the more that you still have the same responsibilities that you might have inside your own data center. The difference is that hardware replacements or options are often quicker to procure, though perhaps with limited choice.

    If you choose PaaS, then you have different DR capabilities and responsibilities. Your vendor might handle some aspects of DR and remove the need for you to worry about hardware, or regular backups, but you might need to worry about other items. Your vendor might give you PIT recovery, but you might not want a database replacement in a busy system, especially if you’ve processed a few thousand transactions since someone ran that UPDATE without a WHERE batch. In that case, perhaps you want to ensure you can restore your database elsewhere, or you have other options.

    Many of us know that managing systems is complex work. Not every environment can be handled in the same way, and we often implement exceptions in both technology and staff knowledge. Ensuring your application and environment can recovery from a DR situation often requires detailed knowledge of both requirements and capabilities of the environment. While I’m not afraid of migrating to the cloud, I’d want to be sure I was prepared to answer questions from management if there are issues. After all, they’re going to look to me, not some vendor, for answers.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

     

  • A Double Dose of Trouble

    It’s Friday the 13th, during the month of October. Halloween, a strange holiday associated with horrible events, and Friday the 13th, a bad luck day, are happening this month. Both are associated with horror movies, implying that this isn’t necessarily a good month for the world. It’s a double dose of bad luck, which is what Equifax would have you believe. I wrote about this yesterday, where the company would have you believe a single employee makes a mistake when a scanner fails, and hackers were ready to attack.

    I’ve, in general, had a lucky career. I’ve worked hard, and things have gone my way. That doesn’t mean I haven’t made mistakes or systems haven’t gone down, but overall things have gone well. As I think about Friday the 13th of October, I’m reminded about Murphy’s Law, where things that can go wrong, may just do so. I’ve tended to expect that some things will go wrong and prepare accordingly. In fact, in keeping with a corrollary that I’ve sometimes heard to Murphy’s law that if one thing goes wrong, I’d expect that something else will fail.

    On this double downer of a day, I’m wondering if you’ve experienced that. Have you had something bad happen at work, only to find that overshadowed by something worse? Hopefully you didn’t make a security mistake and then get hacked, but perhaps something else has broken?

    In my career, I’ve had this happen, though not directly to me. I consulted with a small company to make a little extra money and because I liked their business. I was rooting for them to succeed and wrote some database code to help speed up their application. One day they call me to say that a drive array failed with the database mdf on it. I wasn’t worried, and while in another state, I was ready to walk them through a restore process on the new array. Unfortunately, this was a default install, and the default backup path was on the old array. I hadn’t set this up, nor was I aware of their backup process.

    The lead developer was proud of himself. He’d made a backup the day before, using a file share on a network drive. He had the .bak file. I thought this would be a quick and easy call until he read the error message that not all media sets were provided. That’s when I realized that the developer, while making the backup in SSMS, hadn’t cleared the previous backup file from the dialog. He’d inadvertently created a striped backup.

    Eventually they managed to locate a backup that was many days old. I’m not sure how far back, but they angered a number of clients, who I’m sure all started looking for a new vendor. Eventually the company failed, and I moved on. There wasn’t much to be done in that situation, but it was a certainly double dose of things going wrong at the same time.

    I hope few of you have had double the bad luck in your career, but if you can share a story, we’re happy to commiserate with you.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Backup Plans for Data Loss

    The British Airways computer failure has been on my mind for a few weeks. It seems like such an epic failure in 2017 from a high public company that many, many people depend on. I still struggle to believe the power explanation, and really, I’m not sure there’s any explanation that I would accept. At this point in our industry, there’s no good reason for any large, global company to not be able to restart services inside of a couple hours in a DR site. In fact, it really shouldn’t even take that long.

    However, for many of us, we will have a failure or disaster at some point. It might not even be a hardware failure or system crash. It’s much more likely that a user will cause an issue. As our systems grow larger, perhaps even more loaded with transactions, we might not always be able to easily separate out good data from bad, and I would expect we’ll experience a restore.

    For many of us this will mean we will lose some data from the system. Even with frequent log backups, we might end up with a short period where we can’t recover data. Most of us should have conversations with business stakeholders on what the acceptable level of data loss is, and plan to meet those requirements. We should also have plans around how to rebuild data. I wouldn’t recommend a full test on a system, but it might be worth a few conversations with those that deal with transactional data and discuss how the latest data might be recreated.

    No one wants to lose data, and in many cases, there are ways to rebuild or recover the data with manual efforts. Perhaps your company has paper records, or maybe there’s an audit trail that could be used to reconstruct actions. Maybe you rely on memory or even customers to provide information again. Today I’m wondering if you’ve thought about how you might recover data in a non-technical way and what methods you’d use.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Need for DRE

    “Today’s database professionals must be engineers, not administrators”

    That’s a quote from an interview at O’Reilly on Database Reliability Engineering. I don’t want to quibble about whether those of us working in technology are engineers in name or not, but in function, I do think the world is changing to one where we need to do more than be caretakers of systems. We need to be actively involved in ensuring we have very reliable, well-designed systems that use the best practices and patterns of highly available and reliable database systems.

    There is advice to learn and work closer with developers. I like that, and I do think the idea of learning to script and automate tasks is important, perhaps more important all the time, for data professionals. Some of the advice to look towards the quick moving, limited lifetime of many software components today doesn’t make sense. A database, at least some parts of a database, don’t fit within that model. There are valid reasons why a datastore must provide some stability and persistence for an application and won’t fit within a model of multiple systems. Or at least not practically within a model.

    However, there is good advice in the piece to learn more about different types of database services and patterns, and consider other ways of implementing datastores other than a RDBMS. I wouldn’t necessarily abandon a RDBMS for some NoSQL store just because developers think it’s easier, but I would consider whether I actually need to very tight coupling between different parts of the database and complete consistency. I think there are a fair number of domains where a CQRS pattern or some distributed store would work well. The move to microservices might be an enabler for your business if you consider the advantages for OLTP type transactions.

    The flip side of all the power that many stores provide is that many also make reporting and aggregating information more difficult. I would venture that in many cases, a data warehouse (perhaps even a RDBMS-based or columnar store) is necessary, along with the ETL process necessary to keep it up to date. These aren’t simple processes, and take resources to build. For most of us, because we don’t work at extreme scales, I’m not sure it’s worth leaving a relational platform, but I do think that we can learn to evolve and enhance our relational databases faster with DevOps ideas and techniques.

    Steve Jones