Voice of the DBA

Tag: disaster recovery

  • A Double Dose of Trouble

    It’s Friday the 13th, during the month of October. Halloween, a strange holiday associated with horrible events, and Friday the 13th, a bad luck day, are happening this month. Both are associated with horror movies, implying that this isn’t necessarily a good month for the world. It’s a double dose of bad luck, which is what Equifax would have you believe. I wrote about this yesterday, where the company would have you believe a single employee makes a mistake when a scanner fails, and hackers were ready to attack.

    I’ve, in general, had a lucky career. I’ve worked hard, and things have gone my way. That doesn’t mean I haven’t made mistakes or systems haven’t gone down, but overall things have gone well. As I think about Friday the 13th of October, I’m reminded about Murphy’s Law, where things that can go wrong, may just do so. I’ve tended to expect that some things will go wrong and prepare accordingly. In fact, in keeping with a corrollary that I’ve sometimes heard to Murphy’s law that if one thing goes wrong, I’d expect that something else will fail.

    On this double downer of a day, I’m wondering if you’ve experienced that. Have you had something bad happen at work, only to find that overshadowed by something worse? Hopefully you didn’t make a security mistake and then get hacked, but perhaps something else has broken?

    In my career, I’ve had this happen, though not directly to me. I consulted with a small company to make a little extra money and because I liked their business. I was rooting for them to succeed and wrote some database code to help speed up their application. One day they call me to say that a drive array failed with the database mdf on it. I wasn’t worried, and while in another state, I was ready to walk them through a restore process on the new array. Unfortunately, this was a default install, and the default backup path was on the old array. I hadn’t set this up, nor was I aware of their backup process.

    The lead developer was proud of himself. He’d made a backup the day before, using a file share on a network drive. He had the .bak file. I thought this would be a quick and easy call until he read the error message that not all media sets were provided. That’s when I realized that the developer, while making the backup in SSMS, hadn’t cleared the previous backup file from the dialog. He’d inadvertently created a striped backup.

    Eventually they managed to locate a backup that was many days old. I’m not sure how far back, but they angered a number of clients, who I’m sure all started looking for a new vendor. Eventually the company failed, and I moved on. There wasn’t much to be done in that situation, but it was a certainly double dose of things going wrong at the same time.

    I hope few of you have had double the bad luck in your career, but if you can share a story, we’re happy to commiserate with you.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Backup Plans for Data Loss

    The British Airways computer failure has been on my mind for a few weeks. It seems like such an epic failure in 2017 from a high public company that many, many people depend on. I still struggle to believe the power explanation, and really, I’m not sure there’s any explanation that I would accept. At this point in our industry, there’s no good reason for any large, global company to not be able to restart services inside of a couple hours in a DR site. In fact, it really shouldn’t even take that long.

    However, for many of us, we will have a failure or disaster at some point. It might not even be a hardware failure or system crash. It’s much more likely that a user will cause an issue. As our systems grow larger, perhaps even more loaded with transactions, we might not always be able to easily separate out good data from bad, and I would expect we’ll experience a restore.

    For many of us this will mean we will lose some data from the system. Even with frequent log backups, we might end up with a short period where we can’t recover data. Most of us should have conversations with business stakeholders on what the acceptable level of data loss is, and plan to meet those requirements. We should also have plans around how to rebuild data. I wouldn’t recommend a full test on a system, but it might be worth a few conversations with those that deal with transactional data and discuss how the latest data might be recreated.

    No one wants to lose data, and in many cases, there are ways to rebuild or recover the data with manual efforts. Perhaps your company has paper records, or maybe there’s an audit trail that could be used to reconstruct actions. Maybe you rely on memory or even customers to provide information again. Today I’m wondering if you’ve thought about how you might recover data in a non-technical way and what methods you’d use.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Need for DRE

    “Today’s database professionals must be engineers, not administrators”

    That’s a quote from an interview at O’Reilly on Database Reliability Engineering. I don’t want to quibble about whether those of us working in technology are engineers in name or not, but in function, I do think the world is changing to one where we need to do more than be caretakers of systems. We need to be actively involved in ensuring we have very reliable, well-designed systems that use the best practices and patterns of highly available and reliable database systems.

    There is advice to learn and work closer with developers. I like that, and I do think the idea of learning to script and automate tasks is important, perhaps more important all the time, for data professionals. Some of the advice to look towards the quick moving, limited lifetime of many software components today doesn’t make sense. A database, at least some parts of a database, don’t fit within that model. There are valid reasons why a datastore must provide some stability and persistence for an application and won’t fit within a model of multiple systems. Or at least not practically within a model.

    However, there is good advice in the piece to learn more about different types of database services and patterns, and consider other ways of implementing datastores other than a RDBMS. I wouldn’t necessarily abandon a RDBMS for some NoSQL store just because developers think it’s easier, but I would consider whether I actually need to very tight coupling between different parts of the database and complete consistency. I think there are a fair number of domains where a CQRS pattern or some distributed store would work well. The move to microservices might be an enabler for your business if you consider the advantages for OLTP type transactions.

    The flip side of all the power that many stores provide is that many also make reporting and aggregating information more difficult. I would venture that in many cases, a data warehouse (perhaps even a RDBMS-based or columnar store) is necessary, along with the ETL process necessary to keep it up to date. These aren’t simple processes, and take resources to build. For most of us, because we don’t work at extreme scales, I’m not sure it’s worth leaving a relational platform, but I do think that we can learn to evolve and enhance our relational databases faster with DevOps ideas and techniques.

    Steve Jones

     

  • A Bad First Day

    I’ve started quite a few jobs throughout my career. In college, I worked in a variety of restaurants, changing employers at the start of every summer and again at the start of the school year. At some of those positions, I had a hectic beginning, with the staff shorthanded days and no one to train me, but they needed me to jump in. More than a few bartender jobs started like that, with me jumping in, making drinks, and teaching myself where things were stored. Luckily there weren’t a lot of mistakes I could make that might severely damage the business.

    Fortunately the frequency of new jobs has declined over time, though if I changed positions at this point, I’d go through much of the confusion that many new starters have. Where are things, how do I get work done, what credentials and procedures do I use, and more would be the questions on my mind for a few days. Perhaps a DevOps shop might smooth the process out and prevent issues, but I could see how this type of mistake could be made: a new intern used a document to setup a database, and ended up crashing the production database.

    There are lots of comments on that post, but essentially a setup document contained production values (url, user, pwd) and a junior developer used those values instead of the output from their local laptop. I common mistake. A document should have no ambiguity, and certainly should never have production passwords inside of it. Individual credentials should always be used to connect to sensitive systems, with some auditing of who connected (and hopefully what they did).

    I’ve had developers crash databases before and delete data. If they’re databases I manage, there’s a backup. I’ve never been in the situation where a database I was managing didn’t have some backup. However, I’ve had clients and coworkers managing databases with no backups. Perhaps the situation I felt lots of empathy for was a client that had developers making backups before they deployed updates to a client database.  One day an update didn’t work correctly, and it asn’t until a day later they realized. At that point, they couldn’t restore the backup. They called me, only to have me inform them that they were actually making striped backups. Unfortunately, they had deleted one of the stripes, and we couldn’t find it. Data loss, that fortunately didn’t kill the company, but it did cost quite a bit.

    There are any number of ways that a person can make a mistake on their first day. To prevent this, a good process should limit the damage that an individual can cause because of ignorance. This might be especially important for anyone that might have access to production data.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.1MB) podcast or subscribe to the feed at iTunes and Libsyn.