Voice of the DBA

Tag: disaster recovery

  • A Staffing Disaster

    There was a failure recently at an Azure data center in Australia when a utility power sag caused equipment to trip offline at one of the Azure data centers in Australia. You can read about it here, but essentially the headline is that there were only three people on site when the incident occurred, and that caused them to be unable to restart the equipment in time before an outage occurred.

    In a little more detail, there weren’t enough people to quickly restart the equipment chillers after the incident. The staff had to access the equipment on a roof when 13 of the units didn’t restart. They were able to get to 8, but when they got to the last 5, the temperature of the water had risen to a level that wouldn’t allow a restart. So they had to power down some computer equipment and go through a more lengthy process to get everything running.

    This sounds bad, but in reality, this is exactly the type of thing I’ve seen in private data centers, who almost never have all the staff they need, or the knowledge necessary, to deal with large-scale failures. While I haven’t seen the chillers, I have seen people trip electrical systems and be unable to restart or reset UPS’s or generators for hours until qualified staff could come in. If you read the incident history, there is a good retroactive of what happened, and then some actions taken to try and prevent this in the future. They increased staff levels but also identified some places where the previous staffing level would have been fine with some equipment and protocol updates.

    I wish more organizations would review incidents and examine them with an eye towards not only what happened and where there were failures, but how to prevent issues in the future. Too often I see people going through this exercise in order to blame someone and “prevent this from ever happening again”, which usually means we fire someone and don’t change anything else. We need psychological safety in reviews of actions to get better.

    As we build more complex systems, or even more complex organizations with lots of teams, people, equipment, procedures, etc., it’s easy to build in lots of points of failure without realizing there will be problems in the future. My goal often these days is to assume I’ll have some inexperienced or less capable staff and design processes and systems to survive issues. To keep things simple, and not get too cute with engineering. I like robust, resilient systems that anyone can operate, not those that require me to ensure my senior superstars are always on call.

    Of course, it often takes the senior superstars to design and test these systems and protocols, which is a good use of their time.

    Many businesses struggle with staffing, in many areas. Technology groups are no different, and we have to learn to work smarter, not assume we will just get more staff and solve our problems.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • How Many Days Can You Survive?

    I saw a great post from DCAC on disaster plans and using them after a fire in an LA data center. DCAC wasn’t affected, and I wouldn’t expect them to be. Denny, Joey, Monica, John, Kerry, and Meagan are all experts in running systems efficiently and effectively. They think about how to ensure that things keep working in the event of disasters and have delivered presentations all over the world helping you become better at managing your own servers.

    However, Denny brings up a great question in the post. How many days could you survive without IT systems? Days? Weeks? Have you ever been in that situation? I’m sure many of us have experienced a failure of some sort. An application crash, hardware dying, or most commonly, Internet access being cut. All of these are small disasters, which typically are fixed quickly. Hopefully, these aren’t issues you experience every week. If you do, maybe you need to call DCAC or someone else and fix more fundamental issues.

    While it’s unlikely that you lose all your systems, it could happen if you concentrate your resources in one data center, one region, one cloud provider, etc. Having a single point of failure is something we try to avoid in IT, and that is true not just for one application, but for your infrastructure design. Most of us depend on one authentication system, and a failure of our Active Directory could lock everyone out of a system. Those are rare, and hopefully, your administrators have enough redundancy (and backups) to recover from this type of disaster.

    I have experienced a few large failures at one enterprise. We had a few viruses, including the SQL Slammer worm in the early 2000s. Our network was shut down for a couple of days, when almost all systems from email to CRM to ticketing systems weren’t available. Everyone had to use whatever paper systems they could to keep business running. While we likely lost some revenue from these outages, we learned we could survive a few days without our network. We also learned that we needed better virus scanning and education for employees, as well as a few more resources for tech people. Before those events, everyone assumed an outage was bad, but had no idea how bad. I have no idea how much this cost us, but it didn’t appear in a 10-Q, so it must not have been too bad.

    I think there are lots of businesses that could find ways to continue to work if some systems were down. However, there can be costs, sometimes significant. Even if the company doesn’t go out of business, perhaps some people get terminated because of less revenue. That might not be the tech person in the short term, but how would you feel if your DR plan didn’t work (or you didn’t have one) and some co-workers were let go?

    The move to the cloud, and the move to more software-as-a-service systems, might help you better survive local disasters, but if you have too many systems concentrated in one place, it is worth preparing for some contingency. After all, even if this fire were to happen in an Azure or AWS data center, it’s possible that their process to move and restore all the systems from one data center to another could take time. Your systems might not even be their top priority as cloud vendors have some large customers. It probably won’t take months, but I wouldn’t want to bet my job on any cloud vendor getting everything moved in less than a week.

    If you’re not in the cloud, make sure you have a plan. If you don’t know how to do that, call DCAC or another consultant to help you.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • How Often Do You Update During Issues?

    I use Libsyn to store podcast files. They’ve been a reliable service for me for years. Once in awhile uploads are slow, but things seem to work. The other day I went to upload a file and got a status message that there was an issue with database maintenance. When I looked at the status page, I saw a few updates. This was for a platform issue, and I saw this set of updates. I have the timing that I saw on the page with my own comments added:

    • 1139 – reports of issues. We’re looking into this
    • Three hours ago – Identified the issue related to db maintenance, working on it, other services affected.
    • Three hours ago – all services affected, working, update soon
    • Three hours ago – emergency maintenance on db systems, we will provide regular updates
    • One hour ago – db maintenance in progress, now healthy nodes, turning things on.

    That was what I saw at around 3:30 my time. I went back the next day and saw a more detailed set of times listed and a note that the cluster was fixed and then all services were restored. While I couldn’t upload things that day, I did check that downloads for listeners were working, and they were at that time.

    I have no idea what happened, and I did appreciate an email the next day that apologized and noted this outage was not the result of malicious attacks and that no data breach had taken place. The latter item hadn’t occurred to me, but I thought that was a good reassurance sentence in the email. I’m sure it was a rough day for DBAs and the Ops staff, and hopefully, they were able to restore all data.

    My concern, however, was that multiple times they noted they would post updates soon, but there were some pretty good gaps in the status messages. While I liked a few quick messages together (3 in 30 minutes), the long gaps are disconcerting to me as a customer. I expect management would feel the same way and hopefully, management was updated more often.

    If you’ve been in an outage, sometimes there isn’t a change in status. A long restore or rebuild of some sort can take time, with platforms not always reporting progress or an estimate of time remaining. Even when you get some progress, we all know that the time to go from 25% to 50% could be shorter than the time to go from 90% to 95%.

    When I have had to report to management, or to an incident team, usually we have regular updates. Even if these are “no change, we’re still working,” it’s good to let others know what you know. I think that’s important for customers as well, especially those that might have time-sensitive expectations for using your application. Without an update, anyone checking a status might not know if anything has changed, if things are worse, or maybe that you forgot to post an update.

    My recommendation is that there is someone dedicated to logging what is happening and taking notes for later review. This person is also someone that ought to be responsible for updating others on a regular basis. Every hour, every two hours, something regular. If you have external customers, then they should expect and get regular updates, even if these are “no change, the cluster is still rebuilding.”

    A little transparency goes a long way for your customers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Flexibility in DR

    Disaster recovery is a topic that many people who work with databases find important. DBAs work to ensure their systems are highly available while practicing the skills needed to recover them in the event of a disaster. While we often practice the failure of a complete server or database, we sometimes forget there can be other issues for which we need to plan.

    I ran across a story about a data center outage in London, which was similar to an incident I experienced. In this case, a piece of power equipment failed and systems lost network access. While many of us aren’t responsible for the network, we certainly would still receive complaints if the database wasn’t available.

    In my situation, I was in a data center watching some workman service UPS systems. At the time I was a manager of database systems and thought they had things under control. I left the data center and went downstairs, only to audibly notice when the power dropped that all servers seemed to stop running. Similar to the London outage, a switch that was supposed to change power from one set of UPSes to the other failed. Our entire global infrastructure for 10,000+ people and who knows how many customers went down. This was in the pre-cloud era where we acted as our own cloud. And not very well.

    There are systems outside of our databases that we depend on. We might not be responsible for them, but we ought to ensure they are a part of our disaster recovery plans and account for various things going wrong. At the very least, we ought to question whether power, network, storage, and more are adequately prepared for major issues.

    We also need to think about minor issues. Atlassian had a major outage, at least for some customers, and they realized that they hadn’t planned to recover parts of their databases. A similar issue might occur for any of us, where we might have to restore parts of a database, whether that’s a table, partition, or a series of rows. Corruption or human error might result in a set of data that’s unreadable or even gone. I know I’ve accidentally caused data issues, and I’m careful. I learned to recover from my own mistakes and anticipate those of others. I practice not only full restores but how to copy over part of table from another location.

    A disaster is a major problem, but it might only affect a minor part of our systems. We need to ensure we are ready for any size or scale of problem and be ready to adapt our thinking and process to meet the disaster with the appropriate actions.

    Steve Jones