Voice of the DBA

Tag: high availability

  • What’s the Cost of an Hour?

    In 2017, we had a number of high profile downtime outages from companies. The British Airways fiasco, United airlines being grounded, Starbucks, Amazon, and more. There was a survey that noted 98% of respondents said an hour of downtime cost their organization over US$100,000. A third put that figure over $1mm.

    This week, I’m wondering if you have any idea what downtime costs your company. If you do, and can let us know, drop a figure in the comments. Maybe you can give us a range, or maybe you can say it’s a large amount. If you don’t know, maybe let us know if it’s not important to your business.

    I haven’t often calculated this, but when most companies look at lost revenue from customers, lost ability to conduct business, and potential losses from employees sitting around, it’s probably a large number. Then again, many companies might just force employees to work extra to make up the work, so perhaps some companies don’t care.

    This certainly isn’t a simple answer. Modern systems are often distributed, with many moving parts, and complex network connections. With modern software development using caching, micro-services, and feature flags, it’s entirely that an application is down for some clients and not for others. Or perhaps, a portion of the application is broken while other parts work.

    Downtime can be disruptive and expensive, and it is something most system administrators plan for and work to avoid. If you know what it cost for either the loss of a system or in your preparations, it would be interesting to know today.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • MTTD

    There are a lot of Mean-Time-To-xxxx acronyms. Many of us have heard of the mean time between failures (MTBF) for disk drives. Some of us use that information when considering which model to buy. In the DevOps world, there are also the mean time to failure (MTTF) and mean time to resolve/repair (MTTR). There is one more that I think is very interesting, and that is the MTTD: the mean time to detect an issue. This is the average amount of time it takes you to detect there is a problem after the problem occurs.

    There was an outage at Monzo recently due to a database upgrade, which was recounted on their blog. In this case, their MTTD, or rather actual time to detect, was a minute. I think that is amazing. In fact, I’m somewhat skeptical that an alert is raised, someone looks at it, the customer service desk calls the Ops team (who were upgrading servers), and the Ops person realizes in the space of a minute or two that there is an issue. It’s possible, but I have found that help desk personnel that discover something can take a few minutes to verify the issue and then scramble to find the on-call phone number. Relaying information can take a minute or two, so if this is accurate, huge props to the IT staff at Monzo.

    Many of us strive to high a high availability number for our systems, especially databases. This is one of the drivers for the growing use of availability groups in SQL Server systems: to ensure the database is highly available to clients. In determining availability, we often speak of the percentage of time that a system is available. The holy grail is five 9s, or an uptime of 99.999% of the year. This gives you just over 5 minutes of downtime a year.

    In the case of the Monzo outage, which took place in July 2019, the alert is reported at 13:14 and the incident was declared at 13:15pm, one minute later. The time to diagnose the issue (maybe another MTTxx item) was 63 minutes, just over an hour. At this point, availability is arguably down to 99.988%. The actual fix was completed at 113 minutes, or 99.978%. That’s the number if nothing else happens this year.

    If you’re attempting to get to 5 9s of reliability, you get less than 6 minutes of downtime a year. Can you figure out what’s wrong in 6 minutes? Much less fix it? That’s a difficult task. I think 4 9s, giving you 52-ish minutes of downtime, is realistic, but very hard. Most of us can likely handle 3 9s, which allows for 8:30:00 of downtime a year. While I’ve exceeded that before, it’s been rare.

    We have a lot of HA (high availability) options in SQL Server, and there are many successful implementations that achieve high levels of availability for the database. The network and the application are another story, but I think the quality of those areas has increased over the years as well. Doing HA well is hard, and if you aren’t 100% sure of what you’re doing, or your system is very valuable, you might engage a consultant, like Allan Hirt, to ensure that you’ve configured things well. SQL Server runs well in HA configurations, but getting it set up can be more difficult than you expect.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • GitHub Downtime

    I didn’t notice any issues with GitHub, but others did. The majority of my interaction is just through the git protocol, so things tend to work fast, and I don’t have any database access. I rarely use the Issues, and other parts of GitHub, which were affected when GitHub had a MySQL cluster fail over. There’s a good write up of the post incident analysis that’s worth reading, from a database perspective.

    I’m not a big MySQL guy, only running an instance to power T-SQL Tuesday. The structure of a write primary and many read replicas that GitHub describes makes sense. It’s similar to what I’ve done in SQL Server, and certainly the idea of some quorum management, handled at GitHub with the Orchestrator software, is something that needs to be configured properly. Allan Hirt has talked about the complexities of quorum in large installations, and it’s not a simple thing to configure.

    In reading about this, there are a couple things that strike me. First, the analysis talks about a degredation of service because East coast applications had to send writes to West Coast database servers. There were some problems with the way the database servers were working, but it seems to me that there should be some sort of application failover that’s possible. If you can’t have an application and database fail separately without customer impact, then there should be some way to fail applications over. Perhaps not, but if you’re responsible for designing HA for the database, make sure you talk to the application people and test for issues.

    The second thing for me is that somehow there was a period of time when writes were occurring to the East Coast system that weren’t sent to the West Coast. My ignorance of how this HA stuff works in MySQL prevents me from making a big deal of this, but this isn’t something that should happen. If the quorum moves data to another node, it must stop writes to the first node. This could happen in SQL Server, but for me, this is the level of data loss I’d need to accept in my RPO.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.4MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Republish: The Cost of Switching

    Still on vacation. Enjoy The Cost of Switching