Voice of the DBA

Tag: high availability

  • Zero Downtime for 2016

    I’ve never felt that zero downtime is possible for a system over any length of time. There are no shortage of companies that seek to prove me wrong, and some are doing very well. For example, when is the last time that the Google search engine was down? It happens, though I haven’t seen this in a long time. Many of the highly available applications out there are built using as distributed applications across many machines, so that even if there are failures, they don’t cascade to interrupt users.

    I was thinking about this as I saw a post asking database engineers to architect for zero downtime in 2016. That’s a good goal, and certainly if you work at a high profile retailer or service company, you should look for ways to improve availability.

    In fact, I would guess that anyone struggling with specific events, like Black Friday, would be working on this problem constantly. I remember years ago Michelle Ufford spoke about the challenges at GoDaddy during the Super Bowl due to the advertisements the company ran. They had log files that couldn’t catch up to the load for hours and spent an entire year working to build a better database system.

    Ultimately I think the best way to handle large spikes of activity is by building an application that avoids putting large spikes of activity on your database. Use messaging and queues to buffer traffic. Use read only copies of your database for traffic that doesn’t need to write to the main database. Anywhere that you can limit the load on your system can help prevent the database becoming a bottleneck for your system.

    Perhaps more importantly, if you can spread the load, you don’t need to purchase more and more hardware. If nothing else, I think this is a good argument for better database architectures for applications. However if you’re like me, most of your systems will have to deal with the hardware that is available. In that case, the best you can do is write better T-SQL and ensure you have given SQL Server enough, but not too many, indexes.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • T-SQL Tuesday #60 – Something New Learned

    tsqltuesdayIt’s been five years, and that’s amazing. Not many things last for a year, much less five, but the T-SQL Tuesday party, started by Adam Machanic (B|T)., has been amazing and lots of fun for me.

    This month, Chris Yates hosts and his theme is Something New Learned. It’s a great topic, especially given the aims of T-SQL Tuesday to spread knowledge out in the world and share it with others.

    AlwaysOn

    This is good timing for me as I took a one day class last week. At the PASS Summit, I spent Tuesday in Allan Hirt’s A to Z of Availability Groups, which tries to help you understand AlwaysOn and the Availability Group portion of the technology. It was a great experience, and Allan did a fantastic job of walking through an overview, and then details. BTW, I can see how this would be amazing, and while I haven’t been, I’m sure that Allan’s Mission Critical SQL Server classes are valuable ways to learn this stuff in a hands on environment in a way that should help you be productive quickly on your own systems.

    The most interesting part for me is that this wasn’t an all day class where I listen to lectures, follow along in a workbook, and then move on. There were actual labs, and not labs that meant I downloaded scripts onto my machine and worked on them. Actually 3 node labs, dedicated to me, with instructions on how things were configured.

    What did I learn? Quite a bit, but for me the big mysteries that I’ve struggled with on AlwaysOn setups have been some of the permissions. Going through the labs, and getting the permissions necessary in the AD domain. The few places I had issues in the lab exercises were almost all related to a permissions issue I missed or had set incorrectly.

    I also went through the advanced versions of the labs, specifically to practice using Powershell for some config items. This was the chance to practice some skills and try to learn a bit more about how I can use PoSh for real world tasks. While the GUI might work well, I know that if I wanted to ensure I could build and create a lab in short order, or on demand, I’d really need PoSh scripting to ensure it was done correctly, and repeatedly. The lab reinforced that.

    I also learned a bit about a better way to teach. I’ve been in a few classes and lots of sessions across the last few years, but this dedicated lab environment really made things much easier for me. The hands on work was valuable in actually working through the concepts. In fact, I’ll be going through it again today as I have access to the labs for 10 days, and the workbook, so I can set up another Availability Group today and see the things I’ve done wrong in my own lab setup.

  • Multiple AGs

    Today was my pre-con with Allan Hirt on AlwaysOn. It was great, and I learned a few things. I knew a bit about AGs, but Allan has a ton of experience, and he talked about some architectural things that I didn’t understand.

    We had virtual labs, which worked flawlessly. I managed to configure AGs with GUIs, PowerShell, and a bit of T-SQL. I configured multiple AGs in the lab, and finished early, though I had one PoSh error that stumped me a bit. I’m looking forward to resetting the lab and trying again on Friday, taking a few hours to try and rebuild AGs in my own way without help.

    Though I suspect I may be sending an email to Allan before this is over.

  • Downtime

    Most of us that work with technology hate downtime. We don’t want a system that we’re using to go down. We don’t want any software that we depend on to fail when we need it. Most of all, we don’t want our phones ringing because some system we’re responsible for has gone down. We do everything we can to keep our applications online. We avoid patches. We try to test as much as possible before deploying changes. We also may apply generous amounts of hope and prayer.

    However that’s not how all companies run their internal systems. Netflix has taken the opposite approach, actually creating downtime for some of their systems using what they call a “chaos monkey,” and they think it could help you. To be fair, Netflix doesn’t take their entire application offline, but they do cause failures in the hardware and software, specifically to see if their redundant and scaled-out architectures can limit the impact on users.

    It’s an interesting idea, though one that I’ve not seen many companies be willing to implement. Netflix thinks you could benefit from it, but they also run a series of services that are scaled our across many machines. Many companies I’ve worked with have services on one machine handling an application, and they accept the risk that a system might fail and users will experience problems. Given the quality of modern hardware, that might be a good bet to place these days.

    However more and more of us are running redundant systems for some applications. If you think the Chaos Monkey could help you, let us know.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.