Voice of the DBA

Category: Editorial

  • Someone Lost My Data

    It was bound to happen. I’m sure it’s happened before, but this event was interesting to me as it wasn’t a customer issue, but a cloud vendor problem. There was an outage in Azure on Jan 29, which happens, but in this case data was lost. There was problem internal code in the Azure cloud that dropped some customer databases using the Azure Key Vault with TDE encryption. That was slightly disconcerting for me as I was setting up and testing Azure Key Vault this week.

    There are frequent snapshots and Microsoft was able to restore the databases from one that was about five minutes old. Microsoft acknowledges that five minutes of data loss might be an issue and is asking customers that lost business or were affected by the drops to raise a support ticket. I have found Azure support to be pretty good about crediting my account when issues occur, and I hope they do the same here, though I wonder if they’ll compensate anything beyond the charges normally assessed to customers. They are offering credit for the restored and original databases for a few months as well.

    Plenty of people are upset, and with good reason. There should not be Azure management code that drops databases. Or should there be? Would this be any different on premises?

    I’ve had cleanup code that removed resources after some time. I don’t work at the scale of Azure, so I usually have things removed after a month or quarter. With the scale of Azure and potential costs, they might remove resources quicker, but I certainly have seen similar home grown, if-this-then-that code that does x when y occurs. In this case, I remove encryption keys, which might be not actually be removed for month or more. At that point, there is code that trigger a drop of databases. I’ve certainly seen users in various organizations drop, or restore over, the wrong database. In some cases they don’t realize it in five minutes, and often don’t have a way to restore from a five minute ago backup. Actually, relatively few people I know have RPOs under five minutes.

    This is bad, but it isn’t necessarily out of the ordinary for complex IT environments. If this happened in an organization, the IT staff would be worried and hoping for forgiveness. Some people would want others fired, but most of the time management would understand these things happen. Perhaps not if this isn’t the first time, but usually we accept some people make mistakes. Most of us don’t have complete control of all aspects of our environment. We depend on network staff, storage people, employees that manage hardware, and more. It’s possible that any one of these people could destroy data inadvertently.

    That happened here, though I don’t want to make excuses for Microsoft. They’re supposed to hire the best people and build processes that are better than what I’d expect inside an organization. Events will cascade into different areas, and there should be circuit breakers that prevent anything that could cause data loss in those events. Protect other people’s data with more care than you think you need. It’s your responsibility to do so.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Google’s Expensive Confusing Rules

    I worked through the Y2K crisis. I call it a crisis because so many of us were worried and millions, probably hundreds of millions of dollars were spent by companies trying to ensure their software systems would work when the date changed to 2000/01/01 00:00:00. I was on call that night, celebrating the new year at home, not drinking because I was slightly worried my paranoid boss would call me in. He didn’t and there were few issues around the world in systems, perhaps because of the build up of the crisis and lots of prep work. I went through a similar set of concerns when the Sarbanes-Oxley act was passed in the US. Not much came of it, and companies spent a lot of time and money preparing.

    Over the last two years, it has been deja vu as the GDPR moved towards the enforcement date last May. Lots of companies, including my own employer (Redgate Software) were concerned and spent time and resources getting ready. The enforcement date came, with some early complaints being filed, but few fines. Perhaps the preparation paid off, but more likely it just takes time for audits to occur and complaints to be investigated.

    It appears that the first big fine has been handed down to Google. Recently the CNIL (the French data protection regulator) fined Google 50 million euros for not compliance with the GDPR. Their reasoning was that Google didn’t provide enough information about their data consent policies and didn’t give users enough control. They complained that the data is spread out in many different places and too difficult to understand. Other countries are investigating, and Google is going to appeal.

    Personally the presentation and dissemination of information from Google should be top notch. That’s their job: to search, assemble, and present data. As someone that has paid for Google Apps and email, has used Google Analytics, and generally tried to understand some of the other products, I think too many engineers and not enough technical writers work at Google. I think they are surprisingly bad at making it easy to understand how to accomplish some task, including finding out some information about my account or my data. I’m not surprised that they were fined, since I think they have had an opt-out philosophy and the many different groups inside the company have considered the data gathered to belong to Google, not the human about which it refers.

    I don’t know how this will play out, but as a person, I do hope that companies will be asked to clearly disclose what data they have, how they use it, and to delete it when the business purpose is complete. I’d prefer that I had more control and understanding of my data, including the crazy cookies and other mechanisms that track my browsing across sites. While some companies use this to provide customization, there is plenty of potential for misuse here, and I’d like it to stop.

    As a data professional, I’d like to have clear understanding of how to treat and protect data. I’d prefer that we better secured it, didn’t use live data in development environments, and we built better habits as technology professionals. Data is truly and asset, and one that can easily help businesses grow, but it also has the power to be abused. I’d like that to stop with sensible rules that I can work within.

    Let me know how you feel today.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Hard Drive Reliability

    Do any of you still use hard drives? I’m sure some of you do, and I’m also sure plenty of you aren’t sure if you’re using HDDs or SSDs. For some of us, the hardware is just a set of resources with a certain quantity, capacity and performance. While we used to spec, purchase, and assemble database servers in the past, these days may of us just specify cores, RAM, and IOPs or some storage capacity, not worrying about how our resources are actually implemented. I know that I assume SQLServerCentral is using SSDs in our hosted systems, but I honestly don’t know, much less care about the type of drives in use.

    I saw Backblaze release their 2018 hard drive stats recently, looking at the reliability of their storage. Since BackBlaze is in the business of storing data, they need to ensure that they have quality storage. I’m a customer, and I appreciate that they’re not just buying and using devices, but also tracking how well the different brands and models work. Since they have over 100,000 drives, this means they can provide some interesting information on reliability.

    As our storage needs grow, many of us won’t use SSDs for all our data. Even all the data in our database servers might not be SSD as the cost is high. I’m sure HDDs are well suited for backups, and potentially even still useful in many servers for data storage when workloads might not be as high. Even at home, I like to use SSDs as much as possible, but for some types of storage, especially pictures, audio, and video, HDDs are much more cost effective with their higher density.

    The Backblaze stats are interesting in a few ways. Certainly it’s amazing to see how larger and smaller drives perform. It seems that most drives have a very low failure rate, though there is an order of magnitude difference between different models of similar sizes. It doesn’t seem that the 10-14TB drives are worse than the 4TB drives, which makes me think that vendors are doing a great job with quality control in manufacturing. I used to worry about larger sizes being released and having lower reliability, but this doesn’t seem to be the case.

    Some drives have very low failures, but these are averages, so I’m not sure they’re necessarily better, but worth keeping an eye on. I also think that since many of these drives fail at < 1% annualized rate, they’re good choices for your data center. Certainly the HGST ones seem to have very low failure rates. I know I may pick up a couple of these for more photo storage as I’m running out. It seems that I never have enough space for all the pictures I take.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • DR Prep Can Miss the Little Things

    The other day we had a blizzard in Colorado and we weren’t quite prepared. My wife and I were away on a trip and watching the weather. We were concerned about flights and about the conditions at the ranch with someone else in charge of horses. We were lucky in that the weather didn’t come as quickly as predicted and we made it home before the snow started.

    Depending on moisture and temperature, we sometimes put blankets on horses, which is a chore that I’m not very good at completing. Fortunately the kids were around and at 8pm, we all dressed in warm clothes and headed out to the barn. The horses depend on us to provide for some of their needs, so we each had jobs to do. One kid secured a temporary holding area to keep horses nearby. One gathered 14 or 15 blankets from storage and laid them out. My wife made grain to facilitate attracting the horses and giving us a way to hold them while blankets went on.

    Me? I had the envious task of double checking electricity and tank water headers. Our employee had thought that one was shorting out and at night, with a flashlight and plug tester, i got to debug 4 water heaters, including one that’s semi-remote from the barn. Testing for shorts involves touching water to see if there is any flowing power (this is very low amperage), which isn’t terribly dangerous, but is a little daunting. We managed to get everything done, and felt like we were prepped for a 4-10″ snowfall, 50mph winds, and a 0-5F temperatures.

    Later I lay in bed, trying to relax with a little TV before calling it a night. While the wind was howling, I heard a pop and the power went out. We have a generator, and I wasn’t too worried, but I did want to ensure it came on. I walked downstairs and across the house to listen for the engine noise. I heard cranking, but the engine didn’t catch. It was then I remembered the tank was low the last time I checked, and with a weekly diagnostic auto run for 5 minutes, I was likely out of propane. Since we needed power to heat water for horses, I had to get up around 11pm and take care of things.

    Fortunately I keep a spare bottle of propane and I went outside in the blowing snow to change it. A true work-at-home-techie, I did this without pants and got things running. Back to bed, though worried a bit about how to get more propane in the morning, just in case of an extended outage. The power came back on in the morning, but I still needed to get more propane just in case.

    Extended disasters sometimes cause problems with our plans because many of us focus on the immediate reactions. Longer term, things like additional fuel, food for humans, even shelter and child care are issues that I’ve had to deal with in teams that no one had planned for. We’ve seen this over and over again in the world, especially when supply chains break down quickly in disaster situations.

    I could have prevented some of the issues and stress if I’d prepped things better. I should have filled propane tanks earlier, knowing that bad weather can come at any time, and likely will at some time. Double checking heaters and power earlier might have made the blanketing process go quicker. There are always things we miss in prep, often ones we don’t consider to be important, but may be difficult to deal with in the moment. Take a few minutes and think about the little things you might dismiss as not important. Imagine how hard some of them might be to deal with under the pressure and stress of a DR event. If you can do a little more prep, some maintenance, or get some work done early, now might be the time to do it.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.8MB) podcast or subscribe to the feed at iTunes and Libsyn.