Voice of the DBA

Category: Editorial

  • You Need Offline Backups

    If you hadn’t heard about it, VFEMail may be dead. At least, that’s what the founder was thinking in this article. A malicious hacking incident took place last week, and though they’re back up and running, who knows if customers will stick by them, or maybe sue them out of existence. I wouldn’t be surprised as a large number of their infrastructure servers were wiped out by reformatting servers. These included mail servers, backup servers, and SQL Servers.

    That’s quite an attack, and whether this was directed at the company or some individuals, a large number of people might have lost their mailboxes and previously stored mail that was in IMAP storage. This is the type of issue that is most likely an annoyance for individuals, but it would be potentially catastrophic for businesses. Imagine your small business hosted with them and all your mailboxes were lost with customer communications and who knows what else. Perhaps you could recover data or keep the business going, but it might be an issue.

    Could this happen with a cloud provider like Azure O365, Google Apps or AWS? Possibly, and while I’m sure they have backups, I’m not sure how reliable those might be for the average individual or small business. This makes me worry slightly as I depend on GMail and wouldn’t even try to backup to few 100GBs of mail I have. I’m not even sure how to do it, though I don’t really keep anything in there that’s really important. In any case, I’d suspect that connecting and somehow wiping out Gmail servers, along with backups, would be very difficult.

    This does make me think about a few customers I know that use online storage for backups. They assume that they will always have either a primary server or the online backup server/share/bucket/container and can download data. The problem is that online systems that connect to the primary can be accessed. If an attacker were to access one, they potentially could access the second.

    The world seems to be moving towards more online storage, or in the case of cloud vendors, a reliance on snapshots. That might be good enough for cloud vendors, but it’s certainly not for any on-premise system. It’s likely that an attacker, possibly with insider help, would wipe out backups first, then primary systems. I’d always want some sort of disconnected offline backup of data, especially database servers. I have seen Murphy’s Law strike two systems at once, so an air gap between copies of data just feels prudent.

    Steve Jones

  • A Supercomputer in My Pocket

    I work with data on a regular basis, and I really depend on my cell phone to help me with both work and life. I regularly make notes and get ideas for articles and editorials from things that happen when I’m on the go. Without a smartphone, I’d be juggling a notebook and pen, perhaps pulling over and making notes that I’d transcribe later. Since I’m in different cars and vehicles with different bags, I’m not sure how well I’d be able to keep track of notes on paper.

    In the last decade, as I’ve purchased and upgraded mobile devices, it’s been amazing to me that I can use many services to help me. I can log into something like Instapaper to save an interesting article I might write about later, or capture a few thoughts in Evernote. Or even send myself an email with an idea for a Question of the Day. I’ve done all of those things while in various points when inspiration has struck.

    In the last few years, I’ve started to use some audio notes for tracking when I’m driving. I might hear something on the radio, or from a podcast, and need to make a note. The power of dictation apps, which have improved tremendously since the early Dragon Speech Recognition days on 486 computers. While I still don’t completely trust audio recognition, what I’ve learned to appreciate is the ability to just record sound with an audio app that I can play back later. There are audio recording apps, though I often just use Evernote.

    I use a lot of data transfer and storage on my mobile device. So much so that when I switched a couple years ago to the Google Fi, I was disappointed with only having 32GB of storage. It was amazing for a kid that grew up with 300kb floppy disks to think that 32GB wasn’t enough, but it wasn’t. I was constantly juggling space and deleting things. I recently upgraded to a phone with 192GB, and I’m hoping that will satisfy my need for pictures, video, and notes.

    It’s amazing to see just how far computing has expanded, giving us incredible capabilities on the go that were science fiction a quarter century ago. Many appreciate the ability to review a document or spreadsheet, or even view a Power BI Report from any device. While I don’t often have the need, it is nice to know that I can catch SQL Monitor alerts and view data on my phone, or even restart a VM from a cloud shell from my car if I have the need.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.6MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Someone Lost My Data

    It was bound to happen. I’m sure it’s happened before, but this event was interesting to me as it wasn’t a customer issue, but a cloud vendor problem. There was an outage in Azure on Jan 29, which happens, but in this case data was lost. There was problem internal code in the Azure cloud that dropped some customer databases using the Azure Key Vault with TDE encryption. That was slightly disconcerting for me as I was setting up and testing Azure Key Vault this week.

    There are frequent snapshots and Microsoft was able to restore the databases from one that was about five minutes old. Microsoft acknowledges that five minutes of data loss might be an issue and is asking customers that lost business or were affected by the drops to raise a support ticket. I have found Azure support to be pretty good about crediting my account when issues occur, and I hope they do the same here, though I wonder if they’ll compensate anything beyond the charges normally assessed to customers. They are offering credit for the restored and original databases for a few months as well.

    Plenty of people are upset, and with good reason. There should not be Azure management code that drops databases. Or should there be? Would this be any different on premises?

    I’ve had cleanup code that removed resources after some time. I don’t work at the scale of Azure, so I usually have things removed after a month or quarter. With the scale of Azure and potential costs, they might remove resources quicker, but I certainly have seen similar home grown, if-this-then-that code that does x when y occurs. In this case, I remove encryption keys, which might be not actually be removed for month or more. At that point, there is code that trigger a drop of databases. I’ve certainly seen users in various organizations drop, or restore over, the wrong database. In some cases they don’t realize it in five minutes, and often don’t have a way to restore from a five minute ago backup. Actually, relatively few people I know have RPOs under five minutes.

    This is bad, but it isn’t necessarily out of the ordinary for complex IT environments. If this happened in an organization, the IT staff would be worried and hoping for forgiveness. Some people would want others fired, but most of the time management would understand these things happen. Perhaps not if this isn’t the first time, but usually we accept some people make mistakes. Most of us don’t have complete control of all aspects of our environment. We depend on network staff, storage people, employees that manage hardware, and more. It’s possible that any one of these people could destroy data inadvertently.

    That happened here, though I don’t want to make excuses for Microsoft. They’re supposed to hire the best people and build processes that are better than what I’d expect inside an organization. Events will cascade into different areas, and there should be circuit breakers that prevent anything that could cause data loss in those events. Protect other people’s data with more care than you think you need. It’s your responsibility to do so.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.7MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Google’s Expensive Confusing Rules

    I worked through the Y2K crisis. I call it a crisis because so many of us were worried and millions, probably hundreds of millions of dollars were spent by companies trying to ensure their software systems would work when the date changed to 2000/01/01 00:00:00. I was on call that night, celebrating the new year at home, not drinking because I was slightly worried my paranoid boss would call me in. He didn’t and there were few issues around the world in systems, perhaps because of the build up of the crisis and lots of prep work. I went through a similar set of concerns when the Sarbanes-Oxley act was passed in the US. Not much came of it, and companies spent a lot of time and money preparing.

    Over the last two years, it has been deja vu as the GDPR moved towards the enforcement date last May. Lots of companies, including my own employer (Redgate Software) were concerned and spent time and resources getting ready. The enforcement date came, with some early complaints being filed, but few fines. Perhaps the preparation paid off, but more likely it just takes time for audits to occur and complaints to be investigated.

    It appears that the first big fine has been handed down to Google. Recently the CNIL (the French data protection regulator) fined Google 50 million euros for not compliance with the GDPR. Their reasoning was that Google didn’t provide enough information about their data consent policies and didn’t give users enough control. They complained that the data is spread out in many different places and too difficult to understand. Other countries are investigating, and Google is going to appeal.

    Personally the presentation and dissemination of information from Google should be top notch. That’s their job: to search, assemble, and present data. As someone that has paid for Google Apps and email, has used Google Analytics, and generally tried to understand some of the other products, I think too many engineers and not enough technical writers work at Google. I think they are surprisingly bad at making it easy to understand how to accomplish some task, including finding out some information about my account or my data. I’m not surprised that they were fined, since I think they have had an opt-out philosophy and the many different groups inside the company have considered the data gathered to belong to Google, not the human about which it refers.

    I don’t know how this will play out, but as a person, I do hope that companies will be asked to clearly disclose what data they have, how they use it, and to delete it when the business purpose is complete. I’d prefer that I had more control and understanding of my data, including the crazy cookies and other mechanisms that track my browsing across sites. While some companies use this to provide customization, there is plenty of potential for misuse here, and I’d like it to stop.

    As a data professional, I’d like to have clear understanding of how to treat and protect data. I’d prefer that we better secured it, didn’t use live data in development environments, and we built better habits as technology professionals. Data is truly and asset, and one that can easily help businesses grow, but it also has the power to be abused. I’d like that to stop with sensible rules that I can work within.

    Let me know how you feel today.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.