Voice of the DBA

Category: Editorial

  • Backups Aren’t Backups Until a Restore Is Made

    One of the interesting things I saw in the recent GitLab outage and data loss was the fact that none of their backups were available. They use PostgreSQL and I’m not familiar with the ways in which the modern PostgreSQL engine handles backups or the options you have, so I’m not knocking either GitLab or PostgreSQL. It’s possible one or the other had fewer options than we do with SQL Server with our full, differential, log, and filegroup backups, all during live database activity.

    There was a live stream and a Google Doc open during the incident, showing the response by their employees (and plenty of Hacker News comments). Kudos to GitLab for their bravery and transparency in showcasing their mistakes and choices. I’ve been in similar situations, and the war room can be chaotic and stressful. There have been no shortage of times when someone makes a mistake under pressure and we scramble to recover from the damage. I’ve made those mistakes and understand how they happen when you get desperate and are tired. This is one reason I’ve usually insisted that when an incident is declared, I immediately send at least one person home to rest. I never know what time I’ll need to get them back.

    In reading the notes, there are a number of issues. One of the respondents doesn’t know where the once a day backups are stored (1). The location they check has files only a few bytes in size, so backups might not be working (2). No disk snapshots in their Azure space for database servers (3), though the NFS servers get them. The snapshot process is incomplete, in that once snapshots are made, some data is removed from production, and will be lost in this recovery (4). The backups to S3 don’t work (5). All of this results in a backup that is six hours old being restored. For people that commit code often, this could be a lot of data. Hopefully there weren’t too many merges and branch deletions in this time for customers.

    A backup doesn’t matter. A restore matters. It doesn’t matter what backup process you have, if you don’t test it, then you don’t know if you can recover. In fact, with databases (really any system), you need to test the restores regularly because the backup process can fail. I learned this early in my career when one of our admins realized his fancy tape changer that let him only change tapes once a week was broken. The drive had stopped writing and he never noticed.

    Not only is it important to monitor that the backup process runs, it’s important to ensure the backup files exist, where we expect them to exist. If this is a remote location, you need monitoring there as well. It’s also important to restore backups regularly. Ideally you’d test every one, but at least get a regular rotation of testing once a week to ensure your process is working.

    If you don’t, then you risk not only data loss, as GitLab experienced, but an RGE. That’s a resume generating event, and it’s something none of us would like to experience.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • A View of the Cloud

    Amazon has been an amazing digital company in the last twenty years. I remember making my first order from them, unsure of whether the online bookstore would work better than browsing locally, or if I’d even get my books. I watched them transform into a great shipping company that could sell anything, to a digital reading company with the Kindle, and even an amazing cloud hosting provider with world class programming platforms. At each stage, Amazon has grown to become even more efficient and seductive, slowly gaining more of my business over the years.

    Whether you like the business model and practices of Amazon or not, part of the reason Amazon has become a successful company is that they have an incredible software development process and great developers, both of which have produced a software stack that is very impressive. Apart from their web site, which is impressive, they host their Amazon Web Services, which is a dizzying array of services that can be purchased by anyone, at any time, and get some software up and running quickly. Even if you look at their free tier, it’s very impressive in the number and type of services, including database access.

    I ran across a piece on the 10 year anniversary of AWS that contains some software development lessons from Werner Vogels, CTO of Amazon. It’s a good list, and while some might not apply to your particular environment in the same way, it’s a good list of principles to follow. Certainly planning to evolve and to handle failures is something that most of us need to consider. Including security from the ground up, however, is one thing that many of us know, but don’t have good patterns, practices, or habits to follow. Even those companies that work in security conscious areas often leave security concerns until they are well into, or even finished, with their core software development.

    The one area that I think is going to become more important for all of us in the future is understanding resource usage. Certainly we’ve needed to know some baselines and expected growth in the past to plan for upgrades and future needs. However, with the move to VMs and cloud services, how many of us really know what resource usage we would require for peak workloads and when. I suspect most people don’t really know how to map their instance resource usage across different virtual systems, much less evaluate whether their system would be cost effective in a cloud scenario. A better understanding of what resources we really use is something that more and more organizations are going to expect from their technical professionals. I only hope we get some tools to help us here as this is a complex topic.

    The move to cloud services is going to happen for many of us. We might not move all our data, or even much of it, but as we experiment with new platforms, as companies work to avoid capital expenditures, and as the platforms and services mature, more of us will end up with some cloud platform in a portion of our organization. During your journey, or even while you still work with on-premise systems, I’d keep the principles in Mr. Vogels’ article in mind. They’ll help you to build better software in the future.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Your Favorite Feature that Needs Work

    The SQL Server platform has grown tremendously since I started working with SQL Server v4.2. We’ve had various subsystems added, lots of enhancements, and even new languages. There have also been a few parts of the platform that have been discontinued, such as Notification Services and English Query. There are items that are deprecated, though far fewer that have actually been discontinued from the product. I’m not sure if I think that’s better or worse for all of us.

    Apart from the deprecated, discontinued, and removed features, there are also lots of features in the platform that are stale. They receive very little development, and perhaps are very lightly featured on the What’s New pages. Or they never get any development from version to version.

    This week, I’m curious what features you use or like that need help. SQL Server v.Next is in development, and while I don’t think major changes will be made between now and RTM, perhaps there are improvements that could be made in some areas. I’m sure there are a few SQL Server developers at Microsoft that have time to ask for a few more pull requests from their particular area of development. With that in mind…

    What feature in SQL Server do you really like and wish would be improved in the future?

    Your idea would be large or small. Perhaps you have some language enhancement. Maybe you run setup regularly and there is a change that might streamline that process. It may be that replication conflict resolution is a constant problem for you and you’d like a few development hours spent on this. It could be anything that you use, or would like to use. Perhaps you have an idea for an improvement, or maybe you just want something done. Let us know today.

    The place to log bugs or feature requests is at the SQL Server Connect site. Post your idea there, then put a link in the discussion and make a case for why people should vote for it. Who knows? Perhaps you’ll get some improvement for the feature you like in the next version of SQL Server.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Great and Powerful Internet of Things

    Will we have the IoUT (Internet of Useful Things? As some have said, the IoS (Internet of Sh**e) is really what we have because of the poorly built, poorly secured hardware and software devices. It seems that almost every month I see new devices introduced in the consumer space, most of which aren’t well designed for widespread use, and certainly aren’t very durable. There are, of course, exceptions, including a few I’d like, but for the most part, the rush to market means that many of the IoT devices sold aren’t much better than most of the home built Kickstarter projects. That’s not to say Kickstarter (or Indiegogo or any other site) doesn’t produce good products, just that some aren’t.

    We’ve got an infographic from Website Guide today that you can examine. It’s an interesting item, which is why I published it. There are a lot of great possible applications for IoT style devices. Plenty of businesses and industries are seeing the benefits of using sensors and devices that are (semi-) smart and connected to a network. Using well written applications, companies can come up with new ways of gathering and using information. When these systems are well designed, this can result in lower costs, higher sales (or usage), happier customers, or all three.

    As you scroll down the infographic, you’ll see the dark side of IoT, which is getting more press all the time. Security is the number one concern, as far, far too many devices don’t include the necessary security to protect either the device or the end user. I did watch an IoT demo at Live!360 last year using the Azure IoT hub, and was glad there was some authentication and ability to shut down devices that might be compromised or misbehaving, but far too many IoT systems aren’t well secured.

    That’s disconcerting for me as a database professional. I expect that more and more companies will use IoT in some way. I’d even like to find time for some IoT sensors around the ranch, tracking horses and ensuring they’re all walking around every day. The thing that concerns me is what happens when someone finds ways to hijack or hack a device? Even sensors could be hijacked. What if you’re processing temperature information in dynamic SQL and someone decides to heat and cool the sensor to create certain values? Are you sure your database isn’t vulnerable?

    These concerns leave out the potential for the misuse of any data that’s collected about users, especially if users aren’t aware of what’s being collected and its use. With so many ways of collecting data now, I’m not even sure that anyone has a good handle on all the data collected about them, not to mention how it’s being stored, secured, and used.

    I think there’s incredible potential for some great IoT applications and data in the future. All the forecasts and spending seem to point towards the next 5-10 years being a period where there will be an explosion of the number of types devices built and deployed. Those devices will produce a tremendous amount of data for us to manage in databases. We will have big challenges in storing, indexing, managing streams, and certainly, the pruning of old data.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.4MB) podcast or subscribe to the feed at iTunes and Libsyn