Voice of the DBA

Author: way0utwest

  • Local Database Issues

    There was a report that Spotify is writing lots of junk data to users’ local drives. Some people have noted GB a day or writes, even when they aren’t using Spotify for listening to music. Some debugging by users show that a local database (SQLite) is getting a number of maintenance calls to the database that are being repeated over and over. This appears to be a bug, and one that is hopefully being fixed. I know it hasn’t hit me, so perhaps there are more factors in play than just using Spotify.

    This does bring to mind a few things to me that we should be careful of as software and data professionals. The first is that while bugs will creep into our software, being able to deliver fixes is important. I’ve seen updates made to Spotify in a smooth fashion, one that doesn’t seem to require much effort from me. While I don’t usually get to decide when these updates will apply, and they can be annoying if I reboot and attempt to use the application, this isn’t a critical piece of software for me. For some of us, though, our users can be impacted if we don’t allow them the chance to choose when updates occur, or at least, when the final time is when the old version might no longer be supported (or stop working).

    The bigger issue, to me, is that the local database is being used in a way that users don’t expect. There’s likely a bug here, causing lots of additional activity, but what’s a normal level of activity? What would our users expect? Do they even know that we’re keeping data locally on their system? In many cases they might not care, or even assume we are, but we should disclose this to our users. Security concerns over data leakage or loss may be an issue in some environments, and it’s important that we ensure our users are aware, or at least they can find, information about the data our software may collect, store, and use.

    Data becomes more important all the time, and we are constantly finding new issues surrounding its capture, storage, security, and use. The way we even think about data is somewhat immature. Data ownership, protection, lifetime, and more are concerns. The problem is that many of us would struggle to articulate how exactly we want to handle the data that is related to our lives. In fact, we may have vastly different ideas on how different types of data are handled. Perhaps we don’t care about our musical listening habits, but we care deeply about someone learning about our reading history.

    If you develop software that uses a database, and many of you do, it would be nice to document (maybe self document with code) the ways in which data is stored and managed. However, if you’re systems are like mine, there might not even be any policies on how data is managed and handled over time. That’s something I hope changes in my lifetime and we start to treat data like a very valuable asset that can impact our lives.

    Steve Jones

  • The Google Solution

    I was watching a talk from Google on how they back up their data. After all, one of the biggest assets Google has is the tremendous amount of data that they have collected over the years. There are many systems at Google, and certainly lots of data in each of them. In this case, Gmail was the system being discussed, and the data is in the exabyte range. If you don’t have an hour to watch the talk, then there’s a nice summary at highscalabilty.

    In the talk, there are certainly things that Google aims to do with their approach. First, they can’t lose data. That’s a priority, as it should be for all of us that manage data. They also focus on restores, not backups. In fact, if they can make restores easily by adding work and complexity to backups, that’s a trade-off worth making. I haven’t typically viewed the restore process this way, though I do think restores are ultimately the most important part of any recovery task. However, I haven’t really thought about how I could actually make restores easier by changing something at backup time. I’ve often tried to make backups quicker, or take them more often, but perhaps this is an area to re-examine. Are there things you can think of that would make restores easier? Maybe not easy enough for your cat to kick off (as discussed in the video), but easy for the average sysadmin at your company?

    Google wants redundancy, which includes people. They can’t depend on any one machine, one tape, or one person. Therefore, they need to have multiple copies of data and more automation that reduce those single points of failure. Along those lines, our clients and customers don’t need to know if we have 3 copies, 7 machines, or any other configuration. Our responsibility is to ensure our customers can access data.

    Why should we care what Google does with GMail or any of their systems? Well, I only see our databases growing, with sizes going from GB and millions of rows to TB and billions of rows, or even to PBs. There are lessons that we can learn about the management of data at scale, and the ways in which our customers might perceive the availability and accessibility of their information. Google has learned they need to be more efficient with resource usage. Whether that’s disks or people, they can’t require 1000 times more resources for 1000 times more data. We should take note of that.

    Perhaps the best lessons from Google are in the areas of testing and expectations. They test constantly, to ensure that they can actually recover data. While I think the SQL Server backup system is very solid, I’d be regularly testing restores to ensure that I really can recover from backup files, from external disks and tape, or even complete a restore on a backup system. The other lesson is that Google expects things to fail, so they plan for that, and aren’t surprised by failures. SQL Server gives us options here with Always On and other HA technologies if we can take advantage of them. With a single RDBMS instance, there isn’t a lot most of us can do, but we can at least be prepared to rebuild our instance elsewhere as a last resort.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.8MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Go Code

    Should we be teaching more people to code? That’s a question I ponder, especially after re-reading an old view against the idea, a newer one, and a pro-code piece. At the end of the day, I think that coding is a basic skill, similar to reading and writing because so many of us, perhaps most of us in the developed world, will need to work with computers.

    I don’t mean to imply that everyone should be able to write an MVC application or produce a graphical application. I also don’t expect most people to understand calculus or produce a thesis. However, I do want a core competency that allows them to work effectively with a checkbook and communicate a point effectively. I want them to understand enough coding to appreciate the logical and methodical way in which computers process instructions. I’d like to have most people appreciate how to create, store,  shuffle, and retrieve a digital deck of cards.

    Certainly I think that those people who continue to learn and advance in many industries can benefit from an understanding of what a database is, how to write a query that might logically fetch information. Even if you use a tool such as Excel, Power BI, Tableau, etc., knowing the ways in which we store information and need to process it can help you better work with the computing machines in your life.

    Perhaps seeing how easy it can be to produce bugs will help people also appreciate the need for better security. I’d hope this would also have them demanding that professional software developers produce better work. Quality matters, and while I might be able to build a car, I would expect that one produced by professionals would be of a much higher quality, and much more reliable than my own efforts.

    Having a basic understanding of how computers think will not only help us understand our systems better (or the people that design them), but can also help us develop thoughts and views on how we should reconcile the digital with analog world. I strongly feel that our understanding of, and the expectations we have in the analog world are very much inadequate for assessing a digital world that can accomplish things many people would have viewed as magic a few years ago. Certainly our legal structures aren’t well equipped to work in a digital era. To help us move forward, I’d like more people to understand computing.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.3MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • A Good Deed After Game Night

    When we set up Game Night, we decided to sell tickets to reduce our risk. However, we didn’t have a good way to easily match up the ticket price with the costs of drinks. As a result, we ended up with about $105 extra from our ticket sales over the bar bill.

    When Andy (b|t) and I work on putting together an event, it’s not usually to make money. In fact, we often joke that our best ideas have no way to make money. That’s OK, since part of what we believe is that we need to make a difference in the world at times. We work to earn money for our families, but we also want to spend time trying to do some good in the world.

    As a result, we wanted to take the extra money and help someone. We had a few ideas, and contacted some people, but eventually settled on this.

    Don Kolenda needed eye surgery. He’s a member of our SQL Server community, and we decided to donate the $105 to Don. If any of you want to help Don, please feel free to do so.

    Thanks to everyone that supported Game Night, and know that a little bit of your ticket sales went to help someone in need.