Voice of the DBA

Tag: data

  • The Randomness of Analog

    One of the joys for much of my early life was walking through a library, looking for a book to curl up in the corner with and read. As a young boy, I would walk to a local library and read at the wooden tables before I had my own library card to check out books. Later, I moved and had a newer library with large, comfortable chairs in which to sit and read a few pages. I’ve enjoyed the same thing as an adult in various bookstores.

    I didn’t often have a specific book I wanted to read, so I’d randomly walk around, looking at spines, covers, and choosing a book in a somewhat random fashion. My fellow founder at SQLServerCentral, Andy Warren, also appreciated the randomness of browsing in a library for bookstore, discovering some new author or story to enjoy. Across the years, we’ve discussed and debated whether or not there was a way to duplicate this experience with technology.

    These days I tend to buy or borrow all my books electronically. The convenience and unparalleled and time is valuable, and I certainly don’t miss the days of packing 4-5 large books for a week long conference trip. However, Amazon and my local library tend to use recommendation algorithms, or popular titles as the presentation method for their sites. I have lost quite a bit of the ability to enjoy the randomness that comes from wandering and happening upon new titles. Andy feels the same way, though none of our brainstorming has produced a way to duplicate the feeling of wandering through bookshelves in an electronic fashion.

    I’m not sure if there is or isn’t a way to deuplicate this electroncially, but certainly the feel isn’t the same on a screen. All too often our focus when working with data is narrowed to a limited set of choices. And often when we build applications and provide data to users, we are trying to be exacting and relevant, not random. So much of what we choose to do in software is to remove much of the randomness from our systems. Event the “browse” features are often scoped or focused in a particular topic, subject, or area.

    This filtering to a particular bubble of data is one of those areas where we have tremendous power in shaping the world. The code and queries we write, the organization of our data, this will have an impact on our users, and I’m not always sure this is for the best. Perhaps overall this is more helpful, but it also serves to prevent us from viewing the forest, only seeing the trees. If you doubt this, try browsing the internet in Private mode sometimes and run searches. You might be amazed how different the Internet can look.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • More Open Data

    Steve Ballmer retired from Microsoft almost three years ago. It seems like a small eternity, and since his purchase of the NBA Los Angeles Clippers, I haven’t heard much about any ventures with which he’s been involved. Apparently, he’s spent some of his time, and quite a bit of money to build a site that discloses data about revenue and expenditures for the US Government. There’s a piece in the NYT about this project, as well as a shorter Engadget summary of the site, which provide a short look at the project.

    USAFacts.org is the site, and it’s a treasure of data sets. I look at this as a really interesting way to examine data sets that might be more difficult to gather than you’d expect. The Data Act deadline takes effect in May of 2017, which should also provide another way for anyone to look at public data and perform an analysis. I am disappointed the downloads aren’t working yet, but I hope that this will come soon, along with some sourcing information about where the data comes from and how it was gathered.

    Having data sets to analyze is important for any organization. Certainly within our organizations we spend a lot of time producing reports and queries that help various people analyze data. In fact, finding, collating, cleaning, and organizing information can be a taxing proposition in any size organization. Our data sets and sources are so diverse and often inconsistently producing data that it’s amazing at times that our organizations run well. It seems on a regular basis someone wants to rebuild the methodology used to gather and organize information. I am not surprised that I constantly find incorrect calculations in software because the basis we use changes too often.

    The big issue for me is that so many of us are amateurs when it comes to analyzing information. There aren’t many organized classes or a good structure for most data professionals to learn how to analyze data. We learn on the job, we make guesses and assumptions, and overall do a good job. However, data analysis is highly inconsistent from person to person. I’d like to see that change, and as I see more and more people blogging and talking about how they look at a particular data set, I hope more people are thinking about how to analyze information and how the choices we make for calculations, visualizations, and even ordering can affect how the results are interpreted.

    I’m glad Mr. Ballmer has started this project, and I look forward to seeing how people might use this data and other data sets to provide some analysis of the world.

    Steve Jones

  • We are All Data Companies

    I have seen more than a few people note that every company is a software company. Or that every company should be a software company because software is becoming so important. I’m not sure that’s true. My mechanic certainly uses software, and he competes well with other businesses by tracking my car, reminding me of issues, and keeping in contact through integration between his point of sale system and some sort of email management software. However, this is a small shop, probably less than 20 employees, and I’m not sure they’re ever going to be in the business of software. They are consumers, and they need support, but they aren’t in the software business. My wife’s company is similar. She needs a website, needs to schedule events and take payments, but she’s not going to develop software to help her. Nor am I going to write code, because that might be a never-ending job.

    It’s easy to get out of touch and think that many larger companies need to be software companies. That might be true for some businesses, especially when they need to manage some infrastructure, but for many smaller businesses, I’m not sure that’s the case. Software is critical to small medical practices and retail stores, and service people, but only as a tool. Not as something they’d want to maintain or even customize beyond a simple look and feel.

    Is that true for data? If many companies aren’t software companies, are they data companies? I think so. Data is important, and having some way to look at data and manage it becomes important for them to run their business well. My wife struggles with data, but realizes that managing her mailing list and tracking the contacts is valuable. It’s hard, and her bits are stored in disparate applications, but she recognize the value of data. The same for my mechanic. They’ve seen plenty of value in treating a string of disparate transactions from me across multiple cars as those from a single customer. They also have learned that finding software to understand me as a customer, as well as string together recommendations and past work on single vehicles allows them to provide better service.

    Many companies want to track us through email, phone, or some other identification data because they realize that many people have come to expect a business to know something about us. We might not like them sharing or selling that data, but we do want to them to provide a better experience for us. I’m still surprised when there’s good integration between disparate systems, or a customer service call allows the company to recognize something about me and provide more value. I believe my kids and future generations will expect this to always be the case, shunning companies those that don’t provide customization.

    I worry about data security, data privacy, data integrity, and more, but I do also think that as more companies realize that they are data companies, and data drives many of their interactions, they will see data as more valuable. What they do with that, I’m not sure, but I can only hope they start to treat our data as a valuable resource worth protecting.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Local Database Issues

    There was a report that Spotify is writing lots of junk data to users’ local drives. Some people have noted GB a day or writes, even when they aren’t using Spotify for listening to music. Some debugging by users show that a local database (SQLite) is getting a number of maintenance calls to the database that are being repeated over and over. This appears to be a bug, and one that is hopefully being fixed. I know it hasn’t hit me, so perhaps there are more factors in play than just using Spotify.

    This does bring to mind a few things to me that we should be careful of as software and data professionals. The first is that while bugs will creep into our software, being able to deliver fixes is important. I’ve seen updates made to Spotify in a smooth fashion, one that doesn’t seem to require much effort from me. While I don’t usually get to decide when these updates will apply, and they can be annoying if I reboot and attempt to use the application, this isn’t a critical piece of software for me. For some of us, though, our users can be impacted if we don’t allow them the chance to choose when updates occur, or at least, when the final time is when the old version might no longer be supported (or stop working).

    The bigger issue, to me, is that the local database is being used in a way that users don’t expect. There’s likely a bug here, causing lots of additional activity, but what’s a normal level of activity? What would our users expect? Do they even know that we’re keeping data locally on their system? In many cases they might not care, or even assume we are, but we should disclose this to our users. Security concerns over data leakage or loss may be an issue in some environments, and it’s important that we ensure our users are aware, or at least they can find, information about the data our software may collect, store, and use.

    Data becomes more important all the time, and we are constantly finding new issues surrounding its capture, storage, security, and use. The way we even think about data is somewhat immature. Data ownership, protection, lifetime, and more are concerns. The problem is that many of us would struggle to articulate how exactly we want to handle the data that is related to our lives. In fact, we may have vastly different ideas on how different types of data are handled. Perhaps we don’t care about our musical listening habits, but we care deeply about someone learning about our reading history.

    If you develop software that uses a database, and many of you do, it would be nice to document (maybe self document with code) the ways in which data is stored and managed. However, if you’re systems are like mine, there might not even be any policies on how data is managed and handled over time. That’s something I hope changes in my lifetime and we start to treat data like a very valuable asset that can impact our lives.

    Steve Jones