Voice of the DBA

Tag: databases

  • Life Logging

    This editorial was orignally published on Dec 8, 2009. It is being re-run as Steve is on vacation.

    There’s a new book from Microsoft Research called Total Recall. It’s actually on my list to read soon. It’s from Gordon Bell of Microsoft Research, and it talks about the idea of life logging, of capturing things about what you do and what you’ve done, in audio, video, and text. These might be things that are private, available to you and not the public. Or things that you might want to share with some people, like your family, but not others. Dr. Bell actually carried around devices to record his life for a long time, with the idea of working through the technological hurdles of life logging.

    Whether you think you’d like this in your life or not, it’s an interesting knowledge idea for companies or individuals. Maybe it’s a good idea for programmers or DBAs. What if you could have all the code you’d ever written? What about tests of that code and the applications where it had been used? Would a walkthrough of tuning efforts you’d made for some types of SQL or even queries that solved particular challenges be useful? What about a walkthrough of the changes to your schema across time?

    There are some interesting possibilities here in terms of your career work. If you could add meta data, annotations, correlations between emails, requests for work, problems, your particular implementation, and then the later results,  you might learn from your mistakes quicker. You could perform some type of root-cause analysis of your efforts across time, or on a system and find out what you are doing wrong, or what you’re doing well.

    I don’t know how well you could analyze someone’s work and help them find out what they’re doing well or not well. I also have some serious concerns about the privacy and security of the details about someone’s efforts being shared without their consent, or even the ownership of this data. But I think there are some great possibilities here as we develop new, and cheaper, ways to collect and store all kinds of data.

    At the very least having a “life log” of a server might prove to be a very valuable troubleshooting tool.

    Steve Jones

     

  • Data is Not Always Enough

    As data professionals, our jobs deal with ensuring that there is data available, accurate, and relevant to the organizations in which we work. However we often go beyond the simple maintenance, gathering, and manipulation of data to help our clients and customers understand the information contained in our systems. In many of my positions, whether as developer or DBA, I’ve been tasked with working closely with business people to make decisions based on data.

    I’ve always been of the opinion that more data is better, though I certainly understand the problems of Big Data and the potential to reveal false patterns or overwhelm systems with conflicting information. The better I understand the domain, and the better educated the business analysts are, the better we can work to extract information from databases.

    However that’s not always enough to help an organization act. I read an interesting piece this week on data in sports, which talks about one of the NHL team’s managers using data to make decisions on how to restructure his team. It’s interesting to see that additional data gives the general manager more insight, and leads him towards a decision, but the rest of the organization can’t follow through. In this case it’s a matter of money, but resources constrain our efforts in many decisions. I think this is a place where more flexible, and perhaps more in-depth, BI-type analysis of what-if scenarios can be more helpful.

    The other piece that caught my eye had to do with real estate, and housing prices, with the author bemoaning the lack of data in determining the value of a house. It’s interesting, but to me, it’s flawed. More data won’t help because houses aren’t like many other commodities. One house is not fungible with another one, and the market is both fast moving, and inherently full of friction. We rarely buy a house without visiting it, a task that consumes time, and slows the movement of information through the system. I’m as frustrated as the author, but I don’t know that more data would help in this case.

    Data is important, and it should be a part of our decisions, but we should remember that data isn’t necessarily going to make the decision for us. We need to be ready to incorporate our own knowledge and judgment into a data set to help us decide on a course of action.

    Steve Jones

     

  • One Database to Rule Them All

    This is what you build to juggle 6,000 tweets a second. That’s the headline that caught my eye and it’s about the challenges of Twitter and the data that they handle. Twitter definitely has a tough problem, one that few of us have, but perhaps they can help us learn to better deal with our own data from their experiences on an edge case.

    The story is journalistic, not so technical, but it is interesting. Twitter has struggled with a blend of data that is partially crucial and must be consistent now (usernames) and other data that can be a bit out of date (tweets). They also have lots of unstructured data (photo/video) that is combined with more traditional, structured data. They’ve used a few different database platforms to store this data and assemble it with their application. That’s the same things that most of us also do when we deal with many different types of data.

    However Twitter is trying to find away around dealing with disparate systems. They’ve had a number of engineers working on Manhattan, their database designed to handle both structured and unstructured data. And because they work for Twitter, this platform is designed to manage all of this data with very high workload demands at scale.

    It will be interesting to see if they come up with any innovative ideas. Certainly SQL Server already has options for managing structured and unstructured data, though perhaps not at the scale Twitter needs.

    Steve Jones

  • The Internet of Things

    The Internet of Things is upon us. I’ve heard this term quite a bit recently, and certainly there’s been no shortage of new hardware items that are being onnected to the Internet. The last ten years have seen all sorts of products get IP addresses: mobile phones and cars, along with washing machines and refrigerators. Some have been silly attempts to make a connection to the Internet without a practical problem being solved, but some have been very interesting.

    However I think we will truly start to see more and more devices created in the next decade. The advent of cheap hardware (Raspberry Pis and Arduinos as two examples) and the ability to construct new cases and programs (with 3D printing) will lead to many individuals, as well as companies, starting to build their own devices to capture data as sensors, or perform small tasks that we find handy.

    Should we care? Yes. Many of these items will produce logs or other status information. Much of this data will be stored in databases. Perhaps in relational platforms like SQL Server, perhaps sampled and queried as a stream with much of the data discarded (using StreamInsight and SQL Server), perhaps stored in some NoSQL type platform (HDInsight/Hadoop, anyone?). For many of us that means more data to manage, new information to develop software against, new patterns to discover with creative queries.

    It should also mean more work, probably more employment, and hopefully, more money.

    Steve Jones

     

    The Voice of the DBA Podcast
    Listen to the MP3 Audio ( 1.9MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at http://www.everydayjones.com.