Voice of the DBA

Tag: data analysis

  • Baseball Analytics

    I love baseball. In fact, for the last ten years I’ve participated in an adult league each summer, playing 12-20 games each year against other 40 year olds. While we play the same game with the same rules as the professionals, we have a much different outlook on the game, with many of us enjoying the competition and distraction, but often viewing avoiding injury as a higher priority than winning.

    Baseball is an interesting game as the professional season is so long and for over a century there have been a huge amount of statistical data captured about each team. Prior to the fantasy sports popularity explosion, people had leagues that competed against each other solely based on this statistical data. I even wrote a program with a friend on an Apple II that allowed us to set up teams of random players, read in statistics, and then compete against each other using pitching v batting statistics, just showing us text results.

    I ran across an interesting post from a consultant that examined a hypothesis he had about winning baseball teams playing better at the end of the season. Using a free data set, the R language, and a Jupyter Notebook, he was able to perform an analysis that proved he was right. Whether or not you agree with him, I would guess that this analysis didn’t take that long to complete, and was fun as well.

    With R coming to SQL Server and the increasing call from organizations to perform more complex analysis on our data, I think this is a great way to get started. In fact, I’m thinking that I should load up some of the datasets myself and see what kinds of things I can prove or disprove about my knowledge of the game. If nothing else, looking at baseball stats is probably a fun way to brush up my statistics skills.

    BTW: Congratulations to the Kansas City Royals for their World Series win (and to the #sqlfamily in KC).

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Amateur Data Analysis

    What’s a good way to analyze data? How do you know if you’re actually looking at data in a way that provides a valid analysis? It’s entirely possible to statistically look at a set of data, run some aggregates, build graphs, and come to a conclusion (or recommendation) that would hurt your business rather than help it.

    In academia, many people have their analysis and conclusions reviewed by their peers. Over time, many of the people analyzing data learn from others and start to build skills in how to look at data sets and consider the interpretations that seem to be more valid. Of course plenty of mistakes are still made, but I think the quality of analysis is pretty good overall.

    In business we are often working in silos, in semi-secretive ways where our analysis might not be questioned or reviewed. How do we build some skills? One way might be to do what Dev Nambi did, publishing an analysis of college costs and including his thoughts on what the data shows. You could also look at this crime report from Samuel Vanga.

    I thought this was a great example of taking a set of data and trying to make sense of it in a variety of ways. While you might have your own thoughts on what conclusions and implications to draw (leave comments about that for Dev on his blog), I think this is an interesting way to approach analysis of a set of data. Many of us could experiment with different visualizations and the analysis of this (or other data), and get comments on our approach.

    For example, I find the stacked bar graphs more difficult to understand than a series of line graphs. I also would like a bit more context in what the author sees as an analysis of each graph, but those are my views. Perhaps if I wrote an analysis of some set of data, I’d find others would let me know the ways in which I present my findings are flawed or difficult to understand.

    I’d encourage you to practice building analysis, along with other skills you find useful in your job. While most of you can’t use business data on your own blog, perhaps you can find a data set that’s interesting to you and dig in to see what information you can extract and present.

    Steve Jones

  • Data or Experience

    Data Driven

    Listen to the data.

    There are any number of phrases that implore us to use bits and bytes, pieces of information that lead us to better decisions. However, can the data ensure we make the best decisions? Do the models we use get better and better over time? It’s hard to tell.

    This has certainly been debated in the wake of Moneyball, the book that ocuses on the use of data over experience to drive decisions for baseball teams. The Oakland As were the first team to do this, without winning a championship, but the Boston Red Sox also followed the formula and won three championships in the last eleven years.

    However they’ve also had some abysmal years, including the current one. Does that mean that the principles of data driven decisions work, don’t work, or something in between? Personally, I think that the ideas of predictive analytics does work, but it’s not magic. It’s also not a guarantee of reaching some level of performance.

    In sports there’s a strong human element involved. While many players do perform at a similar level from one year to the next (slightly higher or lower), there are also times when a player dramatically diverges from the past. There’s also the notion that a sports team is a very small sample size for statistical analysis. Trends tend to be easier to predict when there are thousands of people’s behavior involved.

    There’s also some variance from transaction to transaction. Even in retail, where I might be able to predict today’s sales fairly accurately, I couldn’t necessarily determine the volume of sales for a particular product or the total on any transaction. Statistics are generated over time, and they’ll be accurate over time as well. Just understand that in the more a human is involved and the more detailed your granularity, the more actual results might deviate from your predictions.

    Even the card counters in Bringing Down the House, for all the millions they earned, still expected to lose some hands, and occasionally some big ones. Keep that in mind when making data driven decisions.

    Steve Jones

     

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Statistical Analysis

    I ran across a book recently that is written to address some of the statistical issues the author has found in many scientific papers. It’s been released online for free, but if you like it and would like a more portable format, you can buy it for a reasonable price.

    I haven’t dug into the book much, and I certainly don’t consider myself to be a statistician. That wasn’t one of my stronger subjects in school, and it’s a weakness I’d like to correct at some point. However I have had to use some of the statistical functions in past applications, and I wonder if I was using them wrong.

    This week I decided to see how many of you are using more complex math in your systems. I’m hoping you understand how the functions work, but I wanted to ask what you’re using:

    What statistical functions have you implemented in a production system?

    I am thinking of functions beyond basic aggregates like SUM and AVG. Are you using standard deviations, linear regressions or some other complex functions? Have you made use of built-in functions in T-SQL, R, or some other language? Are you implementing custom functions in code or CLR Aggregates?

    I think this is one of the areas where our profession will grow more and more across the next decade. As we deal with lots of data of varying types, and our organizations look to gain some strategic advantage through deeper insight into their information, we will have lots of chances to experiment and learn more about complex data analysis.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.1MB) podcast or subscribe to the feed at iTunes and LibSyn. feed