Voice of the DBA

Tag: data mining

  • Data or Experience

    Data Driven

    Listen to the data.

    There are any number of phrases that implore us to use bits and bytes, pieces of information that lead us to better decisions. However, can the data ensure we make the best decisions? Do the models we use get better and better over time? It’s hard to tell.

    This has certainly been debated in the wake of Moneyball, the book that ocuses on the use of data over experience to drive decisions for baseball teams. The Oakland As were the first team to do this, without winning a championship, but the Boston Red Sox also followed the formula and won three championships in the last eleven years.

    However they’ve also had some abysmal years, including the current one. Does that mean that the principles of data driven decisions work, don’t work, or something in between? Personally, I think that the ideas of predictive analytics does work, but it’s not magic. It’s also not a guarantee of reaching some level of performance.

    In sports there’s a strong human element involved. While many players do perform at a similar level from one year to the next (slightly higher or lower), there are also times when a player dramatically diverges from the past. There’s also the notion that a sports team is a very small sample size for statistical analysis. Trends tend to be easier to predict when there are thousands of people’s behavior involved.

    There’s also some variance from transaction to transaction. Even in retail, where I might be able to predict today’s sales fairly accurately, I couldn’t necessarily determine the volume of sales for a particular product or the total on any transaction. Statistics are generated over time, and they’ll be accurate over time as well. Just understand that in the more a human is involved and the more detailed your granularity, the more actual results might deviate from your predictions.

    Even the card counters in Bringing Down the House, for all the millions they earned, still expected to lose some hands, and occasionally some big ones. Keep that in mind when making data driven decisions.

    Steve Jones

     

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • The Value of Data

    How valuable is your data? It’s a good question, and certainly the type of data your organization has along with the business in which you are engaged will make your data more or less valuable. More and more we find the differentiation between companies is in the way they collect, manage, and use the data available to them. So much in business is based on guesses, but more and more the guesses have some basis in data. We are starting to see those who make decisions in business feel some need to justify or support their choices with data.

    Is data the new oil? Oil was arguably the most important commodity of the twentieth century (and perhaps still is). The SQLRockstar wrote a piece with that same title, with the idea that knowing more about how valuable data can be will make you more successful in business. The post is based on the review of Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, and talks about some of the challenges of using data to make decisions.

    I certainly believe in the power of data, and that more data often gives us more insight into how the world works, as well as allowing us to draw some inferences about the future. Not necessarily better insight, but certainly more. I do think that computer extrapolation of patterns to the future is vastly overrated as most of our algorithms are far too simple, using too little data and discounting the increasing effects of small variables as scale increases. In short, I don’t think we’re anywhere close to a Foundation-like computer that can help us predict the success of new products, much less the future of a country.

    However I think that using analytics to make small decisions, and help guide our directions is important. We will still need humans that apply their internal supercomputers to interpret data, and continue to evolve the algorithms, and I hope that more and more of you are gaining deeper industry insight in your particular field. After all, many of us data professionals will be needed to help guide analysts in gathering, transforming, interpreting, and displaying data in ways that allows us to make decisions with more confidence.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Problems with Big Data

    Big Data is constantly in the news. We’ve been asked at SQLserverCentral to try and develop some articles, perhaps even a stairway to explain what Big Data is and how we might use it. I’m still trying to grasp the concepts myself, and unlike the amorphous cloud, I’m still looking for some good examples of what Big Data really is.

    When I ran across this piece warning that Big Data isn’t the final solution to all our questions in the world, I wasn’t surprised. The piece notes that Google Flu hasn’t been very accurate in its predictions of outbreaks. At first glance, this gives lots of credence to the idea that the good, solid data analysis and mining techniques we’ve used for years are just as good as any new Big Data fad.

    However as I read more about the piece, it’s not that big data and the analysis of large quantities of information is flawed, it’s that a solid hypothesis matters. Researchers need to be willing to evolve their algorithms as they learn more about a problem. Probably they should also assume their algorithms are not correct until they’ve proven their ability to predict actual trends for some period of time.

    We’ll constantly be searching for ways to better interpret information and make better decisions. No new technology or product is going to magically solve our problems. Good solid understanding of the problem domain will continue to matter as much as the data itself.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.8MB) podcast or subscribe to the feed at iTunes and Mevio . feed

    The Voice of the DBA podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

  • Mining for Quitters

    This editorial was originally published on Jun 10, 2009. It is being re-published as Steve is on vacation.

    I thought this was pretty interesting, a story about a story about Google trying to figure out which people are likely to quit. Apparently Google is worried about losing talented workers, like  those they’ve taken from Microsoft, Yahoo, and other places. Actually they haven’t taken anyone. They built a company that excited people and anyone that wants to get a job there (and can), should be allowed to go work there.

    Some people have decided they don’t want to stay, and quite a few executives, and Google wants to try and see if they can keep them. So they are looking for signs that a person might think about leaving, and then hopefully doing something about it. And in typical Google fashion, they’ve built an application that crunches data to identify those people likely to leave.

    I think it’s great, especially in an age where it seems so many companies don’t care if you stay or go. While there’s no guarantee that they’ll be able to convince people to stay, at least they can make an attempt to convince them. Sometimes that’s all takes. Show someone that you care, and that you want to keep them around, and many times they might decide to stay. And perhaps it’s in response to this note that the Justice Department in the US considers any agreement among tech firms illegal. Even if it’s an understanding not to poach each others’ executives.

    With any technology, there’s a flip side as well. Who knows if employees at Google will try to “game” the system, fake the data that shows the might leave, and then see if they can get a raise or some other benefit from the company. It might not work for all employees, but it might work for some.

    It will be a flawed system, but it’s a good idea. It’s a good use of business intelligence computing to try and protect the company’s most valuable assets: its people.

    Steve Jones

    The Voice of the DBA Podcasts

    Everyday Jones

    The podcast feeds are available at sqlservercentral.mevio.com. Comments are definitely appreciated and wanted, and you can get feeds from there.

    You can also follow Steve Jones on Twitter:

    Overall RSS Feed:  or now on iTunes! 

    Today’s podcast features music by Everyday Jones. No relation, but I stumbled on to them and really like the music. Support this great duo at www.everydayjones.com.

    I really appreciate and value feedback on the podcasts. Let us know what you like, don’t like, or even send in ideas for the show. If you’d like to comment, post something here. The boss will be sure to read it.