Voice of the DBA

Category: Editorial

  • Small Data

    Big data is a hot topic these days. Like the “cloud” terminology we’ve been hearing about for the last few years, there isn’t a good definition of what “big data” really is. The best one I’ve seen so far is data that “doesn’t fit in Excel,” which I like. So many people perform their analysis on a spreadsheet of sorts, that if the data doesn’t fit inside their edition of Excel, they’d probably consider it big.

    The problem with big data, however, is that it while it contains more information, it can also contain more irrelevant information. That’s noted in this piece on small data (from Brent Ozar, PLF), where the author states the signal to noise ration may be decreased when you examine very large data sets. You may find that there are correlations that appear to causations. With enough data, with enough things to examine, you can often start seeing patterns that aren’t really there. These ghost patterns can lead you to draw incorrect, or at least less correct, conclusions if you do not investigate further and test your ideas on portions of your data set.

    Some of you might have noticed fractal patterns like this:

    Mandel_zoom_00_mandelbrot_set

    This is a well known Mandelbrot Set pattern. However if we were to zoom in on this picture, we’d find that the patterns repeat over and over again. What holds true for the largest image we have holds true inside smaller sections. The pattern repeats.

    The same thing can happen with patterns in business. We may see a pattern in a large set of data, but we should verify that it also holds true for subsections of the same data set before we make a decision based on that pattern.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • Microsoft Research

    Microsoft is investing heavily in research.
    Microsoft is investing heavily in research.

    I’ve always thought that the idea of investing in research products was a good one in many industries. Some of the companies that were very successful across the last hundred years were those that invested in learning about the technologies and processes in their industries. From these ideas and inventions, engineers have built the products that companies sell and which enrich our lives. Government and academia have also helped fuel the advance of technology in America, but it takes business interests to provide the practical implementations of innovations that can be bought and sold.

    There are people that think corporations like Microsoft and Google should focus on their core businesses and maximize their profits. However I think the investments that these companies make in various research areas allow them to learn about new opportunities that may become products at some point. It can be hard to determine if a technology area is viable for products, and the targeted research these companies engage in can help them improve existing products, or invent new ones.

    I think that having research areas enables diverse opportunities that may help attract, or retain, some extremely talented employees. I know I’d love the chance to take a sabbatical for a year and work on a research project of some sort. The top talent at many large companies may want to work in a theoretical area for a period of time, and companies may find very talented academics that might want to try working on more practical endeavors.

    I was watching this keynote from Microsoft Research and it was amazing to me how much of an effort Microsoft is putting into their research. They claim they are the number one organization doing basic research in computer science. They publish more research, and funding more graduates student hours of work than any other single institution. Considering how large some universities are, this quite a commitment to research. It’s one that I hope they continue in the years to come.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.

  • State of the Cloud

    The Azure cloud is doing well.
    The Azure cloud is doing well.

    Microsoft has joined the billion dollar club. For Azure, that is. They announced that this platform and service will generate a billion dollars in revenue. It’s been growing, and at least one analyst expects the revenue to double annually. Profit is something else, but I know from various people at Microsoft that they are working hard and investing heavily in this service, and I’m sure at some point it will be profitable, though I’m not sure how much.

    Amazon has had a very successful AWS platform, providing a wide variety of services and products. Their aim is to go for the high volume, low margin business, which makes their service competitive with the costs of buying and managing your own hardware. As long as Amazon and other competitors are out there, I suspect cloud services will be cheap enough to make many companies consider moving some of their infrastructure to this type of environment. It certainly eases the time and effort of building new machines and encourages better development habits.

    As IT departments, especially the management of those groups consider using cloud type services, they still have to be aware that this isn’t a savings that comes for free. Code changes, staffing skill growth and mindset evolutions to take into the account the challenges of working in a cloud environment, as well as the SLAs that seem to favor the cloud providers more than the cloud customers, mean this isn’t an easy decision. Add to this the potential tax implications, and the decision to move away from current IT practices is not always obvious.

    The idea of cloud computing, and cloud services, is still very immature. From the definition of what this means, to the offerings, to the architectures, the industry has a lot of growth to do before most of us will feel comfortable moving substantial parts of our infrastructure out of our own facilities.

  • Factivism

    Bono is embracing his inner nerd.
    Bono is embracing his inner nerd.

    I believe that we should all give back to the world outside of our families at some point in our lives. This might not be the time for you, and you might not have a cause right now. However I think at some point performing volunteer work to better the world will give each of a deep sense of purpose in our lives. I was reminded of this recently while watching a talk from Bono, on the state of hunger and poverty in the world. Bono has devoted considerable personal energy and resources in working with one.org to fight extreme poverty around the world. This may or may not be your cause, but it was an interesting talk, one worth watching.

    The presentation wasn’t what I expected, with Bono embracing his “inner nerd”, as he put it. He talks not only about the state of his cause, but he uses data to present his case and inspire people to continue to work to make things better around the world. The term he used was not activism, but factivism. This is the idea of using facts and knowledge to further a cause. Whether you agree with Bono’s vision, his case, or his goal, the idea of using data to make changes is something most of us embrace as data professionals.

    The idea of using more data, more information to make decisions is something most of us do each day. However we don’t have to limit ourselves to just the data and knowledge we have at work. I find many people looking to showcase their skills or improve their brand. This might be a way in which you choose to do this. With the availability of data from governments and organizations, you can present your own data analysis, or even a visualization that shows how you work with a set of data. It’s the chance to make your own mark on the world, perhaps making the world better, or perhaps disputing the claims of others. Having data freely available allows us to check and verify someone’s work or dispute their conclusions. Both of which are important for a healthy debate.

    However you might tackle some data interpretation, you don’t need to make a political or social statement. You could just provide a story on how you examine the data, perhaps learning along the way. By showcasing your knowledge of data in some way, perhaps even explaining how to load the data, you get experience and practice in both a technical area, as well as in communicating your knowledge to others. If you choose to do so in pursuit of a cause, perhaps the world may be a little better for your efforts.

    Steve Jones

    The Voice of the DBA Podcasts

    We publish three versions of the podcast each day for you to enjoy.