Voice of the DBA

Tag: data analysis

  • Analyzing Data in Real Time

    A few years ago I was listening to a presentation from Microsoft where they touted a customer that had real time analytics with Analysis Services. Since my exposure had been that cubes needed to be processed, a resource intensive process, I was wondering what this meant. Imagine my surprise to hear that the cube was being updated within 5 seconds of the changes in the OLTP system. Talk about real time.

    Of course, the definition of real time isn’t really well known. We linked a piece in this week’s newsletter that looks at a survey of companies and their investment in real time, streaming, data movement pipelines. The definitions from companies about real time range from sub-second to 24 hours. I’m sure I think that 24 hours is real time, but if you’re used to processing cubes weekly, maybe that feels real time. I tend to see real time on the order of minutes, maybe a few tens of minutes. Once we get to hours, that feels more like batch processing, albeit fairly rapid batch work.

    I am sure there are systems that need real time analysis, especially in some areas where decisions must be made quickly. As we move to more and more machine learning algorithms and automated intelligence, there will certainly be more call for real-time data movement. Yet another opportunity for data professionals, especially those that work with ETL pipelines. There will be tough problems, not only with moving data, but tracking lineage, recovering from issues, even weeding out bad data quickly.

    However, that’s an opportunity for you, not me. I bet there are a few data professionals out there that feel the way I do. There are great challenges in solving real time problems, and building systems that can handle high volumes of data. The thing is, I value my sleep. I value not working all-nighters, I value a balance in my life. While I find the problems fascinating and the money involved tempting, I prefer to work on more pedestrian systems.

    I’m glad there are people that want to work on very difficult problems, and I wish you all the best in taking advantage of these opportunities. I hope you’re well paid, and you have a great time building these impressive systems. I also hope to read about some of the amazing things you do, so please, share the knowledge where you can. This is a great, exciting time to work with data, and it’s one that I continue to enjoy every day.

    Steve Jones

     

  • Data Science Education

    One of the good skills to have for a data professional is how to analyze data. Most of us could learn more about data science and data analysis for some aspect of our jobs. We are data professionals, so we should understand how to analyze data. I’d expect that a competent data professional would be able to put together a report on some set of data that means something to an end user. If we administer systems, then analyzing usage (index, space, etc) is a skill we need. If we write code, sooner or later we’re going to write some report for a client. Either way, we need to perform some sort of analysis.

    How can you learn more? There are lots of resources available. I thought it was interesting that Microsoft has teamed up with edX for their own data science degree. While there are mixed feelings on this, I think edX has a good platform and strong partnerships for teaching. There are other places, such as Coursera, that are doing the same thing, offering a variety of courses online.

    In fact, it appears that quite a few educational institutions and businesses are starting to increase their data science related offerings. There’s a good summary of some of the options in a piece from Dataversity. In response to all the demand, or at least perceived demand, there is everything from a boot camp getting you up to speed on some quick analytic techniques and tools to full degree programs. Some programs have different levels, depending on the amount of statistical knowledge you have.

    Learning something about statistics is probably the best way for most people that would like to get started. I’ve seen quite a few people start learning by reviewing some statistics techniques. From Buck to Mala, putting your own learning down in a blog will force you to ensure that you actually understand the principles. As for getting started, there are lots of great books you can buy, or read information online. One note, be careful and double check the information written on random blogs. Or check out places like the Khan Academy.

    In some sense this reminds me to the rush to get a CNE in the early 90s, or the many boot camps that appeared to help people a MCSE certification in the mid to late 90s. Lots of people passed the tests, but weren’t very qualified, which has contributed to the general distrust of certifications today. I hope that doesn’t happen too much today, though I’m sure it will to some extent. The chance to make more money, or just find a job, will drive lots of people to look for quick wins.

    While I’m sure some of you will get better jobs, perhaps making a lot of money with minimal data science experience, I hope many of you continue to learn and improve your skills in the data analysis area over time, whether you are paid to be a data scientist or not.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Mevio .

  • Too Much Information for Sports

    Many of us deal with lots of data and information, sometimes conflating the two when they are really separate things. Our jobs usually revolve around somehow  helping others to extract useful value from the data we store in some way. This could be with reports, or with transfers to another system, or even a simple explanation of what data in a column means from a business perspective.

    Like many industries, pro sports have been using more and more data to help them better manage their business. This was popularized with Moneyball, a book (and movie) about baseball. Since this time, plenty of other teams across many sports have started think about data analytics. The NBA gathers lots of data for each team, even making some available publicly (including spatial shot data). The NFL uses lots of sensors, and more sports, especially Olympic sports, are gathering data to help athletes perform better and learn more about the impact of their decisions.

    However, more data isn’t necessarily what makes decisions on the field of play. There was a great quote from Doug Baldwin, an American Football player for the Seattle Seahawks. In looking at all the data collected, he noted this:  “Yes, the data and information is useful, and give it all to me. But at the end of the day, the user has to use it the right way.”

    That’s a great quote, especially considering the owner of his team is Paul Allen, founder of Microsoft. More data doesn’t necessarily solve a problem, tell you what to do, or how to do it. Instead, data analytics and analysis is still a bit of an art with some science behind it. Data matters, but when we reduce a problem to a simple set of statistics, we can be missing the subtleties of the actual situation.

    I wouldn’t think most industries could do without data gathering and analysis, but people still need to be involved and work with the bits and bytes. Whether you’re in sports or business or some other area, it’s important to use data, but don’t discount the human factor in applying experience and understanding to extracting information from that data.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Mevio .

  • Fuzzy Processing

    This is really interesting. A computer chip that doesn’t exactly perform mathematical calculations correctly. That sounds crazy, and in some cases it is. Most of us want math to be exact, and work perfectly every time when we run calculations. Can you imagine if Excel didn’t always add up the totals in your expense report according to the rules of addition?

    However in more complex operations, such as pattern matching for images, we might not want exact calculations. If two radar images (one problem in the article) don’t match exactly at the binary level, would we want a search for similar images to discard one? After all, exactness and perfect matching are great in some areas, such as financial accounting, but in other areas, such as imaging, there is plenty of noise that isn’t important to the content of the image.

    Do we care? Perhaps. As data professionals, I suspect that we will get more and more data of disparate types, including images, that will become part of our databases. There will be clients the need to search and query this data, which means that we may need fuzzy search tools that work well.

    New search tools will likely mean that we will need to learn more about how to tune queries, or even indexes, that might deal with data in a way that isn’t exact. I suspect that there will be opportunities for those that learn how to deal with these types of problems in an effective manner. There might even be really good paychecks that come along with these jobs.

    It’s a bit disconcerting to think that we might want computer applications that don’t work exactly as we expect with regards to calculations. However, I think the science of uncertainty will become more relevant and valuable to us data professionals in the future as we deal with lots of data that is more complex than a simple addition problem.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn.