Voice of the DBA

Tag: data analysis

  • Opening Up Data

    Tim O’Reilly has been an advocate of open data access and standards for some time, especially from governments. He’s pushed for more interoperability and certainly more accessability from all sorts of groups. He gave an interview earlier this year to LinuxVoice where he talked about a variety of things, but data was foremost on his mind.

    There are some good thoughts, but I was pleased to see him looking for more software to adapt how it works with data rather than asking data to match the application. An interesting thought he had was in the area of control systems. Does every device or sensor need a separate application and way of interacting or should we have some guiding design principles that let similar applications work in similar ways with different data? That almost sounds like good data modeling and normalization principles in action, backing a data driven application.

    I also liked his acknowledgment of the fact that so much of our data isn’t very portable. Between social networks and proprietary storage, it becomes hard to move data around. The pattern of downloading data, perhaps editing, perhaps not, and then uploading elsewhere works great with ETL tools, but it’s cumbersome for many users and applications to deal with. Building ways for us to interact with disparate data, allowing for queries to remote sources, sometimes transforming and copying data, all of this needs to be easier to implement and integrate inside software.

    In some ways, I think the 3.0 model of our Internet interaction will take place around data. I think SSIS will continue to be one of the most valuable tools in SQL Server (along with lots of demand for work), but it still needs improvement and enhancement to catch up to other ETL tools. I really hope Microsoft believes this and continues to invest in the tool.

    I also think that the data professionals that really stand out in the next decade will be those that learn to make the choices about when to use R, JSON, XML, HADOOP, or whatever non-RDBMS tool to meet a need. But also when not to use these tools. The better data professionals will make good decisions about when to query data, and when to move it to another system.

    It’s an exciting time to work with data as the opportunities and rewards continue to expand and grow. I look forward to what the future will bring us.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.2MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Microsoft and R

    Microsoft purchased Revolution Analytics recently, a commercial company that works with the R programming language. This seems to be a decision to improve the analytics and analysis offerings from Microsoft. I’ve heard this will be incorporated into their Machine Learning offering, but I wouldn’t be surprised to see some basic R support in SQL Server at some point. If you’re not familiar with R, we’ve got a basic piece at SQLServerCentral on it.

    Microsoft has made lots of acquisitions before, and while there’s some question of whether Microsoft can make R easy to use, I think they can. Years ago, Microsoft bought Proclarity, a company that had some BI type products. Did they succeed with that acquisition? I think they did. A number of the technologies likely made their way into PowerPivot and Power View even though the Proclarity products have somewhat disappeared, being incorporated into other products.

    The R language and environment is complex, and I’m not sure how many database developers or DBAs want to become experts. However I do think that it could be possible to make the language easier, perhaps by building functions into SQL Server that help with the data analysis and computational features, and adding extensions in ADO.NET or other client libraries that might support easier rendering of visualizations.

    We have lots of tools to help here already, with PowerPivot and other related Excel add-ins, and I suspect that any offerings here will overlap with those products, but allow Microsoft to woo an even wider audience of data professionals.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.9MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Small Experiments in Data

    Amsterdam is trying to become a smart city, with technology and investments being made in ways that will (hopefully) improve the city living experience for residents. There are a lot of items mentioned in the article, but one of the main items that I think underpins the entire experiment is data.

    The piece notes that people and organizations can contribute to public data, which is then accessible by anyone that wants to build software or applications. The idea that groups can add to, and consume, data, is important. We need lots of data to better understand, and work with, our environment. We also need to ensure that the data is accessible widely otherwise it can be stifling to the innovations that people may dream about.

    Amsterdam is participating in the CitySDK, a project that tries to standarize APIs for cities. That’s important to avoid every environment having to reinvent their own APIs. This also means that software applications written by talented developers in one place might be easily ported to other locations. Ultimately, we might find that not only do our cell phones easily work in all locations, but specific applications that we have (perhaps for something like buses or grocers), might also easily port to other places.

    I also think this provides a lot of opportunity for those of us working in software. I can see that there would be lots of organizations that have ideas for applications, but lack programming skills. There may be lots of opportunities in the future for developers to work with those that dream of new systems by bringing an idea to life. Even if you donated your time, you could polish your development skills, your communication skills with non-technical people, and dramatically improve the impression your resume makes with potential employers.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Computational Biologist

    I saw this job in a NYT times piece and was intrigued to see the mention of a computational biologist at the Mt. Sinai Medical Center. This is a job that couldn’t have existed in past, as throughout most of history we haven’t had computers that could do work in the way we do it today. I suppose we could possibly have had large numbers of people performing these tasks, but I’m not sure that would have been practical. I’m not even sure it would be feasible as the amount of resulting data might have easily overwhelmed any attempt at organization and understanding by a few individuals.

    What I found interesting in the piece is that the “team does not do the basic science. … His group works on the computational pipeline. They try to improve the affordability and effectiveness of cancer treatments through automation. Their goal is to help scientists and doctors tailor treatments to an individual’s situation in a way that should increase the likelihood of a successful outcome.”

    This use of data science and analysis is tackling quite a challenge here. The treatment of diverse humans is challenging, where each person can be slightly different in their biochemistry. What’s more, these diseases are still not well understood, and this doesn’t allow for a simple pattern that can be easily discerned from mounds of data. However bringing more analysis to the problem could have huge rewards for people suffering in the world.

    I have hope that in the future data will continue to solve more problems like this and make the world a better place.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 1.8MB) podcast or subscribe to the feed at iTunes and LibSyn.