For a couple years, Big Data was heavily hyped, and Hadoop became incredibly popular. In fact, so popular that Microsoft build HDInsight and Polybase to allow us to take advantage of these technologies and integrate them into our own systems. While the year has seen less hype on “big data” specifically, more and more of us are dealing with large amounts of data every day. There isn’t a good definition of Big Data I’ve seen, but whatever you thought it meant five years ago has surely changed to mean larger volumes today.
One of the important things that many organizations are learning is that they don’t necessarily need more bits and bytes of all their data. They’re increasingly learning that they need more of the right data, which is the data that is useful to them. Often this is the data that lets them make decisions that improve their revenue, profits, efficiency, etc. As we move to GDPR this spring, it might also me more auditing data that prevents problems or satisfies regulators.
I ran across an interesting article that talks about companies needing the right data, which can often mean unstructured data outside of their traditional OLTP databases when dealing with customers. The article focuses on NLP (natural language processing) and social media data, but it could just as easily mean audio/video data from customer calls (or emails) or even sensor data from systems that are managed and track the ability of customers to use your product or service.
As the world of computing advances, many of us know that we need to find new and better ways to provide value to our employers. This might be with managing and gathering or tracking a wider variety of data, perhaps meaning that some of us need to keep some of those tweets or posts inside our systems. It could be that we need to provide new ways of analyzing data, maybe with some sort of ML (Machine Learning) or AI (Artificial Intelligence) processing. Perhaps it’s that we need to collate and collect detailed auditing information we can produce on demand to ensure our organization complies with legal requirements.
Perhaps there’s some other way that our work as data profressionals will change, but I’m sure it will continue to change and evolve across the next decade. I’m also sure this means lots of new and different data opportunities for us, if we’re willing to grab them with the right data.