Voice of the DBA

Tag: big data

  • Do You Have Big Data?

    Data sizes are always growing. Stats on world data are astounding, as are the stats many of us experience in our lives. Plenty of us have moved from MB management to GBs, and I see plenty of people dealing with TB storage at home. Most of that data is likely from images and video, but I wouldn’t be surprised to find some people capturing lots of IoT and other random data about their lives.

    As data professionals, likely we don’t quite have the explosion of data inside our organizations, but we certainly do have increasing database sizes. I see this with customers all the time, often with continued or expanded collection of data and no archival plans. The number of customers with > 1TB databases increases every year.

    Overall data volumes are growing at a faster rate than our growth in processing capabilities. That is nothing new, as we’ve always collected more data than we can easily process. Even if you avoid image, video, and audio data, likely you have more CSVs, extracts, and export files than you know what to do with. Maintaining this data is hard, and dealing with all the associated systems is a challenge.

    I wonder today how many people think you have big data issues. No matter whether you deal with GBs, TBs, or something larger, do you feel that managing all your data is a big part of someone’s job? Can you analyze all your data, at least at the rate your customers would like? Can you keep up with queries and extracts?

    This article talks about some of the challenges, and how the problems with big data are going to to continue to plague us for some time. Does the article resonate with you? Or perhaps you feel that you have everything under control. I’d be curious in either case. Leave me a comment and let me know.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • What Happened to Hadoop?

    Six or Seven years back, Hadoop was the big thing. It was going to solve our big data analytic needs, it would provide cheap storage and query power with commodity servers. More and more companies were going to be using it. Microsoft invested in HDInsight, SQL Server got Polybase to query data directly from HDFS. I was seeing the Hadoop elephant everywhere. I still remember popping into a few sessions at SQLBits to try and learn a bit more about how Hadoop worked.

    In the last couple years, Hadoop has somewhat dropped off the radar as “the thing” that most companies need to become data driven and deal with large amounts of unstructured data. I found this analysis that talks about why Hadoop hasn’t taken over the world.

    The short answer? Real time needs, cloud computing, and containers. I think really, though, the complexity of Hadoop became a problem. It was too hard for most companies to deal with, and with too few willing to invest in the large infrastructure needed and skills required to manage a system. I’d say the same thing about Kubernetes, but it’s evolving rapidly to become easier, and it’s dirt simple in the cloud. I suspect we’ll see more Kubernetes cloud deployments than on premises.

    The other issue is the Hadoop batches jobs, which isn’t what many organizations want. They already deal with, and complain about, plenty of relational batch jobs. Whether this is ETL to a warehouse, cube processing, or some other delay. They want queries on data in place, which is becoming more commonplace all the time.

    Of course, one other important point from the piece is something I believe. The relational database, or data warehouse, is not going away. It’s still important to many organizations, and it’s useful to handle lots of reporting. With the growth of the SQL Server platform, you might even do more AI/ML analysis on your data in place, without the need to move it to an HDFS platform.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Big Data Analytics

    How large is your analytics system? Do you have more than one machine for analytics? Do you have a cluster of machines that run Hadoop in a YARN cluster to analyze your data? Are there hundreds, or even thousands, of nodes that are being used regularly? Some of you might have what you consider to be a large system, but I bet it isn’t as large as Microsoft’s cluster.

    They think they have the biggest YARN cluster, with over 50,000 nodes in a single cluster. This is used to process multiple exabytes of data from their various properties and systems. I certainly haven’t heard of a system this large, and I really wonder what this costs to run. After all, I’d think a 50,000 node cluster has to be a significant cost, though perhaps in the grand scheme of Microsoft’s $100 billion in revenue and $38 billion in expenses, even 100,000 machines can’t really impact their numbers.

    The cluster has essentially been running a private version of Azure Data Lake for years that their internal developers and analysts use to access a common pool of data. In fact, because of their scale needs and the desire to limit the copying of data between clusters, they have contributed back to the Apache Yarn project a number of fixes to help ensure the software can scale to tens of thousands of nodes. There is some discussion of how they’ve allowed YARN to grow to larger scales, and it’s an interesting solution that essentially allows some overbooking of resources, knowing there are always some spare cycles available for processing data. It’s a great test site for Azure Data Lake, and something that more of us might use in the future.

    I doubt may of us would need to work on data sets that large, and I know I certainly wouldn’t want to be responsible for that much of a data lake, I do think these are interesting problem domains that someone should look at. Certainly there are always large organizations and governments that have ever growing pools of data that will likely end up in a data lake of some sort. And who knows, perhaps, the definition of large will continue to grow to the point where 1,000 nodes in a cluster is considered “small”, and it’s what many of our businesses might implement in the future.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.0MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Fuzzy Processing

    This is really interesting. A computer chip that doesn’t exactly perform mathematical calculations correctly. That sounds crazy, and in some cases it is. Most of us want math to be exact, and work perfectly every time when we run calculations. Can you imagine if Excel didn’t always add up the totals in your expense report according to the rules of addition?

    However in more complex operations, such as pattern matching for images, we might not want exact calculations. If two radar images (one problem in the article) don’t match exactly at the binary level, would we want a search for similar images to discard one? After all, exactness and perfect matching are great in some areas, such as financial accounting, but in other areas, such as imaging, there is plenty of noise that isn’t important to the content of the image.

    Do we care? Perhaps. As data professionals, I suspect that we will get more and more data of disparate types, including images, that will become part of our databases. There will be clients the need to search and query this data, which means that we may need fuzzy search tools that work well.

    New search tools will likely mean that we will need to learn more about how to tune queries, or even indexes, that might deal with data in a way that isn’t exact. I suspect that there will be opportunities for those that learn how to deal with these types of problems in an effective manner. There might even be really good paychecks that come along with these jobs.

    It’s a bit disconcerting to think that we might want computer applications that don’t work exactly as we expect with regards to calculations. However, I think the science of uncertainty will become more relevant and valuable to us data professionals in the future as we deal with lots of data that is more complex than a simple addition problem.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.6MB) podcast or subscribe to the feed at iTunes and LibSyn.