Voice of the DBA

Category: Editorial

  • What Happened to Hadoop?

    Six or Seven years back, Hadoop was the big thing. It was going to solve our big data analytic needs, it would provide cheap storage and query power with commodity servers. More and more companies were going to be using it. Microsoft invested in HDInsight, SQL Server got Polybase to query data directly from HDFS. I was seeing the Hadoop elephant everywhere. I still remember popping into a few sessions at SQLBits to try and learn a bit more about how Hadoop worked.

    In the last couple years, Hadoop has somewhat dropped off the radar as “the thing” that most companies need to become data driven and deal with large amounts of unstructured data. I found this analysis that talks about why Hadoop hasn’t taken over the world.

    The short answer? Real time needs, cloud computing, and containers. I think really, though, the complexity of Hadoop became a problem. It was too hard for most companies to deal with, and with too few willing to invest in the large infrastructure needed and skills required to manage a system. I’d say the same thing about Kubernetes, but it’s evolving rapidly to become easier, and it’s dirt simple in the cloud. I suspect we’ll see more Kubernetes cloud deployments than on premises.

    The other issue is the Hadoop batches jobs, which isn’t what many organizations want. They already deal with, and complain about, plenty of relational batch jobs. Whether this is ETL to a warehouse, cube processing, or some other delay. They want queries on data in place, which is becoming more commonplace all the time.

    Of course, one other important point from the piece is something I believe. The relational database, or data warehouse, is not going away. It’s still important to many organizations, and it’s useful to handle lots of reporting. With the growth of the SQL Server platform, you might even do more AI/ML analysis on your data in place, without the need to move it to an HDFS platform.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Automatic Refaction of PII

    Data privacy and protection has become a hot topic. Across the last few years, as I’ve worked with customers that deal with the GDPR or other laws, they’ve become more concerned and careful with their use of data. I find less and less resistance from developers about using sensitive production data in development environments, but still too much.

    Using data that’s stored in databases or other text files is one thing. What about data in less structured forms? I’ve dealt with a few customers over the years that recorded customer interactions for various purposes. Call center or financial organizations commonly do this, and sometimes deal with sensitive information. I know I’ve given my date of birth, credit card number, bank account number, and other data to representatives at various times.

    Those calls are often recorded, and the IT staffs have often had to ensure extra security is applied to these files. Not everyone has access to listen for various reasons, but certainly when there is sensitive information inside the audio (or video), this data needs the same protection we’d apply to data in other forms. Providing protection, or redacting the information, isn’t an easy task.

    I saw recently that the Amazon Transcribe service will now redact some PII information automatically. This service can be configured to automatically remove the information in text, which is fantastic. This is a great way to start to use technology in a safe way to ensure that we have less data  leakage when we re-use data. Certainly people might look at this data to better train reps, but it’s also likely someone wold look through transcripts to determine why customers are calling in and use that information to better design applications. In either case, there isn’t any need to expose PII data to them.

    This doesn’t protect against the data inside the audio, but perhaps companies can delete and remove those recordings sooner with transcripts available and more quickly reduce their potential attack surfaces. We’ll always have some liability, but reducing that and not unnecessarily creating issues is part of what we want to do when protecting data.

  • Reviewing Your Past Efforts

    It’s about the start of a new year for me now. I know, it’s late February, and we’re almost a sixth of the way through the year, but I’ve been on sabbatical. In fact, with planning for the break, I’ve really missed almost two months of work at this point. I deliberately did not make any resolutions or planning at the end of 2019, since 2020 wouldn’t really start for my career until about now.

    Now is the time to plan, and I will do some writing on my blog for the things I want to tackle, but I also want to think about my plan. The main reason for this is the SQLskills newsletter from Jan 7 (#214), in which Paul Randal wrote an interesting editorial on the start of the New Year. There are a few things in the newsletter, but one of the items was about resolutions and planning for the new year, but by starting the process while looking at the previous year.

    I wrote about learning goals last year, and then blogged a bit about them throughout the first third of the year. I then pivoted a bit, with an outside commitment that ate up a bunch of my personal time. Despite having a reminder in my calendar, I never got back to the books I’d planned on reading.

    2019 Rating : Failure

    I’ll do more of a review on my blog, and then planning, but the important things I got from Paul’s note is that I didn’t go a good job of not only reviewing last year, but also I didn’t end up reviewing it periodically after a few months. Not only did my goal fail, but my review of my goal failed.

    There’s a lot to be gained from some self-analysis, and maybe even a review of your analysis by a spouse, partner, mentor, or close friend. In my case, I ended up doing more speaking and traveling than I’d expected, didn’t even complete my commitment, focused my reading time on fiction and escaping from life, and really ignored a bit of my career goals.

    I don’t know if I’ll do better this year, but just writing this and thinking about the process will help me do something different. Hopefully you are keeping up with your goals, but if not, maybe you need a mid year review, as I’m doing now.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Data Orchestration

    We continue to deal with larger and larger data sets all the time. In fact, it seems that most people find themselves outgrowing the capabilities of some of their OLTP databases, often RDBMS stores, and need to upgrade hardware or re-architect software. It doesn’t matter if you have a 100GB database on a few cores or 10TB on dozens of cores, there is often a need to upgrade to meet our workload demands.

    In addition to the transactional needs, there is a growing demand for reporting and analysis workloads. Some people use a separate warehouse, and some want to just query data where it is. Certainly ETL processes and platforms have grown tremendously over the last few decades for those that want to implement the former process, but there is plenty of demand for the latter. In fact, I’m amazed how many customers have inquired if Redgate’s SQL Clone product will enable them to do this and spread their workload to other systems (it’s not designed for this).

    I’ve been thinking that with SQL Server 2019 we will start to access data where it lives, not move it to another place we want it. To me, this is more of what future data orchestration might involve. I ran across an article that takes a slightly different approach, thinking AI and other products will help better move data around, and perhaps that’s true, but I do think more and more we want to query data where it lives, and use larger, distributed compute platforms to do this.

    The scale out capabilities of SQL Server 2019, with the separation of compute and storage in Big Data Clusters, is a huge change that I think will be the future for many of us that look to meet reporting needs. The ability to grow hardware to match the workload needs is huge. This alone is a good reason to think about doing this in a hybrid or public cloud scenario.

    Of course this doesn’t come cheap, easy, or quick. There is work to be done to evolve systems, but it is an area I think is worth experimenting in during the coming year. I bet many companies would be interested in some PoC work here to determine how to better meet the reporting requirements of larger data sets. Perhaps this is something you suggest to someone in your organization.

    Steve Jones