Voice of the DBA

Category: Editorial

  • The Evolution of AI

    I saw a study recently where an AI system was used to analyze code and trying to decide if authors were good. The conclusions were things we’d expect, and quite a few people laughed about this on Twitter. After all, if AI comes to the obvious answer, is it useful? Perhaps, but it’s also a little disappointing.

    I’ve felt that way about AI for some time. Years ago I went through an AI demo for the Titantic data set, coming to the conclusion that the lower you were in the ship, or poorer, the more likely you were to have died. I went through a tutorial with Microsoft on flight data, trying to determine why flights were late. It turned out the later they left, the more likely they were to be late arriving.

    While this might seem silly, and perhaps obvious, I think that’s OK. We don’t expect these early AI systems to be better than humans, at least I don’t. However, what I do expect is that they can do at least as well as humans, and having them return results that we could figure out is a good step. To me, this also gives me confidence that a computer can be used to analyze a complex problem or situation.

    There is more evolution coming in the AI and ML worlds. I have no doubt that these systems will improve, and likely find niches where they are valuable and very helpful in making decisions. The world is gathering more and more data, and it’s becoming hard for humans to analyze it, and maybe more importantly, react quickly. The rate at which data can change is overwhelming, especially for humans. I think that we will need computing systems to help process the vast amounts of data we continue to accrue.

    While I don’t know that I want AI/ML systems making decisions for me, I do think they can help reduce the burden of looking over data and help humans to better focus their analysis in certain areas. A symbiosis of computing software using AI, data, and a human to closely watch it is what I think will really help us become even more efficient and focused in our particular organization.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • What Happened to Hadoop?

    Six or Seven years back, Hadoop was the big thing. It was going to solve our big data analytic needs, it would provide cheap storage and query power with commodity servers. More and more companies were going to be using it. Microsoft invested in HDInsight, SQL Server got Polybase to query data directly from HDFS. I was seeing the Hadoop elephant everywhere. I still remember popping into a few sessions at SQLBits to try and learn a bit more about how Hadoop worked.

    In the last couple years, Hadoop has somewhat dropped off the radar as “the thing” that most companies need to become data driven and deal with large amounts of unstructured data. I found this analysis that talks about why Hadoop hasn’t taken over the world.

    The short answer? Real time needs, cloud computing, and containers. I think really, though, the complexity of Hadoop became a problem. It was too hard for most companies to deal with, and with too few willing to invest in the large infrastructure needed and skills required to manage a system. I’d say the same thing about Kubernetes, but it’s evolving rapidly to become easier, and it’s dirt simple in the cloud. I suspect we’ll see more Kubernetes cloud deployments than on premises.

    The other issue is the Hadoop batches jobs, which isn’t what many organizations want. They already deal with, and complain about, plenty of relational batch jobs. Whether this is ETL to a warehouse, cube processing, or some other delay. They want queries on data in place, which is becoming more commonplace all the time.

    Of course, one other important point from the piece is something I believe. The relational database, or data warehouse, is not going away. It’s still important to many organizations, and it’s useful to handle lots of reporting. With the growth of the SQL Server platform, you might even do more AI/ML analysis on your data in place, without the need to move it to an HDFS platform.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Automatic Refaction of PII

    Data privacy and protection has become a hot topic. Across the last few years, as I’ve worked with customers that deal with the GDPR or other laws, they’ve become more concerned and careful with their use of data. I find less and less resistance from developers about using sensitive production data in development environments, but still too much.

    Using data that’s stored in databases or other text files is one thing. What about data in less structured forms? I’ve dealt with a few customers over the years that recorded customer interactions for various purposes. Call center or financial organizations commonly do this, and sometimes deal with sensitive information. I know I’ve given my date of birth, credit card number, bank account number, and other data to representatives at various times.

    Those calls are often recorded, and the IT staffs have often had to ensure extra security is applied to these files. Not everyone has access to listen for various reasons, but certainly when there is sensitive information inside the audio (or video), this data needs the same protection we’d apply to data in other forms. Providing protection, or redacting the information, isn’t an easy task.

    I saw recently that the Amazon Transcribe service will now redact some PII information automatically. This service can be configured to automatically remove the information in text, which is fantastic. This is a great way to start to use technology in a safe way to ensure that we have less data  leakage when we re-use data. Certainly people might look at this data to better train reps, but it’s also likely someone wold look through transcripts to determine why customers are calling in and use that information to better design applications. In either case, there isn’t any need to expose PII data to them.

    This doesn’t protect against the data inside the audio, but perhaps companies can delete and remove those recordings sooner with transcripts available and more quickly reduce their potential attack surfaces. We’ll always have some liability, but reducing that and not unnecessarily creating issues is part of what we want to do when protecting data.

  • Reviewing Your Past Efforts

    It’s about the start of a new year for me now. I know, it’s late February, and we’re almost a sixth of the way through the year, but I’ve been on sabbatical. In fact, with planning for the break, I’ve really missed almost two months of work at this point. I deliberately did not make any resolutions or planning at the end of 2019, since 2020 wouldn’t really start for my career until about now.

    Now is the time to plan, and I will do some writing on my blog for the things I want to tackle, but I also want to think about my plan. The main reason for this is the SQLskills newsletter from Jan 7 (#214), in which Paul Randal wrote an interesting editorial on the start of the New Year. There are a few things in the newsletter, but one of the items was about resolutions and planning for the new year, but by starting the process while looking at the previous year.

    I wrote about learning goals last year, and then blogged a bit about them throughout the first third of the year. I then pivoted a bit, with an outside commitment that ate up a bunch of my personal time. Despite having a reminder in my calendar, I never got back to the books I’d planned on reading.

    2019 Rating : Failure

    I’ll do more of a review on my blog, and then planning, but the important things I got from Paul’s note is that I didn’t go a good job of not only reviewing last year, but also I didn’t end up reviewing it periodically after a few months. Not only did my goal fail, but my review of my goal failed.

    There’s a lot to be gained from some self-analysis, and maybe even a review of your analysis by a spouse, partner, mentor, or close friend. In my case, I ended up doing more speaking and traveling than I’d expected, didn’t even complete my commitment, focused my reading time on fiction and escaping from life, and really ignored a bit of my career goals.

    I don’t know if I’ll do better this year, but just writing this and thinking about the process will help me do something different. Hopefully you are keeping up with your goals, but if not, maybe you need a mid year review, as I’m doing now.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.