Voice of the DBA

Tag: AI

  • Self-Healing ETL

    The process of building ETL flows is cumbersome and time consuming. Many consultants have made a lot of money by building ETL processes that manage the movement of data in and out of databases, including to data warehouses. There is a lot of importance placed on consistent, quick, and accurate data loading. In fact, one of the most popular job needs these days is still for ETL developers. Even as data science grows and becomes popular, quite a bit of the work in data science is data preparation, which often involves ETL, or ELT.

    One of the struggles with an ETL process is the tedious nature of building lots of flows that are very similar. There may be a wide variety of ways to get data to move, especially as we have a crazy set of inconsistent formats in which data appears, but once we have the flow, we often repeat that for many different files or tables. I think Biml makes this better, but it’s still not simple, and there can be plenty of issues that still arise as the source inputs change.

    There was an interesting article that describes the use of AI to assist in ETL flows. While there is some basic matching up of source and target data right now, what if AI would add a few things. The idea of looking at the data you’ve matched and suggesting alterations that can potentially help with data quality is interesting. The system would have to learn what issues occur over time and what good data looks like, but perhaps this could even extend when the source data gets slightly mangled in formatting. Perhaps detecting fixed width fields that have grown and shifted all columns would be possible, and even adjust the data flow.

    I am actually more interested in detecting PII data and alerting developers, or even operations people that the ETL process needs to be changed or secured in some way. AI should be able as well to help Operations people with scheduling, even pausing or moving ETL work to other machines to better scale the flow. My view would be that any AI system ought to be looking for anomalies and detecting those, as pattern recognition is one thing AI is good at.

    I’d think this would be more of an alert capability as I’m not sure I trust an AI system to actually adjust data. ETL flows are often too important to have data mangled in an unexpected way. Perhaps that’s too cautious, as that’s what developers do all the time. They mandle data, and we find ways to recover. Ultimately, I think that having lots of logging on what a developer or AI system does is more important. This might be especially true if voice commands are used to build the flows. I could see lots of ambiguities and mistakes from business analysts that are trying to describe how to build an ETL flow.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 4.2MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • AI Regulators

    With the GDPR now being enforced in the European Union, there are plenty of companies that are getting concerned about the potential fines from regulatory authorities if they aren’t complying with the law, or at least, making an attempt. There certainly is leeway for regulators to adjust fines or give warnings if a company is making efforts to comply. This has likely contributed to the work inside many organizations to move towards compliance.

    There are likely some companies that might not worry, since there are relatively few regulatory employees and many companies. There are lots of complaints coming in, which could easily overwhelms the relatively small staff in each EU country. Complaints might not be investigated in a timely manner or even lost because of the workload. The problem will likely get worse as more consumers complain about data processing practices. I don’t expect regulatory authority staffing to increase, so I’m sure only the most aggregious or complained about companies will get caught.

    There is one way to help amplify the capabilities of the relatively small staffs reviewing complaints. There are researchers in the EU Institute in Florence that are are working with consumer organizations to create AI programs that can help by performing some of the work. The initial thrust is to evaluate privacy policies of companies. If there are issues, the software doesn’t assess a fine, but it does alert a human to perform additional checks.

    In one sense, this is exactly what computers can do well. They amplify the capabilities of humans by doing a piece of the work. We can build systems, whether traditional programmed ones or AI based applications, that handle a piece of the work that requires lots of human labor. Once initial evaluations are made, a human can review the work and make more refined judgments.

    The danger, to me, is that humans will be lazy. They’ll start to trust the AI systems as authorities and use less of their own judgment, mostly because it’s just easier. I could see these systems evolve over time to actually train humans involuntarily. New employees would initially trust the AI results, learning from the AI rather than teaching it and constantly evaluating its effectiveness.

    I think AI can really help improve the way that we accomplish work in many ways, but it should be regularly audited and approached with some skepticism. There certainly needs to be some sort of supervisory group overseeing the program that isn’t involved in the outcome. We should be sure that the goals and results from any AI system continue to be focused on what we want to achieve, and that we transparently define those goals for anyone impacted. Otherwise we might end up having AIs evolve in ways that are counter to the original purpose.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • AI Regulators

    With the GDPR being enforced in the European Union, there are plenty of companies that are getting concerned about the potential fines from regulatory authorities if they aren’t complying with the law, or at least, making an attempt. There certainly is leeway for regulators to adjust fines or give warnings if a company is making efforts to comply.

    Other companies might not worry, since there are relatively few regulatory employees and lots of companies. There are lots of complaints coming in, which could easily overwhelms the relatively small staff in each EU country. The problem will likely get worse as more consumers complain about data processing practices.

    There is one way to help amplify the capabilities of the relatively small staffs reviewing complaints. There are researchers in the EU Institute in Florence that are are working with consumer organizations to create AI programs that can help. The initial thrust is to evaluate privacy policies of companies. If there are issues, the software doesn’t assess a fine, but it does alert a human to perform additional checks.

    In one sense, this is exactly what computers can do well. They amplify the capabilities of humans by doing a piece of the work. We can build systems, whether traditional programmed ones or AI based applications, that handle a piece of the work that requires lots of human labor. Once initial evaluations are made, a human can review the work and make more refined judgments.

    The danger, to me, is that humans will be lazy. They’ll start to trust the AI systems as authorities and use less of their own judgment, mostly because it’s just easier. I could see these systems evolve over time to actually train humans involuntarily. New employees would initially trust the AI results, learning from the AI rather than teaching it and constantly evaluating its effectiveness.

    I think AI can really help improve the way that we accomplish work in many ways, but it should be audited and regularly approached with some skepticism by some sort of supervisory group. We should be sure that the goals and results from any AI system continue to be focused on what we want to achieve, and that we transparently define those. Otherwise we might end up having AIs evolve in ways that are counter to the original purpose.

    Steve Jones

     

  • Interviewing Avatars

    Imagine that you go to interview a candidate over Skype or even a phone call and the first thing you see or hear is: “Hi, my name is Hank, and I am an avatar. I will answer all your questions about Nick factually and completely, as if you’re talking directly to Nick. Are you still looking to fill this position?”

    That’s a quote from a blog post about HR and hiring issues. In this post, the author talks about a new system that allows a job seeker to use an avatar as a way to quickly apply for thousands of jobs. While this might be nice for the applicant, would any of you want to interview an avatar? Maybe you would a few times, or maybe you’d even start to like this as a way to get some basic information about a candidate. Of course, if the candidate has loaded up all sorts of technical facts from Inside SQL Server or How Would You Move Mount Fuji, then maybe you’re less interested.

    One one hand this is scary. On the other it’s awesome. I’d love to be able to answer some questions and provide some data about my career and goals and let an avatar go through initial screens. I wouldn’t want this to be used anywhere past the initial interview, but certainly it would help me to present a little more of myself in an interactive format. It could be a nice brand, and how fun would it be to interact with Steve’s avatar?

    In some sense, we might move in this direction. After all, this just levels the playing field between candidates and employers. Plenty of recruiting and HR people use their own software or algorithms to make quick decisions about a large number of applicants. I certainly know that you need to stand out with your brand and ensure your resume and online profile attract attention and this could help with that.

    Think about this. Is there a way to better present the data and information about ourselves, using technology to respond and help ensure that the position is a good fit for each of us? Hiring managers might use this, why not us? Actually, they are using Alex and Mya. Would this be a better way for you to control and show off some of your work? Even have the avatar show how you solved problems or responded to different situations? While this seems silly or science fiction, I wouldn’t be surprised to see this type of avatar-avatar interaction being used early in the process to decide if both parties should move forward.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Libsyn.