Voice of the DBA

Tag: AI

  • The Role of Databases in the Era of AI

    I’m hosting a webinar tomorrow with this same title: The Role of Databases in the Era of AI. Click the link to register and you’ll get some other perspectives from Microsoft and Rie Merritt.

    However, I think this is an interesting topic and decided to try and synthesize some thoughts into an editorial today, partially to prep for tomorrow and partly because I’m fascinated by AI and how this technology will be used in the future.

    The title says the role of databases, not data professionals. You might worry an AI is going to take your job as a DBA or developer, or you might think there is no way an AI can do your job. I tend to think the latter, but only if you are above average in your role and you add value by understanding your employer’s business. In those cases, the AI will help you (as a co-pilot, not a pilot) and allow you to get more work done or work done faster. You choose. If you churn out average, or below-average work, or cut/paste from Stack Overflow or SQL Server Central or anywhere on the Internet, then yes, you should worry.

    Databases store lots of information, and extracting that out is hard. I see no shortage of poor data models, no shortage of overloaded data in fields, de-normalized structures, repeated information, and more. Humans jump through lots of hoops to build reports or screens or other interfaces to present to humans looking for answers. We may load join data in Excel with values in a database or vice-versa. I’m sure many of you have plenty of stories on how you get data to move between some data store and a text format. I’m sure you also have no shortage of frustrations from your efforts.

    AIs will get good at this. At the Small Data 2024 conference, I saw many people working at using AI without a semantic layer, which I think is possible, but will likely fail. We store data in too many crazy ways, and companies will need to make it easy for customers to create a semantic layer that describes what data is stored in each place. They’ll also get the AIs to help not only with this but with creating a way to simulate Master Data Management without requiring every application to use Redgate Software, Inc. as a name. We need to ensure Redgate, Red-gate, Redgate Software, and RG stored in different fields can all joined as if they were the same value. Which they are.

    Fuzzy matching is the domain where AIs can shine, as the models can do this quicker than humans, without getting annoyed and with fewer mistakes. AIs can adapt with our feedback as we find ways to train the models better and overload the AI prompts with semantics that help translate the (extremely) poor data models in our databases, data lakes, spreadsheets, and even PDF documents. Companies that require a semantic layer can ease the process of building one with AI assistance so that customers can quickly start to query their wide array of data sources.

    The best use I’ve seen for AIs is as an easy-to-use, context-aware, powerful search engine. When we learn how to tune these for specific sets of data, such as all the datastores and spreadsheets in a company, we’ll start to see some amazing gains in information analysis. I don’t know that humans will analyze any better than they do today, but the process of getting the information to analyze will be easier. I think AIs will also help in the analysis phase, but that’s going to require more co-work between humans and AIs to improve the quality of analysis.

    There are other things, but I see databases as incredible stores of information that AIs will make easy to access. I’m also positive AIs will be used to more easily update information in databases and assist in easily moving data from one format to another or one location to another.

    Tune into the webinar tomorrow and see what Microsoft thinks and ask any questions you have.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • AI Is Great and Tech is Failing

    On a recent weekend, I got a text from my bank that they had declined a charge to one of my business accounts. I called them back and they let me know there had been a couple of weird charges on the account that their AI system detected. This seems to happen every year or two so I wasn’t overly worried. I cancelled the card and ordered a new one.

    A day later, my wife got a call about our credit card with the same issue. She cancelled the card and got new ones ordered. However, I use that card to travel and I had a trip booked. Suddenly I was without a credit. Luckily, we have another card for my wife’s business that I could use. I called the bank and had a card expedited, but the situation created some stress. In fact, I panic-bought an RFID-shielded wallet. I’ve resisted for years, using an older, large wallet me daughter bought for me one Father’s Day that always reminds me of her. The timing across a few cards was weird, and I suspect my wallet got scanned somewhere and both card numbers were stolen.

    A few things. First, be careful with the new tap cards, as they can be scanned and read from a distance, albeit a short one. Second, having a spare payment method might be nice in this age of non-cash transactions. Third, why is technology failing with new cards?

    I lost a card last year and knew it was gone. There were no charges, but I couldn’t find it and needed a replacement, so I cancelled it and ordered a new one. In minutes the digital cards on my phone (and watch) had been replaced. I had new numbers and could transact business.

    Why would this be different? The banks arguably have better knowledge of my digital wallets, and replacing those is much easier than relying on a snail mail server and the time it takes to deliver cards. In my rural area, we regularly have reports of stolen mail, with thieves targeting credit cards and physical checks sent by snail mail.

    This was a minor issue in my life, and I am fortunate I have other ways to manage payments in this minor crisis. I was (and am) happy that AI systems are often detecting fraud. I haven’t had any fake charges go through in a decade and almost every real charge is approved, even with my crazy travel schedule. However, I’m disappointed in technology in this case.

    Many organizations are engaged in a digital transformation. They’re hiring software developers and trying to take advantage of all the data they have to improve services and efficiency. Security, service, and spending would be served better with a little technology improvement here.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • AI Sings the Blues

    Quite a few of you aren’t thrilled or enamored by GenAI (Generative Artificial Intelligence) with things like ChatGPT or CoPilot. Some of you love it or find it fascinating. My guess is many of you are like me and you’re not quite sure how useful this is with a healthy dose of skepticism.

    No matter how you feel, I find this piece on AI Blues fascinating to read. It talks about some of the problems with AI and how people are becoming less tolerant of the small errors that AIs make. I think people might be upset with some of the issues, but I also think these are types of things we might see in a colleague as well, and we might tolerate them for much longer than we think.

    The thing in the piece that really caught my eye is that training models to make less errors might make them less creative. I don’t think they are very creative, but I do think part of the idea is that they might do something better than we could do, at least for those of us working in a space where we’re a beginner to intermediate (or advanced beginner). In many cases, we’d expect better output from a model.

    The note about training not being often or in-depth enough as well is interesting. Perhaps some of the models ought to be trained separately, with some having a more specific focus, like learning Python and producing very few errors, or learning Python to produce interesting solutions to large scale problems even if there are some errors in the code.

    I could see similarly more tightly trained models for other areas, such as marketing email or legal issues. Maybe we need a lot of different models tailored for different jobs. Of course, if companies are struggling to generate revenue on AI already (and I think they are), then can they afford to train and retrain specific models focused for smaller audiences?

    Maybe, and maybe Moore’s law will help bail out some AI experiments, but I’m not sure. For now, I continue to experiment, and I think skilled tech people ought to do the same. I don’t think it will replace us, but I do think it can be a useful tool in many ways to help us become more productive.

    Steve Jones

  • Expanding Northwind with Prompt+

    I was trying to do a little testing of our subsetter and needed a larger database. So I decided to ask Prompt + to help me with a little AI assist. This post looks at what happened.

    This is part of a series of experiments with AI systems.

    Getting More Data

    I had a Northwind_FullRestore from my experiments with Alex Yate’s quick PoC tool. I selected that database and asked for a little code. I asked the AI to give me 1000 more rows of data in the Customers table. I got this code.

    -- This script inserts 1000 rows data entry into the dbo.Customers table
WITH numbers
AS ( SELECT TOP (1000)
             ROW_NUMBER () OVER (ORDER BY
                                   (SELECT NULL)) AS num
      FROM sys.all_objects)
--INSERT INTO dbo.Customers
--  (CustomerID, CompanyName, ContactName, ContactTitle, Address, City, Region, PostalCode, Country, Phone, Fax)
SELECT
   'CUST' + RIGHT('0000' + CAST (num AS nchar(4)), 5)
, 'Customer' + CAST (num AS nvarchar)
, 'Contact' + CAST (num AS nvarchar)
, 'Title' + CAST (num AS nvarchar)
, 'Address' + CAST (num AS nvarchar)
, 'City' + CAST (num AS nvarchar)
, 'Region' + CAST (num AS nvarchar)
, 'PostalCode' + CAST (num AS nvarchar)
, 'Country' + CAST (num AS nvarchar)
, 'Phone' + CAST (num AS nvarchar)
, 'Fax' + CAST (num AS nvarchar)
FROM numbers;

    Then I decided to go big. I changed my prompt, as you see below, and got more data. My new prompt: expand this to add 10GB of data to this table

    I got this code:

    -- This script adds approximately 10GB of data to the dbo.Customers table
-- Firstly, we need to calculate the approximate size of each row. 
-- For simplicity, we'll assume each row is ~1KB in size after considering nvarchar data expansion.
-- Therefore, to add 10GB of data, we need approximately 10,000,000 rows.
WITH numbers
AS ( SELECT TOP (10000) -- This will generate 10,000 rows in the CTE
             ROW_NUMBER () OVER (ORDER BY
                                   (SELECT NULL)) AS num
      FROM
        sys.all_objects a
      , sys.all_objects b)
INSERT INTO dbo.Customers
   (CustomerID, CompanyName, ContactName, ContactTitle, Address, City, Region, PostalCode, Country, Phone, Fax)
SELECT
   'CUST' + RIGHT('00000000' + CAST (numbers.num + seqnum AS nvarchar), 8) -- 8 character CustomerID to accommodate large number of rows
, 'Customer' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Contact' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Title' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Address' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'City' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Region' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'PostalCode' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Country' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Phone' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
, 'Fax' + CAST (numbers.num AS nvarchar) + CAST (seqnum AS nvarchar)
FROM
   numbers
   CROSS JOIN
     ( SELECT TOP (1000)
              ROW_NUMBER () OVER (ORDER BY
                                    (SELECT NULL)) AS seqnum
       FROM sys.all_objects) seq;

    The Results

    The results were a bit underwhelming. I ran it and got this:

    8-2-2024-0005

    The problem here is that the CustomerID is a 5 character field. The comment in the code notes that this is now an 8 character field, but that’s not great as I might not want to change the size of the field.

    I then asked this: change this script to ensure the customerID is less than 5 characters and alphanumeric

    I got the same script, but the first line of the select was:

    'CUST' + RIGHT('00000000' + CAST (numbers.num + seqnum AS nvarchar), 8) 
-- 8 character CustomerID to accommodate large number of rows

    Now when I run it, I get a problem with postal code, which now has the truncation error.

    My Thoughts

    It’s easy to say this was a failed experiment. I didn’t get working code. But I got a bunch of code that was close to what I need, in a fraction of the time that it would take me to write this, even with SQL Prompt. Then add in the fact that I can edit this code to what I need, which works, and saves me times.

    I think this has potential for shortcutting some work and getting me closer to what I need quickly, even if it’s not perfect. If I’d have asked a junior dev to help me with this, I might still have to edit their code. Just as I do with my AI assistant.