Voice of the DBA

Tag: AI

Adding Copilot to Azure Data Studio and a Few Experiments

As a part of a recent Data Exposed that I was on, there was an ADS update which mentioned Copilot being added. Since I’ve been experimented, I decided to give this a try.

This is part of a series of experiments with the ChatGPT and other AI systems. Lots of Copilot lately.

Installation

One of the easy things about VSCode and ADS is that you can install things from the marketplace very easily. You search and it appears, as shown here. Click Install to add it.

Once you do this, it’s enabled. I have access to the Copilot with my GitHub account, so I assume this picked this up.

Using Copilot in ADS

There’s a blog about what this tech is and a few workflows. However, I decided to try something first. I picked a table in this connection and started typing. Copilot added a suggestion.

It’s not a good one. That’s a table, not a function.

Yeah, some AI isn’t reading my schema and sending it back to GitHub. Boo, this isnt’ helpful. I can only guess that seeing “StringSplit” had the AI guessing this was related to string_split(). Something a new dev might think.

OK, let’s pick a table and see if intellisense works better. I’m guessing it doesn’t. Here’s a suggestion based on what I typed.

Not a bad guess. I have some schema in here, and there is an OrderHeader related to a Customer table. However…

There is no orderdetail. This clearly isn’t intellisense. It’s an AI taking guesses.

Trying Comments as Prompts

Let me try a few things from the blog. I’ll enter a comment.

I don’t like that. While I think this is valid for MySQL or PostgreSQL, it isn’t for SQL Server. Setting aside why LIMIT isn’t in T-SQL, why doesn’t copilot recognize this is T-SQL? Let me help it.

Better, but I don’t want to add “in T-SQL” to everything.

Another experiment. I tried a couple of prompts, but realized that asking for “orders” doesn’t mean that the AI knows I mean orderheader. Context around what or where an order is stored is something humans might include in natural language. However, I got this:

Which doesn’t work. I’d left the “in t-sql” from the previous prompt, so I’m not annoyed there, but this isn’t looking at schema. I also find the Oracle help not helpful.

However, this was interesting. I wanted a shipdate (this is a column) in 2023. The first suggestion was = Jan 1, but when I deleted that and hit enter again, I got this.

Not bad. I tried a longer prompt, with a few comments. All of the code was suggested a line at a time, and when I hit Enter, I got the next line.

The blog does note that adding the table create at the top of the query helps. Let’s try that.

I’ll add creates for customer and orderheader. I then try a simple prompt and get this.

Note, I tried “write a query”, but that didn’t work. I used “fetch” from the blog and that got me something. Though, not sure why this term is needed.

I also changed my prompt trying to get all matching orders and got this instead.

I also tried changing this to get matching orders. No idea why the customer name is limited in here.

What about aggregates? I know some people have trouble with these.

Not bad.

Conclusion

Adding Copilot to ADS is easy, but using it isn’t. The code it produces can have lots of mistakes, and not even work. It can make some strange assumptions, like junior devs might do.

Adding context makes things better, but for many of these simple queries, I have to describe things enough that I might as well write the query. Especially with SQL Prompt, I’d use SSMS instead.

However, learning to use prompts is a skill and it’s one to work on. Hopefully the AI will improve over time and I’ll start to see more useful suggestions, especially with different types of code.

So far, I’m impressed very little with this tool.

August 16, 2023
Copilot Experiments: A little PowerShell help
It’s been a little while since I’ve had time to relax a bit and try some AI help. This is another experiment I made.

A user on SSC asked about PowerShell to copy files with a date appended.

This is part of a series of experiments with the ChatGPT and other AI systems. Lots of Copilot lately.

I added some code to a new file and typed a prompt:

If I run this, it does work. Sort of.

It made a folder copy, not a file copy. However, the filter worked.

Let’s try again. I’ll modify the prompt and get Copilot to explain what it’s doing in the code. I get this:

Which works:

Hmmm, can it do what I want.

I tried a few prompts in the code window, but I kept getting things that wouldn’t work, like call copyfiles.bat, or something that didn’t work.

Let’s move on.

Copilot Chat

I got access to the Copilot Chat as part of Redgate. There is a new chat extension to add to VS Code, which I did. I opened it and got this with my prompt:

Good, this code will copy the files, but does all of them.

One advantage of AI bots is I don’t need to start over. I did this:

This worked correctly.

This was a simple example, but it produced about the same code as I did, albeit slightly cleaner. Mine was this:
```
$source="c:\fileloading" #location of starting directory
$destination="c:\filecopy"; #location where files will be copied to
$files="*dys_ihhist*" #files matching this pattern

# write a powershell command to get a list of files in $source matching the $files pattern
$a = get-childitem $source -filter $files
$a | foreach {write-host $($_.basename)-$(get-date -f yyyyMMdd)$($_.extension)}

# write a powershell command to copy files from source to destination appending the date to the filename
$a | foreach {copy-item $_.fullname $destination\$($_.basename)-$(get-date -f yyyyMMdd)$($_.extension)}
```
I don’t know enough PoSh to know which is really better. And honestly, I don’t feel like testing at scale. Let me know if you have knowledge here.

However, the chat window for copilot produced this quicker than I did, without me having to try and remember the PoSh parameters and structures of the functions. I had to dig around on SO to remember basename was what I needed and look up the parameter for get-date.

The code window isn’t great, and partially I think because I don’t know how to get prompts to work in the comments, but I do like the chat window. I’ll keep playing.
August 4, 2023
Constitutional AI

I will admit that I don’t know a lot about AI (Artificial Intelligence) systems and how they are built. I’ve been playing with them a bit and haven’t been overly impressed with the results. I think some of this is that the my work is creative and I’m both used to being creative and I find the AIs less creative. And less accurate. And require a lot of editing. I don’t mind editing, but not if it takes longer than just writing things myself.

From my understanding, a lot of the models behind AI systems (chatbots, recommenders, etc.) are built with humans giving them feedback on their responses in what’s known as RLHF (Reinforcement Learning from Human Feedback). Essentially paid (often low paid) people that help to “guide” the AI into responses that are useful.

I don’t quite know how that looks, and I certainly don’t want a job doing that. Definitely not if it’s looking at a lot of UIs like the one in this article. Can you imagine being paid to read things like this and then try to rank them? I can’t imagine they keep getting great input from evaluators across the day. Maybe 9am-10am, but I’d bet the 4pm-5pm responses are quick clicks.

There was an article about a company trying something different: constitutional AI training. There’s a better description on the Anthropic website. It seems in this case they are creating some principles and limited human feedback, but then relying on an AI to give feedback to another AI? Or itself? I have to admit that I’m not completely sure of what happens here.

Ultimately, I like the idea here, but I think the idea of a single LLM/AI model that suits every situation, or one that works in every geography doesn’t make sense. We have different thoughts among people and different cultures all over the world. I’d expect that we might have different types of AIs in different situations or environments. The one that helps decide how to deal with nuclear safety likely needs to be different from the one governing traffic signals. I certainly don’t want one TruthGPT to be the one true voice on all things.

The idea of AI systems, assistants and more seems complex and more strange than anything I’d have imagined from reading science fiction. As with many things, the reality is far different from the speculation I’ve had about how I would respond or want a system to behave. I think that’s the nature of science fiction; it picks specific situations and tailors the story to fit. The real world is much more messy.

I don’t know where we go, but I’m curious as many of you have had more exposure to AI. Is it helping? Hurting? Useful? Are you excited or worried for the future? I’m curious what you think, mostly because I’m not sure what I think.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

August 2, 2023
We’re Not Faster with AI

At Redgate Software, we’ve been trialing Copilot from GitHub with our developers. I managed to get access for this experiment and have tried a few things, though I’m not sure I’ve found it very useful. I’ll continue to work with Copilot, but for now, I just don’t find Copilot AI helping me with the types of tasks I do.

However, our chief of technology, Jeff Foster, was interviewed about how he sees AI and Copilot in Redgate. It’s a good read, and I found a few fascinating things in the piece. First, I was worried about this a bit, as we have explicit guidelines in developers that say we can’t take the output from an AI and put it in our codebase. We worry about legalities since we sell software. How does that work with Copilot?

Well, I learned that there are a couple of things in here. First, with a business subscription, we can have the AI not show us copies of public code. If more than 150 characters of public domain code is in a suggestion, it’s hidden. This is in the Copilot docs as well. I don’t know if 150 is a good length or too long, but at least there are limits. We also found that Copilot doesn’t retain any snippets, which is important for us. We don’t want to share Redgate code with others, especially competitors unless we are explicitly putting a public repo out there.

The second thing I found really interesting is that we have a lot of guardrails around the way we produce code. We have tests, lots of automation to evaluate things, and plenty of peer review. I see requests for PRs in Slack constantly amongst our teams. We know that the code which gets generated likely needs some editing, but also that we want to be sure that whether it’s sent as is or edited by a developer, the code is still reviewed. Having these guardrails, test, etc. help us experiment with new ideas without impacting our productivity.

Perhaps the most interesting part for me was near the end. Jeff doesn’t think that Copilot makes us faster. He says “We probably invest ten times more time finding the right problem to solve than on the actual coding.” That makes sense, given what I’ve observed over the last 15 years. We are deliberate and careful about picking problems to solve and producing high-quality code. Sometimes it makes me a little crazy as I want things to move faster, but I’ve learned to appreciate that overall, this is an approach that doesn’t get us into places where we struggle to onboard developers to new projects or adjust our code to meet new requirements.

I’ve got a few quiet weeks coming up, so I’m going to give in more and use Copilot a bunch to see what I can get out of it. If you’ve tried it, let me know what works or doesn’t. If you want me to try and solve some problem for you, drop a note in the discussion with the challenge you have.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

July 31, 2023