Tag: AI

  • Local Models Using GPUs with Ollama

    Can you use your GPUs when running a local model under Ollama? You can, and it really depends on how you run Ollama and what your hardware is.

    Ollama supports NVidia and AMD GPUs with some exceptions. You can read about their hardware support here. There are drivers required from the vendors, and configuration, but it can help with performance.

    Of course, GPUs aren’t cheap.

    I have an NVidia GeorForce RTX 2060, which is listed as being supported with a compute capability of 7.5. I need a driver version of 550+. I’m supported with driver version 560.94.

    2026-09_0096

    However, I need NVidia CUDA drivers installed for this to work. Those don’t really install on Windows, so I have to install them in the WSL subsystem, with the Linux install guide for this to work on my system. I don’t use Docker, I have Rancher for reasons …, so I haven’t done this so far.

    If you use Docker Desktop, there is native support for GPUs.

    On Macs, there is native support.

    If you are looking to run local models seriously, you’ll likely want either a dedicated machine, or you will go the Docker Desktop route as an individual. In an org, you might pick a dedicated server and allow multiple users to connect and get answers in a secure, controlled way.

  • Creating a Local Model Chatbot on Windows

    After my session at VSLive last week, I had a few questions from the audience. I’m adding some of these into blogs, and this was one:

    How do I run the docker compose to get a local chatbot?

    I didn’t include the dockerfile in a a repo, mostly because it’s simple, but just to make it easy, it’s in this zip: ollama-docker-compose.zip.

    To start this, I’ll assume you have Docker (or equivalent), but no images. Create a folder structure like this:

    2026-09_0091

    Put the docker-compose.yml in here. Then open this in a CLI and run: docker compose up

    You should see something like this. The images start pulling. This is slow, but once the download is there, it should be smoother.

    2026-09_0090

    Once the images are pulled, the containers should start and you should get some output in your CLI window. Something like this. Note the first part of the output is the container.

    2026-09_0093

    Once it’s up, it will first ask you for an admin account. Enter your credentials, as I’ve done. I didn’t use this email, but it doesn’t matter which one you does. It’s local!

    2026-09_0094

    You should then get the chatbot interface. Note, like me, the first time you run this, there’s no model, so use the post below to add your model and get started.

    2026-09_0095

    I have a few posts on this that might help. Note that these are from the past the the GUI has changed slightly, but the items are still in the GUI, just moved.

  • Running a Local LLM On Your Laptop Resources

    Thanks to everyone that attended my sessions at VS Live San Diego yesterday. It was a fun session and I’m glad a few people came up to ask questions.

    Slides are here: Running a Local LLM

    Other resources:

    I’ll write a bit more and I’ll answer some questions here next week.

  • A Daily Report from Claude

    Redgate had some Claude training recently, which I went through as my knowledge has been gained in bits and pieces and is fragmented. I wasn’t sure I’d get a lot out of it, but I was surprised by a few things I learned.

    One of these was scheduling a task to give me a daily report. This post looks at that process for me. I’ll walk through what happened, and then a summary below of what I think of this.

    This is part of a series of experiments with AI systems.

    Getting Started with Skills

    One of the first demos shown was to build a daily snapshot of what things need to be looked at and handled. The default in Claude is the /morning command. I ran this, looking to customize it, and Claude decided to create a new skill for me, based on this, which it named “morning-plus”.

    2026-08_0267

    I had previously added the Microsoft 365 Connector, so the skill could read my email. That’s good, as this is a read-only view for me, and it’s local to my machine. I’m sure there is some security risk, but this is managed and limited by Redgate, so I’m assuming this is good enough for my work.

    The next thing was to alter the view. I had a weird graph of lines that seemed to try and show the height of a graph implying the length of a meeting or a business of the day, but that’s not helpful. I decided to ask for a calendar list.

    2026-08_0268

    This gave me a nice look at my day, which is shown below:

    2026-08_0269

    Another thing I wanted to do is keep an ongoing TODO list as I have various things that aren’t in my calendar. A bunch of these things are repeating or ongoing, but having a list helps me stay organized. I might need a way to order them, but for now I asked Claude to add this. It guided me to using a folder to store this, which is what I wanted.

    2026-08_0271

    I also decided to keep the report here, as this helps me stay organized. I can keep the todo.md open in VSCode or elsewhere and edit as needed (or ask Claude). I like to do some editing myself, rather than waste tokens. If I have to type in a prompt, but not just edit the doc.

    2026-08_0272

    In the past I’d have tried to figure out how to schedule things. Here I just asked Claude. It did the work for me.

    2026-08_0273

    Once done, I have an item entered in my Scheduled area. This is a Claude thing, so Claude needs to run, which is fine. My desktop is running all the time.

    2026-08_0274

    I clicked the item and selected Run Now from the menu that popped up. This ran and went through all the tasks and produced a report. It took a few minutes, maybe 5? It’s slow, but not a lot of stuff going on when this will run for me, so it’s not a big deal. The report:

    2026-08_0278

    Not bad, but I also want to get anything from overnight that is important. I tend to look at Slack for mentions/links, so I added the Slack connector, which Redgate has authorized. This will let the agent scan for things that mention me and I can get a summary rather than searching through all folders.

    I added a prompt to help here, which is great.

    2026-08_0280

    That didn’t work, and Claude explained why.

    2026-08_0281

    Sad smile

    One note, each time I altered the skill, I had to approve the changes after Claude decided what to do. It’s a good idea, but the list of stuff it lists in a stream of text is too much to read, so I’m not sure how well this secures things.

    I also have to re-approve a changed skill to access my e:xxx folder on each run. I guess it’s good, but in a practical sense, I’m going to just approve this.

    The last thing I added was a Windows Scheduled task to run Edge and load this page. This was easy to do, and it will bring up a browser on my desktop that I can see when I log in. Now I can skip some of the checks I might do early in the am, knowing that I’ll get a view of my day when I get to my desk.

    Summary

    One of the interesting things the trainer noted was that when you start using a new tool, productivity can go down until you get the hang of it. This applies to many tools, but certainly Claude and AI chat/agents. It takes some time to see benefits, which is something I’ve noticed myself. Part of this is learning to use the tool, and part is deciding how you want to use the tool.

    In my case, I was skeptical this was actually a useful thing. My days can be chaotic and crazy. However, I’m willing to give it a try. I am starting to have more and more things that I need to keep an eye on, and things that span more than a day. I use calendar reminders constantly, but there are a lot of moving parts here. Keeping a todo list is a start, though I likely need to add some dates or priorities. I’ll play with this over time.

    The lack of Slack integration is annoying, but if I can keep other things organized, this will be helpful.

    Is this helpful and worth it? I don’t know. This is an experiment on my part, which I’ll run for 3-4 weeks to see if this is helpful in tracking lots of disparate work. I certainly forget some things and then end up working late, or miss deadlines.

    Is this worth the cost? Running a report costs $0.82 in its current iteration. Is it worth $1 to organize my day and prevent missed items? Perhaps. I’m not sure, but this is an experiment.

    2026-08_0279

    A small AI thing, but at $200 a year, this might be a nice productivity helper for me, partially to ensure work is done, and partially to reduce stress from the chaos of my life.