Voice of the DBA

Category: Editorial

  • Fixing P1 Queries

    When we look at the performance of software, we use Pnn notation to indicate the latency of an issue. A P95 problem is one that exceeds the time that 95% of the other queries take. In other words, this is the 5% slowest things happening, which can include database slowdowns that impact your application.

    For many years in software development, we have tended to work on the P95 or P90 issues, the slowest items. This is primarily because those can make a big difference to the system’s performance. If I fix the slowest things, then the system feels faster. Certainly, I know lots of DBAs and developers will apply this logic to database queries. They focus on the slowest queries and tune them to improve the system. If most things are quicker, especially the things most users notice, then the system feels faster.

    We’ve often limited our work on latent items because the time involved in working on other issues is hard. If you choose to work on items in the P70 range (30% slowest) or the P50 items, does that make sense? There are slower items, so doesn’t it make sense to focus on those P95 or P99 items when trying to increase the speed of the user experience? In software, I think this does make sense.

    However, in the database world, we aren’t often thinking of one user’s experience. We’re thinking about the volume of requests from all clients. We might find there are queries that seem fast, say 400ms queries, but if these are running thousands of times a minute, we might see more benefit from clients if we reduce this to 200ms than if we speed up a 30s query to 2s. For the database world, we often think about the rate of items, not just their latency (or speed).

    For database queries, how do we choose what to focus on if it’s not the slowest queries? we might not also want to tackle the most frequent queries either. My question might be: do we need to choose?

    In the age of AI, especially if tokens aren’t too expensive, do we need to spend engineering time deciding? Would it make sense to let our AI coding agent evaluate and propose changes to all the queries? With a team of agents, we might have time to investigate the entire workload and try to speed everything up. This does mean we need a good system for guiding agents, separating out work, and ensuring that each agent isn’t trying to add an index for every query. That’s something humans have done, and agents likely will try if they don’t have to coordinate their work.

    The effort of this work isn’t zero, and I’d bet there will be a significant amount of time to get this set up in some organizations, though I’d hope as people discover ways to organize agents for database work, with controls to scope their efforts and validation techniques, this becomes easier and easier. I’d like to think that at some point, DBAs will work in the same way software developers are hoping they work: thinking about problems, evaluating potential solutions, and managing AI helpers who type the actual code.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Building Your Own Software

    Lots of AI usage has been spent inside companies on building new software. Sometimes people are trying to rebuild existing software. I’ve seen more than a few articles say that Slack is dead (it’s changing), or Monday is dead, or some other SaaS isn’t going to survive because people can vibe code their own replacement. I know a few people trying to do this, with very mixed success.

    I saw this post from Jason Friend of Basecamp, where he noted that most products wouldn’t exist if everyone were an entrepreneur. This is because the great products exist precisely because an entrepreneur had a great idea and followed it through. They made mistakes, they learned, and they spent a lot of time getting the product right. The coding was likely a part of this, but the mistakes, the changes of direction, the decisions on what to add or take away, those are time-consuming things not shortcut by AI. The coding isn’t the big delay in lots of products. A great response to the post notes this: there are many decisions that go into a product. Many people don’t think about all those decisions.

    Certainly, some people will use AI and do a great job coding a replacement for some existing product. They’ll make their own version of a product, likely learning a lot from existing software. They will shortcut some design and testing time since they know what they want. It’s the software they already use, and they know how it works. They don’t need to design it.

    However, the software is just part of the process, especially in the SaaS world. Infrastructure and operations are big. While a replacement Slack might not need quite the level of hardware, networking, DR, disks, etc., it needs a lot more than an individual can build and manage. An LLM might help you determine where your bottlenecks are and help you build the systems needed to manage more resources, but it’s going to take time. Especially if you don’t know what sorts of hardware and bottlenecks will occur as more people use your software.

    It’s also going to make mistakes.

    Imagine your LLM gives you a piece of bad code that doesn’t perform. We can refactor and fix that. Now imagine your LLM doesn’t configure DR correctly. Fixing that is a little harder, especially if you don’t realize DR is lacking until an incident occurs.

    Software is hard. One thing I’ve learned over the years in advocating for customers’ requests in Flyway, SQL Prompt, or Redgate Monitor is that the coding takes time and effort, but much less than the decisions on what and how to build something. Those eat up time and focus. Once you solve these, AI definitely helps the rest go quicker, especially if you have AI building tests. However, until you solve the problems around coding, it’s not necessarily faster or better.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Make It Routine

    The first one is hard. The rest are boring.

    I heard this statement from someone recently, and it sounds like something a technical person would do. It’s been a goal of mine, or maybe a direction to aim for, though I sometimes think that goal is more aspirational than actual. It can be hard to do that for every task.

    The first time I tackle something, it should be hard. It’s new work. It’s a new process/code/thought/action/etc. It’s unfamiliar, and I spend more time on it than I want. After that, I’d hope I could repeat the thing again in much less time. That’s the goal, and that’s what we aim for in a lot of DevOps work. Make the things we think are hard, less hard. Make them boring by codifying things, using automation, and have the computer replicate the task.

    I think the second one is often hard as well. How often have you tried to reproduce something you did and can’t quite get it? Heck, I now depend (and use) SQL History constantly because I will write some code, change it a bunch and then realize that I can’t reproduce the version that did thing thing (or broke the thing) I was working on. I need to rewind things and figure them out again. Usually by the 4th time I’ve done something, it’s starting to become easier. By the 10th its boring.

    Unless I’m playing guitar, in which case, some things take a few more reps than 10.

    I’ve had plenty of developers say never repeat yourself. If you can automate it, you should. In practice, that’s hard. Sometimes I’m unsure of whether I’ll do something again, or often enough to spend the time automating it. There are also times I’m not sure it’s worth the effort. I spent a day once trying to automate a bunch of Outlook appointments, only to realize the whole Office API and deluge of information out there made this much harder than the 15 minutes a year I spend putting in all my Database Weekly reminders.

    I don’t want to discourage you from automating things and making them routine or boring. My database deployments ought to be routine. The daily checks should be so boring and automated that I don’t bother with them because I know the machine is doing the work and will let me know if there is something I should examine. The efforts to refresh dev dbs, or respond to audit requests, or even reset a password ought to be boring and easy. Some of us build ways to smooth these tasks and ease our jobs, and some of us treat every one as an ad hoc thing that we do over and over.

    The stuff you do over and over is going to be handled by AI at some point (maybe soon). Maybe soon, maybe in a few years, but a lot of simple stuff that you keep doing- that mindless, tedious stuff that doesn’t require a lot of thought – is going to be handled by AI agents. Either you’ll direct them, or your boss will ask someone else to do it after you leave. AI can handle things like figuring out when disks are full and cleaning out old log files, shrinking databases, archiving things, and then writing scripts (and scheduling them) to prevent issues. If that’s your job, your days are numbered.

    Make things routine by thinking about the pattern, how we could reduce or eliminate a lot of labor, and how we can use a computer to handle them. Even better, learn how to guide an AI to do that work and prove your worth.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Another Model, More Data Loss

    This past week I saw an article on eWeek that the newest OpenAI GPT-5.6 (Sol) model has deleted local files and live data. Files I’m less worried about, but data concerns me. There are lots of file backups, and certainly version control should be enabled for any developer tasks on which an AI works. However, data is harder, since it can change quickly, especially in live environments.

    There are different reports, some of which seem like more human error issues than the model’s, but we should account for human error when we use models. In one report, a model had access to a live production database and cleared tables for integration tests. Database testing is hard, as I’ve learned over the years. Many developers don’t think about how testing works with live data and how it is different from mocks and stubs. It is different, and if you use a lot of testing frameworks on live databases, you run the risk of there being issues. You could have data loss, and almost certainly will have some level of downtime disruption.

    This is a great reason why adding AI coding to database tasks needs governance around what is allowed. It is easy to have success with AI-assisted work and forget that these models can make mistakes. And if you trust them with too much, like you might be tempted with an impressive but junior DBA, you might get burned. These automated tools can get hyper-focused on a certain task, and not consider the ramifications of their actions. The article notes an agent ran “rm -rf” on a dev machine. That’s similar to what happened with Toy Story 2, where a human made the mistake. Both that person and this GPT-5.6 model weren’t thinking widely enough about what tasks they were managing.

    The article notes that this model can be overly persistent and assume actions are permitted unless explicitly prohibited. Not unlike a lot of humans who assume if someone has given them rights, they can do the task. OpenAI noted in an internal case that a user approved the removal of three VMs. When the coding agent didn’t find these, it looked for other machines and chose three other VMs. Crazy, but I’ve seen humans do this in the past.

    I don’t think these models are getting closer to Artificial General Intelligence (AGI) or that they are anywhere close to sentience. However, as they become more capable and handle more complex predictive actions, they are also likely to do the same stupid things humans do by not considering all the possible consequences of their actions. After all, we have plenty of data where humans haven’t considered the effects of their actions, and that data is in the AI model training set.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.