Voice of the DBA

Category: Editorial

  • The Dangers of Dependencies

    Many of us working with databases know the problems of a single point of failure. We build HA/DR technologies into a lot of systems precisely because many of us know if the database goes down, a lot of stuff goes down. Broken software is easier to fix and rollback, but a broken database can be a much bigger problem.

    We also know an overloaded server doesn’t handle a workload well, hence our quest for well-written SQL code, but we often lose that battle with developers.

    In any case, as we move to a world where AI technology is used by many organizations, who often have a contract with a vendor to provide services, there is a potential issue. Imagine that you’ve setup workflows, maybe agentic loads and you depend on a company, say Anthropic, to provide those services. What if your organization gets banned?

    That happened to a company (reported on Reddit). A user got a note from Anthropic, but his entire organization got banned. That’s quite a dependency where a user in your company could cause an issue. In some sense, that’s like someone in your company sending an email that gets your organization’s email blacklisted or has Google/Microsoft/etc. cutting off access. Imagine the disruption there?

    Some of these companies providing AI services aren’t that large, and aren’t suited for the enterprise. Some of you are using vendors that might be contracting with these AI firms. Imagine your monitoring, DevOps, etc. service suddenly not working because they lost access to their AI services?

    I’d like to assume this doesn’t happen with the very large cloud vendors, but who knows. I’d like to think that not only enterprises, but even smaller companies don’t lose access because of the actions of one person. Or if they do, there’s any way way to get a response from customer service. However, I also know that Google, Amazon, and Microsoft have made it harder to get an answer from a real person.

    I didn’t talk about this recently when presenting on local AI models, but I think the future might be companies having more control over their AI tech, running them the same way we run servers in the cloud, in IaaS, with more effort required, but more controllable by the organization.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Who is Using CAGs?

    While talking to a customer a few weeks ago, they mentioned that they used Contained Availability Groups (CAG) everywhere. They also said they were amazing and wondered why everyone wasn’t using them in other environments. Of course, I questioned the “everywhere”, which turned out to be more of a default for new systems than a standard across all systems. That’s likely true of most things since it’s rare we get to update/patch/set something across an environment of any size and ensure every system is the same.

    Still, setting a CAG as a default makes some sense for enterprises. This ensures that in an HA situation I have my logins, jobs, etc. already on a secondary node. That’s been one of the challenges of using lightly linked systems that only sync up database level information. Log shipping, Replication, Availability Groups can all work to keep a secondary ready to take over, but they all miss information that is stored in master or msdb.

    That’s the stuff we have to sync manually. It can be done, but it’s work. We’ve had numerous articles at SQL Server Central on syncing logins and other objects outside of your database.

    Today I wonder how many of you are using CAGs in your environment? As the default for new systems? Moving all ones to this setup?

    Or do you even know about them? They are relatively new, since SQL Server 2022, and I have to admit I’ve heard relatively little about them in the community or from customers. Many people use Availability Groups, but not many seem to use Contained Availability Groups.

    Maybe another question is would you want to use them? There are a few things you have to consider and they can be slightly tricky, but they do some reduce some of the work when you have failovers. Of course, like any other technology, you need to test that your failovers work and you understand  the ins and outs of how they work, just in case that switch isn’t as smooth as you expect.

    It should be, but sometimes things break. If they do, you want to ensure you, or someone on your staff, knows how to fix them.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • A Tool is Better than a Script

    While working with a customer recently, I heard this sentence: a tool is better than a script. The reference was that this customer preferred a known, tested, approved tool for most of their staff rather than a script built, lightly tested, and perhaps changeable by anyone in their organization.

    I was surprised, because in many ways, I’ve depended way more on scripts, more often, than “tools” in my career. Often I struggled to find tools that actually worked in the way I wanted them to and built them myself with Unix shell utilities, VB Script, PowerShell, or some combination of those or other technologies.

    I asked them if they wouldn’t prefer to customize things and let one of their senior people write their tools. They said that it wasn’t necessarily a problem for a few people, but too many people might edit and change the script, even the senior people, and then others couldn’t depend on them. They weren’t sure if they were reliable if a known script wasn’t being run. Even among their senior people, someone would edit a script then they thought something wasn’t quite right, or there was a new requirement and then the script would return unpredictable results.

    I felt they didn’t completely trust their staff, but I also understand. I’ve changed my own “tools” and broken them or gotten back incorrect results. Depending on what the script did, and how busy I was, I might not even notice the problems. That might cause me problems if I expect data or actions to occur one way and they work differently.

    Scripts can be changed by anyone.  The results might not be what’s expected on a team. I get that. At some point in my career, I put all our DBA “scripts” or “tools” into a version control system and we had a path to the trunk (main) branch of them that was read-only, so that we knew what was being executed. If we wanted to change the script, we had to open an branch in the repo, make our edits, and then have another DBA approve the change before it would be merged into the main, and subsequently pulled into the folder in our path. This didn’t prevent changes, but it did give us some versions and history of changes.

    There’s a balance between flexibility and customization that is challenging to tread amongst team members. Sometimes a known, reliable execution is better than one that changes too quickly. We’ve been asked at Redgate Software to ensure people using Redgate Monitor can audit what actions are taken in the tool. Sometimes a team member changes a setting, such as an alert threshold, and other team members aren’t aware. That can impact their ability to diagnose problems when they assume the tool works one way, but it’s actually working another way.

    Tools we decide to use, whether purchased or OSS, won’t always do what we need. There is a requirement to build our own, but we also need some expectation and assumption of how they work. Especially in a team. I don’t want to have to re-read the code every time I run a tool; I should know how it works.

    At the same time, I can’t have every tool change each time someone wants something new. I love sp_whoisactive, but I depend on it working a certain way and wouldn’t want Grant to change how it reports data unless I know about the change.

    Working in a team requires teamwork, which means that whatever type of tools we buy/download/build, we need to ensure we all are aware of how they work.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Half of All Engineers

    The AI LLM boom seems to show no sign of slowing down. Each time I think we’ve reached some level of crazy use or predictions, things take another turn. I still find myself pinging back and forth between this will be amazingly good and horrifyingly bad.

    Sometimes on the same day.

    Today, I’m a little more down on AI. I was listening to Steve Yegge on the Pragmatic Engineer podcast, and they were discussing the curve of AI usage at companies. He points out that he’s mad that Amazon let 16,000 engineers go and might let more go. He worries that companies might let 50% of their engineers go. Not necessarily because the top 50% will be more productive with AI than 100% of engineers without it. Rather the concern is that companies will get rid of half their salaries to pay for the AI tokens for the other half.

    Steve Yegge is an accomplished software engineer that has worked at Amazon and Google. Steve wrote Gastown and has been someone who not only is successful at producing code but also thinks a lot about how we produce more software.

    How coordinated and powerful are the new models? Can they really do a lot of software work that we do today? Steve thinks so, but to be fair, he’s got a lot of experience and can architect and design software well, which means he can also guide AI LLMs and agents to write more code. He also thinks the latest models, like Opus 4.6, are way more capable that most people believe.

    I also caught this post on X about a paper predicting that AI might cause economic collapse as less knowledge workers are used in various tasks. This might happen faster than we can absorb those workers who are laid off in the name of AI back into the economy in other positions. It’s a scary thought.

    The positive side of this, at least from me, is that so many organizations move slower, and so many people aren’t extremely competent software engineers, so we’ll get a lot of bad software written by non-technical people that doesn’t scale. We’ll have more database (and application) performance issues, and that will slow the use of YOLO, vibe coding.

    Plus plenty of companies just aren’t implementing or looking to pay for lots of tokens. They’ll just move slower and the world will change, but not anywhere near the pace that Steve or others think it will. We’ll see, but let me know what you think.

    For a more positive spin, I’ve been reading Reshuffle, which is a little less depressing about the future.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.