Voice of the DBA

Category: Editorial

  • Preserving Data

    Most data professionals I know go out of their way to take care of the data entrusted to them. Most people ensure backups are running, lots (hopefully most) test their restores. A few will ensure a good rotation their data offsite. Some of you might have formal rotation schemes, and some might just keep a rolling list of xx backups available.

    Likely a few of you don’t worry about anything other than the last full backup, which is a risker approach than I’d take.

    In the past, I’ve often kept around monthly backups for a year, usually with more granular backups inside of the month. Beyond that, depending on the system, we might keep quarterly backups or yearly ones for a longer period. These days with cheap storage and automated solutions, I’m sure many of you just assume your backup system keeps xx backups around. the cloud providers will keep point in time backups if you use Azure SQL DB (7 days by default) or AWS RDS (1 or 7 days, depending on the provisioning method). If you use VMs or other systems, you ought to be aware of how long backups will be kept and ensure that fits inside your RPO.

    This came to mind as PBS, a US broadcaster, almost lost most of their archived data recently. Their provider stopped responding to them and they couldn’t get to the data, which was stored in Iron Mountain data centers. They sued and got access to ensure they could access and use this data. A win for them, and really, for many of us.

    I don’t expect Azure or AWS to go under, but some of you are using third-parties for backups, even in the cloud. If your provider goes under, can you access your data? I can guarantee if any of these companies has issues (financial, hacking, ransomware, etc.), there will be a few clients that need a restore that day. Resolving this in the courts is likely to be successful, but after how many days?

    Backups are important, but only when they facilitate a restore. It’s easy to get complacent and think because you’ve been running backups that you’re protected. Test that restores work. Have a plan, and maybe a contractual clause, that ensures you can get to backups in the event of issues. This won’t protect you from everything, but it can help speed things up.

    And if you’re like me, keep at least the last backup (or two) local to your system, in a place that you can get to quickly and easily. It’s fine to assume the automated PIT backups will be there, but if AWS or Azure has issues on the day I need a restore, I’d like to think it’s worth a small cost to me employer to ensure there’s at least one backup separate from the automated service. That SLA refund isn’t likely going to cover the lost business when your database is down.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Collecting Data is Hard

    Data is the lifeblood of much of the world today. Not necessarily big data, and certainly not perfect data, and definitely not just digital data. Organizations, individuals, governments, really everyone out there are making decisions based on data. You might think it’s going to rain, so you cut the grass today, or maybe defer adding fertilizer. Your organization sees demand for a product increase, so it orders more and produces more. Government is always using data to make decisions about resource allocation. We might not think governments make great decisions, but they do use data and data matters.

    Recently I was reading a science fiction book (I, Starship) about the future, where a person’s brain (Henry) becomes uploaded to manage a starship. This ship will travel light years away for 80 years and they need the human crew asleep in hibernation to survive the journey. The interesting thing, to me, was a part in the book where there is a discussion of why Henry was uploaded and why AIs aren’t advanced enough to run the starship. There’s this quote: “The first generations (of LLMs) performed well, but as time went on, we entered a situation where more and more of the data available to train them on was itself machine-generated. So, instead of mimicking high-quality human output, the outputs got more garbled.”

    I worry about this as the current models are sucking up so much data to learn, but so little of the new data is being generated by humans. We already see plenty of AI-slop on the Internet, with fewer and fewer articles, blogs, etc. being human generated. I’m sad because people don’t share as much of their own thoughts, knowledge, etc. This is especially true in light of the AI companies taking individuals’ work for training without compensation. Indeed, I worry that many places will go the route of Stack Overflow, where they essentially fail.

    I’m worried about that here at SQL Server Central, as I see less questions being asked by humans and less discussion about the nuances of database challenges.

    However, there are AIs out there also polluting the world. This was a piece from last year that more AIs are taking surveys and polls. Reddit is seeing questions being asked, and I’m sure there are AIs answering them. How long before the amount of AI generated traffic dwarfs human generated traffic? I mean new data, not consumption. I expect plenty of humans are going the way of the people in WALL-E and just consuming data. They’ll continue to watch untold numbers of reels, shorts, Tik-Toks, etc.

    Are we going to see less “real” data and more generated data? I already have seen no shortage of issues from customers trying to use synthetic data for testing. It doesn’t match the real world well, but if they stop getting real data from customers and more from other bots, maybe it won’t matter. Of course, I’m not sure how well their systems will perform in the real world.

    GIGO is a real issue, and I expect a lot of companies will learn this as the volume of AI-generated data increases.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Admin Rights for Everyone

    I was chatting with someone that works at a smaller organization Still a few hundred employees, but the technical teams (dev and ops) were less than 20 in total. They mentioned that everyone had admin rights to the systems as they worked as a team and sometimes developers provided production support.

    I haven’t encountered that in quite some time. Is it still a thing to give a lot of people administrator writes across many systems? I know for many organizations there is concern about developers being able to change things in production, but if you aren’t a public company or a regulated one, then Sarbanes-Oxley, HIPAA, PCI-DSS, or other restrictions don’t apply. In those cases, if you have a tight team that functions together, would you be worried about this practice?

    My perspective is that I am worried, and I’d still want to restrict production access to a few. I might allow developers to merge code and approve pipelines to run, but I’d want to ensure there are audit trails. Ideally, I’d even restrict DBAs and others from using their credentials and force them to use pipelines, but I know reality. In the moment, during a crisis, they might need access in a quicker way that allows interactive work.

    Sometimes production issues are hard to diagnose without being on the actual system.

    What I might want to enable instead is a specific account (or a few) for sysadmins that can be used for production access, but with an extended event trace limited to capturing just their actions and all their actions. This wouldn’t trigger for most activity, but it would if an admin accessed the system. In my mind, this is less about a worry of malicious activity by an admin and more a way to ensure log all actions so we can troubleshoot mistakes.

    I’m sure none of you make mistakes in a crisis, but I do. For my own safety, I’d want a record of my actions.

    I might even set a policy of screenshot recording as well. Many of us work in SSMS, and it’s easy to forget if we ran a query, or what the results were. SQL History in SQL Prompt saves me often if I forget what query I ran, but it doesn’t capture results. If I’m running scripts, whether DDL/DML or clicking in SSMS, I would like a record of what happened. An audit trail we can review.

    I do try not to click things in SSMS in production, and instead copy/save the scripts and then run them. It’s a better habit, but in a crisis, I know I might forget, as would others, so putting a system in place to capture actions is helpful. Recording your screen is an easy way to do this.

    Admin rights widely distributed have been shown to be a bad idea, especially in the era of ransomware, social engineering, etc. However, some entity needs them, so try to ensure you have good governance around actions taken. Just in case someone makes a mistake.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • A Challenge of Our Knowledge

    AI is here to stay. It will evolve, it will get better at some things, and we might decide that it’s not good for certain tasks. It’s a weird, new, different technology that somehow seems magic, extremely intelligent, and at times as dumb as a box of rocks. It can do things that I could never do, or would never do, for myself. Heck, I’m not sure I could or would pay someone to do this by hand. Yet this was less than a minute for a computer system to take this image and transform it into something fun.

    Christian Buckley wrote an interesting post about AI challenging our identity, which sums up nicely one of the struggles many of us have with AI. Many of us identify with our work. We spend most of our lives for decades toiling away at a craft that we (hopefully) enjoy and in which we have success. We build skills, and we’re proud of our accomplishments.

    Some of us are more proud of our scars.

    Either is OK, but AI challenges that. AI can do work in seconds that we used to take minutes, often tens of minutes. Sometimes hours. It can remember things that we spend time googling or looking up in SQL Server Central forums. Our ability to search and navigate docs for an obscure setting, like that strange exit code you found in your CI/CD pipeline. We’re proud of where we’ve been and what we’ve done.

    I wrote recently about experts wanting to still solve problems themselves, without AI assistance. Some people don’t embrace AI because they think it devalues their knowledge. Others are afraid of the technology and potentially making mistakes with code an AI wrote that they don’t understand. They see this as a risk (and they should).

    However, choosing not to use the technology at all, or not trying to learn how and when to use it, is a mistake. We need to embrace the tools in our world, learning to take advantage of them.

    And more importantly, show our current and future employers we can do so.

    AI does challenge us. It challenges the way we used to work and some of the skills we used to rely on. We have to learn to flow with this challenge and make it a part of our future career.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.