Voice of the DBA

Tag: administration

  • The Case for Patching

    Recently I was testing a feature in SQL Server on 2017 and 2019. There was supposed to be an improvement across versions, but I didn’t see it. Then I realized that I was on SQL Server 2019 CU 2 on my laptop, and the current CU is 17. I took a few minutes to download that and install it.

    I have often been a lagging patcher in production environments, often looking to stay a CU or two behind, depending on my workload. SQL Server has been a very security -table platform, so that’s often worked well, though there are security updates at times. For those, I usually prioritize a patch getting applied.

    Windows (Linux, MacOS, etc.) tends to get patched more often than other software, especially by administrators. At least on desktops. Servers sometimes lag a bit, which can be a problem. I saw this week that a lot of attacks in 2022 Q2 were for a vulnerability Microsoft patched in Sep 2021 but hadn’t gotten all their customers to apply the patch. That situation was a problem early in my career with many vulnerabilities, and it’s still apparently an issue now.

    If you’re wondering how big a deal patching can be, remember the Equifax hack? This occurred because administrators hadn’t patched a system. Whether it’s a host OS, a database platform, or some other software system, it’s important that you keep somewhat current with patches. We never know when vulnerabilities will appear, and honestly, for most of us, we can’t spend the time tracking every piece of software and the various vendor disclosures.

    We can, however, patch relatively quickly. While I don’t expect that most, or even many, people will patch within a month, I do think that delaying six months is probably a bit long. That being said, I need to check a few of my servers and make sure my admins are keeping them up to date.

    Steve Jones

  • The Useful Cloud Tags

    One of the things I’ve seen in working with cloud based resources is that you get a lot of them in your account quickly. A database might end up with a server, an IP address, a network, security groups, and more. A few clicks of the mouse in the Azure Portal can create a new Resource Group that doesn’t just have the one thing you need, but 3, 4, or more other items.

    In addition to the quantity, there are also the problems of namespaces in cloud resources. Some of the items you provision are publicly named in a domain, such as database.windows.net. In these cases, the resources need to be unique among all cloud customers. Just like domain names, this means that you might have collisions with your favorite name. While I might like jones.com, there are a few other people that would likewise prefer this. The rest of us might have to choose jones2.com, stevejonesincolorado.com, or some other variation. In large organizations, you might end up with LA34532345454.database.windows.net for an Azure SQL Database.

    That means that the names of the systems don’t make sense to anyone, and many of the people that need to use or manage them will not even know which resource belongs to which system. This has been true for servers in many organizations for a long time, even with their own domains. Often there is some document, perhaps stored on the root of the machine or in an online share, that provides more information for people accessing the system.

    The cloud makes keeping track of systems harder, and there is a greater need to classify, categorize, and tag resources. Most of the cloud providers have built extensive tagging systems that help users add metadata to various resources and search/filter by these tags.

    Today I’m wondering which tags are useful in the cloud? How do you decide which tags, or type of tags, to apply? You might choose to apply an application name, a business department or owner, or some other type of information that helps you keep track of which resources are needed and how to deal with them. Is there some guideline on the type of key-value pairs you use for tags?

    I also would you want to easily have tags for on-premises SQL Server instances? We do have extended properties, but those are cumbersome. Would you want some sort of easy system query that retrieved all tags. Perhaps something like @@InstanceTags or @@DatabaseTags that retrieved all the data you’d stored in extended properties.

    We are only seeing more and more resources that need to be managed, patched, and deployed. Tagging is one of the ways that helps our organizations keep track of resources, especially if we provision them and we move on to our dream job somewhere else. Let me know how you handle things today.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Migrating a Large Database

    I have upgraded lots of SQL Servers from one version to the next, and for the most part, the process has been smooth. That’s not always the case, and there have been some long nights where the Operations staff had to scramble to fix things, script out old logins, call Microsoft support, and perform various data exports and imports to get a new instance running. I’ve been a part of quite a few of those teams.

    Most of my upgrades were with relatively small databases, at least small for that time period. However, I have upgraded a few “large” databases in the past. We had a 400GB database on SQL Server 6.5 in 1999 that was a challenge to move to SQL Server 7. I also upgraded an 800GB database in 2001 from 6.5 to 2000 for our Financial team, which involved a lot of stress.

    I haven’t upgraded a 1TB database, much less a 4TB one, but I know this can be very time-consuming to move all that data. Even with much faster hardware and networks these days, working with that much data can require a decent amount of downtime. That’s not very acceptable these days, especially for applications that are used by customers all around the world and all around the clock.

    I found an interesting upgrade story from a SaaS provider that runs PostgreSQL. I don’t use that platform, but I found the write-up of their process to be aligned with some of my experiences with SQL Server. They delayed upgrading an old version of PostgreSQL and then had the desire to move to as current a version as possible and delay future upgrades (again). I’ve certainly experienced that. Their goal was also to minimize downtime, as their customers are constantly connecting.

    The plan for the upgrade made sense to me. Use replication to move most data and then minimize downtime. That’s a technique that can work in SQL Server, though we’d be more likely to use log shipping to simulate this. They also trimmed and cleaned some data, removing the need to upgrade some of the tables. That’s something many of us might be able to do every year in some databases, especially for large logging tables where older data might rarely be read. I’d even think about moving that data to another database and using a synonym to access it if it were needed. That’s just a good idea for general DR planning.

    I also appreciated them creating a runbook and testing the process multiple times in staging. Their big takeaway here: practice over and over with a realistic workload. They created some problems for themselves by not using a real enough workload. That ought to be something your organization does on a regular basis to test your software and ensure new code performs well. Using the same process for upgrades is a bonus.

    Large upgrades are stressful and often they are “big bang” deployments where you can’t go back to the old system. Practice as much as you can, make sure you have backups, and then be ready to adapt to whatever challenges come up. Plan ahead, and be prepared for a long night. If you do, you’ll likely have success. If you don’t, I am fairly sure something will go wrong and you’ll regret not thinking ahead.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Scripting Makes Mistakes Easier Than Ever

    A number of you likely use Atlassian products like Jira, Confluence, Opsgenie, or something else. You might have been affected by a large outage they had (post incident blog, Company Q&A, TechRepublic report) recently which lasted at least 9 days. I don’t know if all customers have their data back and are working, but this was a surprisingly poorly handled incident according to a number of reports from customers. There’s a great write-up from the outside that you might want to read.

    The bottom line in this issue is that Atlassian looked to deactivate a legacy product with a script, but they apparently didn’t communicate well among their teams. The script ended up using the wrong customer IDs and also marked the sites for permanent removal, not temporary removal (soft delete). While they supposedly test their restore capabilities, they weren’t prepared for partial restores of subsites. I’m guessing this is likely a partial database restore, which many of us know is way more complex than a full database restore.

    Leave aside the issue of a software-as-a-service (SaaS) company failing their customers, and the lack of communication with customers. The more interesting thing for me is the challenge of poor coding and communication internally. Clearly, the project to deactivate their legacy app wasn’t well planned or tested and the code used was probably executed at too wide a scale initially.

    When we deploy code changes to a large number of items, we want to test them at a small number first. Whether we are deploying to multiple databases, against many customers, or different systems, a standard method of making changes at scale involves working in rings. Azure DevOps describes this in docs, and they actually use rings to change the platform. We used the same pattern 20 years ago for software and database updates to many systems. We would internally deploy to a few users to look for issues. Then a week later we would deploy to a small number of systems to check for unexpected issues. Then typically to most systems in the third ring with a fourth ring a week later to catch up stragglers that needed more time to prepare.

    I find many customers, especially those with sharded/federated databases or many systems unwilling to spread out deployments in this manner. Often they yield to pressure from business users to ensure everyone gets the same update at the same time. I would never recommend this approach as we need to ensure we are looking at scripts in a controlled environment, or even two, before we deploy things widely. I’d be even more cautious about one-off administrative scripts that might make a change similar to the one Atlassian attempted. Those are often not seriously tested enough.

    At the very least, any of us working with multiple customers in a single database or in multiple databases ought to ensure we can backup and restore a single customer, but more importantly, can you restore a group of customers. If you make a mistake like Atlassian, which scripting allows us to do extremely rapidly, can you recover a partial set of data? Many of us don’t test this, but that’s likely something we ought to consider when we work with scripts that are designed to only change some data. Most of us don’t experience complete failures, but partial ones, usually because of human error. We ought to know how to deal with these situations.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.