Voice of the DBA

Category: Editorial

  • The End of SQL Server 2019

    Well, not really the end. I doubt anyone running SQL Server 2019 is going to stop (or upgrade) just because mainstream support ended. Actually, I wonder how many of you know that SQL Server 2019 passed out of mainstream support on Feb 28, 2025. I do think the 6 or 7 of you running Big Data Clusters likely knew this was the end of any support.

    I saw a report in the Register on this, which includes a survey of which versions are still running. This is from an IT asset firm and matches Brent Ozar’s Population report. 44% of you are running SQL Server 2019, which is the largest percentage. Since there’s an additional 32% of you running versions older than 2019, I’m sure that upgrading isn’t a priority.

    It seems like just a couple of years ago that SQL Server 2019 was released. At the end of February Microsoft ended mainstream support for this version. There will still be security fixes released, but no more cumulative updates. The Register says if you don’t upgrade, you might run into a bug and not get a fix (unless you buy extended support), but that’s never worried me. If I haven’t hit a bug 5 years in (or likely 3-4years after my last upgrade), I’m not too worried. If I run into something it’s likely from new code and I’ll just change the code to work around the issue.

    I do expect to run a database platform for a decade, and I am glad that Microsoft continues to supply security patches for this period. While I certainly want every database firewalled, reducing the attack surface area of known vulnerabilities is good. I also find myself less concerned about the security of older versions. If there is a big security vulnerability discovered in 2017 tomorrow that exists in previous version and I had a 2012 server, I’d just prioritize an upgrade then.

    Upgrades are hard, eat a lot of valuable time, and don’t necessarily provide many benefits. Most applications tend to use basic CRUD features and whatever was available at the time in that version. If I use a tally table to split strings in 2017, I’m unlikely to rewrite that code to use STRING_SPLIT with an ordinal if I upgrade to 2022. That certainly isn’t a selling point for me to upgrade. My boss knows that isn’t something we’d take advantage of in older code.

    I’m not a bleeding edge person, and I wouldn’t push for upgrades. If you want to stay somewhat current with versions and are running 2019, I’d be waiting to test my application on SQL Server 2025 at the end of the year or early 2026. If I were mandated to stay current, I’d still be doing that, not jumping to 2022 right now. However, I do recommend that everyone patch their systems with cumulative updates to ensure their security is up to date. There have been several security patches in the past few years that you should have applied and if you haven’t, this is a reminder to do so soon.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Monoliths and Microservices

    I have run into a lot of people in the last few years that love decoupled software and microservices. It seems many people are aiming to move their work in this direction, and while I see some appeal, I also see tremendous additional complexity that has moved out of the software into the operations and debugging space. As I read the book Observability Engineering, I found myself thinking the complexity of setting up more logging and instrumentation in an observability framework as well as the costs of managing a system start. That caused me to think this is overkill for most software.

    To be clear, I don’t think that Uber could have been built as a few monoliths, and there are other examples of such systems, but most of us don’t work at that scale. There are lessons to be learned about large, real-time software systems, and certainly Google, Amazon, Spotify, Netflix, etc. can help us understand how certain techniques work better at scale, but most of us don’t work at that scale.

    Scale to me is thousands of connections and terabytes of data. Those companies tend to work at a couple orders of magnitude above that. Those are also companies that must have real time systems up to survive. The vast majority of companies I’ve worked at might suffer a loss with an outage of a system, but honestly, we could survive a day or two of recovery.

    Heck, look at all of the companies that have had portions of their digital infrastructure knocked offline from ransomware the last few years. Some failed, but most didn’t. The incident sucked for IT staff, but those companies survived. I still remember the SQL Slammer worm forcing us to take our entire network offline for multiple days at a large software company. We still ran sales, and support, and other systems independently or from paper. Granted that was 20 years ago, but I’m sure Redgate and many other organizations could survive a few weeks with no network.

    Uber on the other hand, that would be a disaster for them. They would likely survive, but at a high cost. How many people would jump to a new service and never look back?

    I was reading a piece on coupling and complexity, which I’ll discuss a bit in another article, but it got my thinking of how often I see people overcomplicating their work by trying to move to a decoupled world instead of hiring and training people to adopt better architectures and communicate better. After all, moving to microservices isn’t going to avoid the training issue. You’ll still have to teach people more.

    And while you’re at it, force every developer to pass a SQL test every year. That might help you build better systems as well. SQL isn’t going away and better SQL coding in will result in much better applications everywhere.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • More Supply Chain Attacks

    The latest code supply chain attack isn’t a direct attack, but a failure of a system designed to be efficient. There is a Go (Go-lang) module that had a malicious module inserted into it years ago. Someone caught this and removed the module from GitHub, but Google had cached this and has kept it alive for the last three years.

    This attack worked because it relied on the Go Module Proxy services, which prioritized caching. Even when the source changed, the cache wasn’t invalidated or reloaded, which seems like a major ovesight or even larger design flaw, but I know everyone trying to maintain software archives at scale tends to cache a lot. After all, so many developers might load modules in their daily work, in CI, etc. that caching matters.

    However, the bigger issue is that criminals are getting more enterprising and aiming for supply chain attacks where possible. One would hope that any PRs (pull requests) are carefully examined, but the truth is that a lot of people might depend on their unit tests passing in a CI build. They are not necessarily looking to see if anyone has actually entered malicious code or even poorly written code. I can see some people worried more about code structure and naming (or tabs/spaces) than what the code does. Imagine seeing a change that looks innocuous at the top of a file, all tests pass, and you merge the change, but didn’t notice there was a few new functions added below in the file because they’re not obvious in the UI.

    Someone maintaining a popular code repo as a side project might be fooled here, but if this were in a corporate repository we might be even more susceptible. We have more reasons to trust the PR isn’t a problem if the code passes tests inside of an organization. After all, who things criminals might insert code into their corporate repo? Not many people do, but we’ve seen quite a few successful supply chain attacks in the past. Who knows how many more we don’t know about.

    Security is a hard business, and when it’s extended to the code we write, it might even be harder. I know there are security scanning solutions you can integrate into your codebase, but those detect what they know about and criminals keep finding new ways to attack us. Ultimately I think we depend on code maintainers carefully examining PRs from outside their circle of trusted individuals, and even then, things can slip through.

    Some days I think it’s truly a mad, mad world.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Lower Your Attack Surface Area

    It’s no surprise that our systems are under attack by all sorts of criminals. Some organized, some opportunistic, some just aiming for vandalism. We need to protect our digital systems to prevent issues, and a part of better protection is reducing the number of places that are vulnerable. Those places include databases.

    This article discusses the rising costs of data breaches and the increased frequency of attacks. It also examines the increasing number of regulations that are demanding proof of stricter security measures. It can be hard enough to defend production systems, let alone protecting dev/test environments. I see an increasing number of organizations that limit access to production systems, even to the point that this impedes some of the daily work habits of technology professionals, but that is probably a good thing. Too many of us are too lax when it comes to security.

    There are lots of approaches to getting better at security, but one of the easier ones is to avoid making copies of sensitive data. About half of you (I hope it’s not more) still use production restores in dev/test environments. That’s a tripling of the places your data could be attacked if you restore a database to a development server and a test server. If could be even worse if you make more copies.

    An easy solution in today’s world is to build a better test data management process for either anonymizing and obfuscating your sensitive data or generating synthetic data. Both have their challenges and I suspect that most organizations need a combination of both approaches to both protect their data as well as build better software for their customers. After all, a huge amount of bugs are data related, where developers have not tested their software against enough different data elements. Both synth data and anonymized data help here to produce enough different edge cases that your testing is thorough enough to increase quality.

    Of course, you need extensive testing, which means automation. Ideally an automated DevOps flow that subjects your software to increasingly complex tests as it moves through your pipeline to ensure it’s ready for release. This also means a good set of test data, not only for QA, but for automated tests. You need a test data management strategy.

    Securing digital systems is a complex task, but we ought to try and make it easier on ourselves by developing good habits in how we manage both access and by limiting the copies of sensitive data.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.