Voice of the DBA

Tag: patching

  • Get Patched If Not Supported

    There are plenty of reasons not to upgrade your SQL Servers to a new version. Perhaps you know the system is working and don’t want to disrupt activity. Often we don’t want to take a chance that some change in functionality causes us issues. In many cases, the new functionality might not be used in your current system, and you don’t see the ROI for the cost of upgrades. Costs certainly are a concern as SQL Server has gotten more expensive across time. In all these cases, it makes sense to stay on your current version. Software Assurance might negate the upgrade cost in money, or even give you a reason to upgrade, but it doesn’t prevent any of the other time and resource costs.

    Patches don’t cost money, however, and they are included in your cost. While I am nervous about applying patches right away, I do want to apply them at some point. If for no other reason, I do want to ensure I’m going to get into a position where I have to apply a patch to fix something or get support in an emergency and not have done any testing. I am more nervous now after the recent Windows Fall Update issues, and definitely want to let others test patches for a month or two before I apply them. Thanks to those of you that patch early and report issues.

    The exceptions I make here about avoiding patches are for older versions of SQL Server. If I’ve got systems that are out of mainstream support, I want them patched. At that point, only security patches are coming and if I get those, I need to apply them, which means I need to be sure all other patches are in place.

    Apparently many of you think like me, but not enough of you. I ran across a post from Erik Darling looking at a cross section of their customers, who I’d like to think are slightly more on the ball than everyone else, but perhaps not. In any case, lots of SQL Server 2008 and SQL Server 2012 systems haven’t been patched, with R2 and 2014 not far behind. While I know some vendors make patching difficult for SQL Server DBAs, we ought to be pressuring them more and more, and even asking our management to do the same.

    We ought to be patching systems on a regular basis. That’s a part of the software life cycle and until we find ways to write bulletproof software, we’re going to be patching. Microsoft is in the same situation as most of us, which is why they deliver patches regularly for SQL Server. They need to patch their Azure databases and ensure they are prepared for security issues.

    Make a plan to test these patches on your systems and start implementing it. None of us wants to be caught in a situation where we have to apply a security patch to an older server next week and we have no plan for how to test the application. Perhaps even worse, none of us wants to have a data breach because we were afraid to apply a security patch *because* we didn’t have a test plan.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • The Combinations of Software

    Security issues seem to be appearing more frequently, not less. I’d expect that we would be getting better at writing software, and I think many of us are. The problem is that more and more people are writing software and we still haven’t found a way to better train developers early in their careers. Perhaps the one good thing is that more and more developers are using frameworks, which create more consistent software. If issues are discovered, a patch can ensure a large swath of systems can be patched.

    The bad news is that far too many development groups build systems quickly, but don’t patch them in an expedient manner. They may be afraid or just not bother.

    A short while ago there was a loss of data from Ticketmaster ticket sales. Apparently a chatbot was used to steal information. As soon as Ticketmaster discovered the issue, they disabled the software. There is some disagreement as to who is at fault here. The chatbot vendor says their JavaScript chatbot should not have been running on a secure payment page.

    The specifics here aren’t important, but it is a concern that more and more often we are assembling applications from pieces of software. We often use plugins on websites and other building blocks when we put together a system. In more and more cases, we will be connecting this software to our data stores. That wasn’t the case here, but often there is some data access, and since we may keep both secure and non secure data in the same database, any vulnerabilities in one building block can cause security issues in others. The weakest link in the chain saying applies here.

    I wonder how many of you worry about issues with the assembly of whole pieces of software. The pieces should be more secure, or at least, more easily patched. There should be more incentive and resources to patch software used by many people, though many times vendors become hesitant to do any more than absolutely necessary.

    I’m not sure if it’s better to build out of pre-written pieces of code, but I do know that security is a shared responsibility and I wish it was more of a priority for all developers. The security of our application can depend on that weakest link.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.9MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • Are You Patched?

    It’s been a few weeks since the Spectre/Meltdown bugs were announced for most CPUs. Microsoft has been working hard to build patches, and they’ve provided fixes for Windows and SQL Server. Other manufacturers have released fixes for other platforms, though I wouldn’t be surprised if more patches are coming. We put together a page at SQLServerCentral with information and links, and if you haven’t checked it out, you should.

    If you haven’t patched systems, patch them ASAP.

    This is a bad bug, affecting many CPUs, across multiple architectures, and includes potential issues with virtual machines. The guidance and conversations I’ve heard from various vendors is that many of them aren’t completely sure of all the potential risks or attack vectors, but they are worried that customers will leave this vulnerabilities open in the future. Since this affects hardware, it’s entirely possible that an exploit could read memory from other applications and processes.

    Again, if you haven’t patched systems, patch them.

    There are reports of potential issues, so everyone certainly needs to test systems. Perform a P->V (Physical to virtual conversion) and patch a VM. Make sure the server still runs. If you’re on VMs, snap a copy and patch it as a test. Older processors might see a performance penalty with the patch, but worse performance is better than having a security hole in your CPU available to operating systems.

    This is the type of fundamental architectural bug that’s is very worrisome. The race to be efficient, to copy what works from others, this leads to less innovation not more. I hope that this is a bit of a lesson that we do need separate architectures and approaches to computing problems, both in hardware and software. I love relational databases, but I’m glad that there are other types of systems being used for data storage. I think Windows works really well, but I like competition and think it’s good that we have MacOS, Linux, and more.

    It’s good to have standards and interoperability, but I do think that a heterogeneous environment is good for security, and I hope the world continues to try new architectures as we advance computing ever further.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 3.1MB) podcast or subscribe to the feed at iTunes and Libsyn.

  • No More Service Packs

    The new servicing model for SQL Server is out, with a great explanation from Glenn Berry (there’s also a webcast). The summary is that going forward with SQL Server 2017, we won’t get Service Packs anymore. I’m somewhat sad, as I was always looking for SPs as a once a year patch if I didn’t have any major issues with the platform. This allowed me to keep up to date, but not constantly test smaller patches. I liked that, partly because I’ve been worried about the quality of CUs in the past.

    We’ve had Cumulative Updates (CU) for quite some time. While I’ve seen a few service packs pulled and re-released, that’s been very rare with CUs. I do find that SPs might have more issues because building one is an out of process, disruptive project, whereas CUs have become a regular, repeatable part of the development process. That’s not to say that the CUs are bug free. A few of them have caused issues, and that’s a problem that I hope improves over time. However, I also can’t imagine what it’s like working on such a large codebase, for installation in a large number of environments. It has to be a tremendous challenge to test all cases and remove bugs. Not that there shouldn’t be fewer issues, especially in some areas, but I’m not sure there will ever be a complete absence of bugs.

    I’ve changed my mind on CUs, and I like the new servicing model for it’s simplicity as well as the automation and deep testing that CUs get. There is no confusion, no resources at Microsoft maintaining an RTM and SP1 branch with patches. I dislike code merges, and I suspect they’re often a source of unintended bugs, so the fewer that are done the better. Now we’ll have SQL Server 2017 essentially with one branch of code and patches. We’ll have CU1-11 in the first year and then CU 12-15 the next year. Across fix years, I’d expect that we’d go from roughly 16 CUs per SP + SP to potentially 28 CUs in a single stream across the five year mainstream support of the product.

    I know there are potentially more issues discovered early in a version’s life, and hopefully fewer over time as developers write more tests and learn to cover edge cases better. I suspect that we’ll see fewer and fewer items in each patch over time. Maybe not right away in SQL 2017, but it seems as though the number of issues corrected in each CU declines over time, and I’d expect that testing and better coverage of feature corner cases will come in the future. Perhaps SQL 2019 or 2020 will have fewer issues in the first year with even more testing.

    I know some people see this as a message that the product isn’t well tested. That’s a fair view, after all, doubling the number of patches in the first year could be taken either way. I prefer to view this as the new way software is being developed. Build a pipeline, including lots of testing and feedback, and be ready to respond quickly if things are broken. I hope that’s what Microsoft is thinking, and based on their push for better security and quality, it’s the view I’ll adopt.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 5.5MB) podcast or subscribe to the feed at iTunes and Libsyn.