Voice of the DBA

Category: Editorial

  • How Often Do You Patch?

    I saw two things recently. First, CU25 for SQL Server 2017 was released. I realized that I was CU8 locally, so I had work to do. Second, I noticed that SQL Server 2016 fell out of mainstream support on 13 Jul, 2021. Amazing that it’s been around for 5 years already and has moved into Extended support. We’ll still get security updates, but nothing will be fixed from that version, which it seems many people are running.

    Microsoft also announced they will put out a final Service Pack for SQL Server 2016. I am glad to see that, as I’ve often wanted to know there is a final patch that is released for each version, as I can know when I’m done patching.

    Today I’m wondering if you track this at all. Do you actively look to ensure your instances are patched? Are you on a schedule of some sort, maybe matching Microsoft’s every-other-month pace, or perhaps just a few times a year? Or is this more ad hoc?  When I go visit companies, I often find instances at all different levels, often without any reasoning as to why systems aren’t consistently patched.

    I used to try and ensure we patched every quarter, though certainly, I faced resistance from some business owners of systems over the worry that a patch might break something. Valid concerns back then, and still somewhat valid today, though if a patch gets through a few months without reports of issues, it’s probably safe for most systems. Still, make sure you test, especially for business critical workloads.

    These days, with so many hacking attempts, ransomware distribution channels, and no end to phishing, it makes sense to stay on top of patches and make sure you are up to date or at least planning to do so. Some of the high-profile hacks, like the Equifax one, have occurred on systems where patches were available but not applied. I don’t know that many of the patches are closing holes in SQL Server, but there are definitely issues with Windows where you would want to ensure your host OS was patched. Maybe that’s something you ought to check on today and ensure you have a plan to apply those updates.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Danger of Management Access

    First we had the Solarwinds hack, and now we have a Kaseya ransomware epidemic. It seems the criminals are moving up the stack. We used to see physical attacks on tapes and keyboards, then we saw OS level attacks. Now we seem to be getting to the management layer for software that is used to help us run systems at scale. Since we often require some level of privileged access for monitoring and management systems, this is scary. I certainly wish that we didn’t require admin access for monitoring, but unfortunately platforms sometimes do.

    Many of us depend on some standardization and some sort of software to ensure we can manage systems at scale. I don’t know about the OS world, but in the SQL Server world, there are relatively few vendors that provide software for managing systems. If one of these were compromised in some way, this could be very bad for many database administrators. Fortunately, many of us know how to air gap backups and ensure that we are prepared for disasters.

    Or we should. If you don’t know how to do this, you ought to be learning right away. Review backup plans, ensure you can rebuild systems, test restores, and brush up all your recovery skills. Be ready for whatever a criminal might throw at you, including having gotten ransomware into some of your backups.

    This attack seems to have taken advantage of a zero day, or very early, vulnerability that was discovered by a Dutch security research firm. The firm looks into management software, especially admin interfaces, specifically because they are worried about the lack of security in many products. In this case, Kaseya builds tools that allow admins to distribute software to other systems on the network. In this case, criminals used the management software to distribute ransomware.

    The updates from the Kaseya are less than stellar, and if I were a customer, I’d be rather upset. They seem to keep setting unrealistic plans to restore service and then constantly revise them across a few days, all the while with customers that are likely stressed and overworked. I’d also be upset in that they claim only a few of their thousands of customers are affected, but they neglect to admit that some of those customers affected as Managed Service Providers, who themselves have thousands of customers using this software.

    There are some technical  details in this piece, in case you want to check your own systems. If you think you have multiple pieces of software that might protect you, read the article. This deployment shuts off some other products, like Microsoft Defender.

    I feel bad for many people here. IT staff at affected companies that have likely been incredibly stressed and overworked recently. The consumers of some affected customers, like those that might shop in the Swedish grocer, Coop, who shut down more than 400 stores. I don’t know the state of grocery shopping in Sweden, but this might dramatically impact many people that just want to buy food for their families.

    Ransomware continues to surprise and worry me. Large profile hacks keep coming, affecting lots of people. Often these are because of previously undiscovered software vulnerabilities or simple mistakes made by privileged users. I hope that at some point insurers and governments start to put more pressure on companies that make widely used software to ensure they are adhering to best practices and have detailed security practices in place to ensure their code is constantly checked for issues, and that they have detailed plans for responding to and patching customers when there are issues. Because, they likely will have an issue at some point.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes

  • Seriously Real Time Data Processing

    There are many of us that work with systems where data is processed in real-time and then used to make decisions. These might be humans viewing reports and then taking action, or some automated system that might react based on a value changing. In many cases, however, the amount of data, timing in which to react with a decision, and the implications for mistakes aren’t that critical. We have some leeway for the processing not being perfect.

    For vehicles traveling at 180mph, the tolerance for mistakes is low, with the chance of a catastrophic crash looming constantly. That is what is happening with a competition at the Indianapolis Motor Speedway, where university students are competing to develop race cars that can move around that track at these speeds. This would be quite a challenge for vehicles moving around the track by themselves, but in this case, it’s a race with multiple cars.

    This might seem silly, but it’s a step towards understanding just how much data needs to be processed and how the results can deal with the chaos in the real world. Each car is independent, so has to react to the other vehicles and make decisions on how to adjust its own operation in real-time, with a sub-second response to prevent accidents. This isn’t different from the decisions human drivers have to make in a race, and there are plenty of mistakes that result in crashes. However, humans can think in new situations and react. They don’t need to have every possible response programmed in.

    These AI-driven race cars will be similar, but how well they perform remains to be seen. This is the type of test environment that will help us move forward in using technology and AI models in less constrained environments, like a public highway. Lots of technology was tried on race tracks before it became available to consumers, and I think this will make its way to retail cars as well at some point.

    There are already some companies trying to build this into cars. Tesla famously has Full Self Driving, although this has been in beta for a long time, and its results are less than stellar in some cases. Waymo has been working on the problem, and I actually had the chance to ride in a self-driving Uber in Las Vegas, though that experience was less than thrilling. The driver had to take control a number of times, so this wasn’t quite self-driving.

    There is a lot of work still to be done here, and I don’t know how quickly this will become safe enough for general driving. I suspect that we’ll see this in very limited areas first, like a zone in a city that only allows these types of cars, or maybe specific highways, like HOV lanes. Somewhere the problem domain is simpler, with less decisions that need to be made.

    There are already lots of safety features in modern cars that help prevent mistakes and accidents, but most of these are simple systems that aren’t making decisions in many ways. Moving to more complex driving operations will require some heavy data gathering, processing, and analysis, something that should be of interest to data professionals. This is a problem domain that will be fascinating to watch in the next few years.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Messy Job Descriptions

    I saw a job description recently for a DBA that asked for SQL Server experience, but also “other RDBMSes, like Cassandra and PostgreSQL”. Not sure Cassandra fits there, or why this says like. I’ve seen some other ads that ask for C# or Python. Some asking for MDX/DAX knowledge along with AWS Cloud Formation and programming APIs. Some have a required and optional or “nice to have” sections, but many include a laundry list of technologies and skills. For software developer roles, the list of skills often exceeds what I think any person in the world might know.

    There was a debate about this at SQL Server Central in one of the threads. It seems some people are split on whether this is a problem or a good thing. Quite a few people noted that they wouldn’t apply when there are so many items listed that they don’t have experience in. For others, we wouldn’t hesitate if we had around 50% of the items listed. I’m in the latter category, as I’ve had plenty of friends, and myself, get jobs that might have seemed out of reach based on the description and our skills.

    In my experience, often a job description is put together on the fly and in a hurry, usually by someone that isn’t familiar with the job. They ask others what to include, and we end up with a large list of things that would be nice, but not necessary. The end result isn’t always what the job entails, at least not completely. Often I’ve found as a developer or operations person I might end up lightly touching parts of different roles, but not regularly and not too deeply.

    I don’t know that I think it’s worth effort to tightly define a job for a new hire, as the job requirements can change, and we might adapt a job to the individual. Not completely, but if someone knows more about reporting or BI than HA/DR, we might have them tackle more of that work and only partially work on clusters or AGs. Others might fill in with more HA/DR and less BI work. The reverse also could be true, so should we have a job description that is narrowly defined to DBA work with an HADR focus? Or one for BI? I don’t know, but I learn towards not tightly defining a job description.

    Hiring is a difficult enough process, especially today, without too tightly defining the roles. I do think it’s worth spending time with the team doing the work to list out the necessary skills (and levels) needed to help them, but adding in other items is useful. It casts a wider net, and it helps you as the hiring group, think about what tradeoffs you might make. If someone is weak with replication, and we use it, but they have some strong Azure skills, maybe we accept that. We know we’ll need to train this person on replication, but they might help us better understand the cloud. Perhaps that’s a better choice than someone highly skilled with replication but without a lot of other experience.

    Or maybe not.

    Ultimately, on the hiring side, include what you want. It doesn’t have to be perfect. If you don’t get candidates, then re-examine it, but list what you really need, and separately, what you want.

    On the candidate side, don’t be intimidated. If you get to 45% of the skills, apply. If you hit 75% of the requirements, apply. Maybe even if you’re a little short on those but you have other skills. Use what you know, and what’s been listed, to help you drive the interview. Emphasize your strengths, and convince them you can learn. Ask questions, use “I don’t know, but” often with examples of how you might gain the knowledge or get help. That often is more important than what you have done in the past.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.