Voice of the DBA

Category: Editorial

  • Do You Have ALL the YAML?

    YAML is a file format, and as with many formats, some of us love them and some of us hate them.  It seems to be better than XML in many ways, and perhaps easier to deal with the JSON. It might not be better than csv/tsv/delimited formats for large transfers but for many of us, it’s a nice format for configuration items.

    While the format felt fairly intuitive to me, and it’s not hard to write, it is quite persnickety about whitespace. This makes using some plugin, like the Red Hat YAML extension for VS Code, important to help you prevent mistakes. Even as easy as the format can be to read and understand, it’s also easy to make mistakes with the whitespace as you indent and try to add subkeys.

    I was watching an AWS talk, and there was an interesting note about using YAML for control planes and being sure that you have some sort of checksum if you do. Why? Because you aren’t sure if you have the entire file. A YAML file could be truncated in any file transfer, and it would still appear to be valid. Hearing that made me realize that those annoying closure tags in XML and JSON might have some value.

    Those of you that work with YAML, how are you sure you got the entire file? Is there something you’d program in? Do you checksum the file and pass that along? Do you include a required, closing key:value tag of some sort? I don’t, but I might think about doing so in any place where an invalid or incomplete file might cause me problems. This certainly seems like something you’d want in a control file, like one used for Kubernetes.

    In most cases, we assume if we can read a file, then we have the complete file. I don’t know of many customers that require some sort of checksum or validation for a file. Certainly, if a CSV or TSV was missing rows, the file might still appear valid to an import process. XML and JSON should have a closing tag or character, so we’d hope we could catch this, but maybe not.

    Moving around data through files, especially data used to drive processes, should include some error handling. That would mean that we have some way to detect if part of our file is missing. There are ways, but it seems that in many cases we’ve gotten lazy about implementing them in file transfers. Certainly, I don’t see people adding a checksum to their YAML files, which seems like something that we’d want to require.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • How Often Do You Patch?

    I saw two things recently. First, CU25 for SQL Server 2017 was released. I realized that I was CU8 locally, so I had work to do. Second, I noticed that SQL Server 2016 fell out of mainstream support on 13 Jul, 2021. Amazing that it’s been around for 5 years already and has moved into Extended support. We’ll still get security updates, but nothing will be fixed from that version, which it seems many people are running.

    Microsoft also announced they will put out a final Service Pack for SQL Server 2016. I am glad to see that, as I’ve often wanted to know there is a final patch that is released for each version, as I can know when I’m done patching.

    Today I’m wondering if you track this at all. Do you actively look to ensure your instances are patched? Are you on a schedule of some sort, maybe matching Microsoft’s every-other-month pace, or perhaps just a few times a year? Or is this more ad hoc?  When I go visit companies, I often find instances at all different levels, often without any reasoning as to why systems aren’t consistently patched.

    I used to try and ensure we patched every quarter, though certainly, I faced resistance from some business owners of systems over the worry that a patch might break something. Valid concerns back then, and still somewhat valid today, though if a patch gets through a few months without reports of issues, it’s probably safe for most systems. Still, make sure you test, especially for business critical workloads.

    These days, with so many hacking attempts, ransomware distribution channels, and no end to phishing, it makes sense to stay on top of patches and make sure you are up to date or at least planning to do so. Some of the high-profile hacks, like the Equifax one, have occurred on systems where patches were available but not applied. I don’t know that many of the patches are closing holes in SQL Server, but there are definitely issues with Windows where you would want to ensure your host OS was patched. Maybe that’s something you ought to check on today and ensure you have a plan to apply those updates.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Danger of Management Access

    First we had the Solarwinds hack, and now we have a Kaseya ransomware epidemic. It seems the criminals are moving up the stack. We used to see physical attacks on tapes and keyboards, then we saw OS level attacks. Now we seem to be getting to the management layer for software that is used to help us run systems at scale. Since we often require some level of privileged access for monitoring and management systems, this is scary. I certainly wish that we didn’t require admin access for monitoring, but unfortunately platforms sometimes do.

    Many of us depend on some standardization and some sort of software to ensure we can manage systems at scale. I don’t know about the OS world, but in the SQL Server world, there are relatively few vendors that provide software for managing systems. If one of these were compromised in some way, this could be very bad for many database administrators. Fortunately, many of us know how to air gap backups and ensure that we are prepared for disasters.

    Or we should. If you don’t know how to do this, you ought to be learning right away. Review backup plans, ensure you can rebuild systems, test restores, and brush up all your recovery skills. Be ready for whatever a criminal might throw at you, including having gotten ransomware into some of your backups.

    This attack seems to have taken advantage of a zero day, or very early, vulnerability that was discovered by a Dutch security research firm. The firm looks into management software, especially admin interfaces, specifically because they are worried about the lack of security in many products. In this case, Kaseya builds tools that allow admins to distribute software to other systems on the network. In this case, criminals used the management software to distribute ransomware.

    The updates from the Kaseya are less than stellar, and if I were a customer, I’d be rather upset. They seem to keep setting unrealistic plans to restore service and then constantly revise them across a few days, all the while with customers that are likely stressed and overworked. I’d also be upset in that they claim only a few of their thousands of customers are affected, but they neglect to admit that some of those customers affected as Managed Service Providers, who themselves have thousands of customers using this software.

    There are some technical  details in this piece, in case you want to check your own systems. If you think you have multiple pieces of software that might protect you, read the article. This deployment shuts off some other products, like Microsoft Defender.

    I feel bad for many people here. IT staff at affected companies that have likely been incredibly stressed and overworked recently. The consumers of some affected customers, like those that might shop in the Swedish grocer, Coop, who shut down more than 400 stores. I don’t know the state of grocery shopping in Sweden, but this might dramatically impact many people that just want to buy food for their families.

    Ransomware continues to surprise and worry me. Large profile hacks keep coming, affecting lots of people. Often these are because of previously undiscovered software vulnerabilities or simple mistakes made by privileged users. I hope that at some point insurers and governments start to put more pressure on companies that make widely used software to ensure they are adhering to best practices and have detailed security practices in place to ensure their code is constantly checked for issues, and that they have detailed plans for responding to and patching customers when there are issues. Because, they likely will have an issue at some point.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes

  • Seriously Real Time Data Processing

    There are many of us that work with systems where data is processed in real-time and then used to make decisions. These might be humans viewing reports and then taking action, or some automated system that might react based on a value changing. In many cases, however, the amount of data, timing in which to react with a decision, and the implications for mistakes aren’t that critical. We have some leeway for the processing not being perfect.

    For vehicles traveling at 180mph, the tolerance for mistakes is low, with the chance of a catastrophic crash looming constantly. That is what is happening with a competition at the Indianapolis Motor Speedway, where university students are competing to develop race cars that can move around that track at these speeds. This would be quite a challenge for vehicles moving around the track by themselves, but in this case, it’s a race with multiple cars.

    This might seem silly, but it’s a step towards understanding just how much data needs to be processed and how the results can deal with the chaos in the real world. Each car is independent, so has to react to the other vehicles and make decisions on how to adjust its own operation in real-time, with a sub-second response to prevent accidents. This isn’t different from the decisions human drivers have to make in a race, and there are plenty of mistakes that result in crashes. However, humans can think in new situations and react. They don’t need to have every possible response programmed in.

    These AI-driven race cars will be similar, but how well they perform remains to be seen. This is the type of test environment that will help us move forward in using technology and AI models in less constrained environments, like a public highway. Lots of technology was tried on race tracks before it became available to consumers, and I think this will make its way to retail cars as well at some point.

    There are already some companies trying to build this into cars. Tesla famously has Full Self Driving, although this has been in beta for a long time, and its results are less than stellar in some cases. Waymo has been working on the problem, and I actually had the chance to ride in a self-driving Uber in Las Vegas, though that experience was less than thrilling. The driver had to take control a number of times, so this wasn’t quite self-driving.

    There is a lot of work still to be done here, and I don’t know how quickly this will become safe enough for general driving. I suspect that we’ll see this in very limited areas first, like a zone in a city that only allows these types of cars, or maybe specific highways, like HOV lanes. Somewhere the problem domain is simpler, with less decisions that need to be made.

    There are already lots of safety features in modern cars that help prevent mistakes and accidents, but most of these are simple systems that aren’t making decisions in many ways. Moving to more complex driving operations will require some heavy data gathering, processing, and analysis, something that should be of interest to data professionals. This is a problem domain that will be fascinating to watch in the next few years.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.