Voice of the DBA

Tag: security

  • The Danger of Management Access

    First we had the Solarwinds hack, and now we have a Kaseya ransomware epidemic. It seems the criminals are moving up the stack. We used to see physical attacks on tapes and keyboards, then we saw OS level attacks. Now we seem to be getting to the management layer for software that is used to help us run systems at scale. Since we often require some level of privileged access for monitoring and management systems, this is scary. I certainly wish that we didn’t require admin access for monitoring, but unfortunately platforms sometimes do.

    Many of us depend on some standardization and some sort of software to ensure we can manage systems at scale. I don’t know about the OS world, but in the SQL Server world, there are relatively few vendors that provide software for managing systems. If one of these were compromised in some way, this could be very bad for many database administrators. Fortunately, many of us know how to air gap backups and ensure that we are prepared for disasters.

    Or we should. If you don’t know how to do this, you ought to be learning right away. Review backup plans, ensure you can rebuild systems, test restores, and brush up all your recovery skills. Be ready for whatever a criminal might throw at you, including having gotten ransomware into some of your backups.

    This attack seems to have taken advantage of a zero day, or very early, vulnerability that was discovered by a Dutch security research firm. The firm looks into management software, especially admin interfaces, specifically because they are worried about the lack of security in many products. In this case, Kaseya builds tools that allow admins to distribute software to other systems on the network. In this case, criminals used the management software to distribute ransomware.

    The updates from the Kaseya are less than stellar, and if I were a customer, I’d be rather upset. They seem to keep setting unrealistic plans to restore service and then constantly revise them across a few days, all the while with customers that are likely stressed and overworked. I’d also be upset in that they claim only a few of their thousands of customers are affected, but they neglect to admit that some of those customers affected as Managed Service Providers, who themselves have thousands of customers using this software.

    There are some technical  details in this piece, in case you want to check your own systems. If you think you have multiple pieces of software that might protect you, read the article. This deployment shuts off some other products, like Microsoft Defender.

    I feel bad for many people here. IT staff at affected companies that have likely been incredibly stressed and overworked recently. The consumers of some affected customers, like those that might shop in the Swedish grocer, Coop, who shut down more than 400 stores. I don’t know the state of grocery shopping in Sweden, but this might dramatically impact many people that just want to buy food for their families.

    Ransomware continues to surprise and worry me. Large profile hacks keep coming, affecting lots of people. Often these are because of previously undiscovered software vulnerabilities or simple mistakes made by privileged users. I hope that at some point insurers and governments start to put more pressure on companies that make widely used software to ensure they are adhering to best practices and have detailed security practices in place to ensure their code is constantly checked for issues, and that they have detailed plans for responding to and patching customers when there are issues. Because, they likely will have an issue at some point.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes

  • Hostage Data

    Ransomware continues to surprise me in different ways. Recently there was an incident where data was not only encrypted, but also copied back to the criminals. In this case, Apple was the target through a supplier with the ransom note saying that without a payment, the data would be auctioned off.

    That’s crazy. Not only might you have business issues where you can’t access data, but now you have the stress of the data possibly being released or sold. While not many of us work in organizations where our data would be worth $50mm, it might be worth a significant amount, especially if the data were customer data. This alone might be a good reason to ensure that you have local data already encrypted without the keys present. At least then the criminals couldn’t read your data.

    This doesn’t help with SQL Server and TDE. In that case, the certificate would be inside the local master database, and if someone could attach it and get access to the master database, they could read your databases. If you have an SMK and a DMK, perhaps this might offer some protection, but I don’t think so. This does mean that Always Encrypted might help, unless you have lots of servers or other machines on your network with the certificates, in which case someone might be able to piece together the keys and read data.

    Attacks are becoming more numerous and creative. Having backups might have protected you against some ransomware, but not if copies of your files are sent to criminals. Perhaps the access from servers to the outside world needs to be more reigned in. Not much fun for administrators, but this might be the future of protecting systems.

    The arms race between sysadmins that protect infrastructure and criminals seems to have taken a leap forward here, and I’m not looking forward to the next step.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Incident Response Data

    I was watching a PoSh session at the PowerShell + DevOps Global Summit recently where a sysadmin had a series of scripts to run when there was a problem. One of these was Rapid Response, which gathers information from a machine(s) and stores it in a series of files. It’s a grab bag of various items, but the data can be used to help determine what’s wrong.

    Some of us have monitoring tools for our databases, and some don’t. I’m wondering, in each case, is there a set of data you want or need when an incident occurs? Do you have separate types of incidents that require disparate data? Perhaps you respond differently to performance issues than security incidents than hardware problems and want different types of data gathered.

    I know that in the past, I’ve often had scripts I ran to respond to some issues, but not others. I’ve also depending on monitoring systems (bought or built), but usually they don’t have all the information I need when something goes wrong. Capturing all the data I need in an incident is often too much to store for any length of time, but it is data that I need for specific issues. Having a series of automated processes that might start collecting data when an incident occurs, perhaps filtered based on an instance, database, user, or some other value, would be helpful. However, I think I’d need a lot of incidents to build the list of scripts myself for different issues.

    A crowd sourced series of scripts, developed by people responding to different problems, would likely be the best way to capture this information. I do see some good resources (PDF, GH) for certain types of problems, but in order for these to be useful to you, some knowledge and familiarity is needed. You need to know what scripts are useful in which situations.

    This is really the best reason for blameless RCA (root cause analysis) work after problems occur. If you have runaway blocking, constant security probes from unknown clients, or any other issue, it becomes important to analyze what happened and how people responded. Build up a protocol for how to respond and ensure that the knowledge is distributed to others that might need it. Practice running scripts and looking at information, perhaps even in a controlled replay of the problem.

    When an incident takes place, you’ll be glad you are prepared. Whether it’s small or large, a little practice will help you get through things more efficiently, and likely with less stress.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Checking Up on Vendors

    Many years ago I was in the Denver airport, waiting to board a plane. At the time I was consulting with a small startup company and they called me in a panic, having had an issue with their database deployment and needing to restore the previous version. However, they were getting the “The media set has 2 media families but only 1 are provided. All members must be provided.” error. Fortunately, I knew what was wrong and verbally walked them through the process. When I landed, I logged in and verified things were working and that they’d followed my instructions.

    A few years later I was shopping for furniture with my wife one Saturday night and I got a call from a fellow Operations person at JD Edwards. They had some security issue with a SQL Server, and since I was the SME (subject matter expert), they called me for help. Sitting in a comfy lounge chair (which I never purchased), I helped them solve the issue. Later, I verified things were working.

    In these cases, I was using remote hands from trusted individuals at an organization. However, these days, many people call cloud vendors (or other hosting companies) for the same type of remote hands support, but without necessarily having the accountability from that staff. Sure you can check if they solved an issue, but are you sure they didn’t read some data or change anything else?

    It’s a concern, but it’s one that I think can be solved with help from vendors. Microsoft recently announced there’s an audit capability for Azure SQL Servers, allowing customer to determine what actions Microsoft support engineers might take on your system. Since we may not necessarily see what is happening when someone else is working on our PaaS or IaaS systems, it’s good to have something tracking their moves. I don’t know this gathers all queries run, but it certainly should.

    I’ve seen some presentations on how customers have to authorize engineers in VMs with Customer Lockbox, and the technical details are interesting. These are time limited and audited access controls to ensure that customers are aware when a Microsoft employee touches their systems. I don’t know if this is comprehensive enough, but it is a good start.

    Over time, more and more of us have to delegate some work to others, either inside or outside of our organization. We need strong controls and good tools to ensure that we can determine what happened, especially in the event of some incident taking place. Building these controls is important, and really, they should be easily enabled in all our systems. While I don’t know we have a lot of rogue employees accessing things, we do have some, and we have more that make mistakes. Auditing their actions should be something we can easily do.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.