Voice of the DBA

Tag: security

  • Incident Response Data

    I was watching a PoSh session at the PowerShell + DevOps Global Summit recently where a sysadmin had a series of scripts to run when there was a problem. One of these was Rapid Response, which gathers information from a machine(s) and stores it in a series of files. It’s a grab bag of various items, but the data can be used to help determine what’s wrong.

    Some of us have monitoring tools for our databases, and some don’t. I’m wondering, in each case, is there a set of data you want or need when an incident occurs? Do you have separate types of incidents that require disparate data? Perhaps you respond differently to performance issues than security incidents than hardware problems and want different types of data gathered.

    I know that in the past, I’ve often had scripts I ran to respond to some issues, but not others. I’ve also depending on monitoring systems (bought or built), but usually they don’t have all the information I need when something goes wrong. Capturing all the data I need in an incident is often too much to store for any length of time, but it is data that I need for specific issues. Having a series of automated processes that might start collecting data when an incident occurs, perhaps filtered based on an instance, database, user, or some other value, would be helpful. However, I think I’d need a lot of incidents to build the list of scripts myself for different issues.

    A crowd sourced series of scripts, developed by people responding to different problems, would likely be the best way to capture this information. I do see some good resources (PDF, GH) for certain types of problems, but in order for these to be useful to you, some knowledge and familiarity is needed. You need to know what scripts are useful in which situations.

    This is really the best reason for blameless RCA (root cause analysis) work after problems occur. If you have runaway blocking, constant security probes from unknown clients, or any other issue, it becomes important to analyze what happened and how people responded. Build up a protocol for how to respond and ensure that the knowledge is distributed to others that might need it. Practice running scripts and looking at information, perhaps even in a controlled replay of the problem.

    When an incident takes place, you’ll be glad you are prepared. Whether it’s small or large, a little practice will help you get through things more efficiently, and likely with less stress.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Checking Up on Vendors

    Many years ago I was in the Denver airport, waiting to board a plane. At the time I was consulting with a small startup company and they called me in a panic, having had an issue with their database deployment and needing to restore the previous version. However, they were getting the “The media set has 2 media families but only 1 are provided. All members must be provided.” error. Fortunately, I knew what was wrong and verbally walked them through the process. When I landed, I logged in and verified things were working and that they’d followed my instructions.

    A few years later I was shopping for furniture with my wife one Saturday night and I got a call from a fellow Operations person at JD Edwards. They had some security issue with a SQL Server, and since I was the SME (subject matter expert), they called me for help. Sitting in a comfy lounge chair (which I never purchased), I helped them solve the issue. Later, I verified things were working.

    In these cases, I was using remote hands from trusted individuals at an organization. However, these days, many people call cloud vendors (or other hosting companies) for the same type of remote hands support, but without necessarily having the accountability from that staff. Sure you can check if they solved an issue, but are you sure they didn’t read some data or change anything else?

    It’s a concern, but it’s one that I think can be solved with help from vendors. Microsoft recently announced there’s an audit capability for Azure SQL Servers, allowing customer to determine what actions Microsoft support engineers might take on your system. Since we may not necessarily see what is happening when someone else is working on our PaaS or IaaS systems, it’s good to have something tracking their moves. I don’t know this gathers all queries run, but it certainly should.

    I’ve seen some presentations on how customers have to authorize engineers in VMs with Customer Lockbox, and the technical details are interesting. These are time limited and audited access controls to ensure that customers are aware when a Microsoft employee touches their systems. I don’t know if this is comprehensive enough, but it is a good start.

    Over time, more and more of us have to delegate some work to others, either inside or outside of our organization. We need strong controls and good tools to ensure that we can determine what happened, especially in the event of some incident taking place. Building these controls is important, and really, they should be easily enabled in all our systems. While I don’t know we have a lot of rogue employees accessing things, we do have some, and we have more that make mistakes. Auditing their actions should be something we can easily do.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Ransomware May Get Worse

    I’ve never had to deal with ransomware, but across the last couple years I’ve been surprised how many friends and customers have dealt with this. For those that are prepared, it’s a time sink and a hassle. For those that aren’t, there is the addition of finger pointing, anger, fear, and often lost data as well.

    I hope things don’t get worse, but I think this might be a sign they are. There is some speculation that criminal groups are working together, in a sort of computer hacker cartel. Apparently four criminal groups announced they would be working together. If that isn’t a sign of the craziness of the world, I don’t know what else would be. Criminal groups announcing they are collaborating? Apparently they must think they are businesses like the targets they attack.

    There has been some investigation, and there haven’t been signs this is a cartel, with no revenue sharing or coordination, but the fact that they are cross posting data and sharing techniques is bad enough. Helping each other out may help them find more targets and develop better techniques to get around security.

    The one concern I’d have with all of this is that these different groups may attack the same companies. Can you imagine getting through an attack, either paying a ransom or not, only to be attacked again? I do think that organizations ought to be worried about their security against ransomware, have good, air-gapped backups, and ensure if they are attacked, they plug any holes in their systems.

    Surviving one attack might be hard for an organization, but if there are two or more, I suspect someone’s career will be in trouble.

    Steve Jones

  • Removing SA from Azure SQL Database

    One of the recommendations from Microsoft SQL Server is to use Windows authentication. This has been in the docs for years, and I’ve heard many MS consultants and employees note this. Many customers and clients have tried to use Windows Authentication only, but often in a cross platform environment with Java or Linux clients, one usually has needed SQL authentication with a user and password. Client libraries have been enhanced so this isn’t necessary, but still some people prefer SQL authentication, especially with clients outside their organization. It’s simple, easy, and developers can make it work in seconds.

    With Azure SQL Databases, some companies defaulted to a username and password, as their Active Directory (AD) wasn’t extended to Azure. That has become easier to do, and many people are taking advantage of it. In fact, some customers are so integrated, they want to do away with usernames and passwords in Azure.

    Microsoft has listened, and is giving them the option. The feature is in preview, but if you enable this, SQL auth is turned off, which means whatever administrative account you set up for the server with a name and password will not work. That’s essentially the “sa” account, though with your own custom name.

    While this feature won’t be useful for everyone, it’s a good option to have. As more companies look to tighten security and limit the attack surface area, being able to make this choice is important. It’s also something that architects and administrators should be aware of and consider in their decisions on how to implement applications in Azure.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.