Voice of the DBA

Tag: security

  • The Cloud Security Problem

    Your management gets a great demo from a cloud vendor and decides that the organization needs to implement the new service/application/etc. quickly. Your team tries to comply, furiously learning and experimenting with integrations, software changes, infrastructure configuration, and more. Things get deployed are working. Clients and management are happy with the new capabilities and you breathe a sigh of relief.

    After a bit of time there’s a security issue and all of a sudden there’s blame pouring down on everyone. The vendor takes a hit because it’s a public security problem, but the reality might be that your organization didn’t completely understand how to configure strong security. The public doesn’t blame your organization, but internally your team don’t know how to make changes to ensure future security.

    That’s a bit of what happened with the Snowflake customer hacks, and a good description of some of the issues is in this piece from Joey D’Antoni. Snowflake didn’t necessarily have bad security, but they allowed customers to have bad security and also limited the options for customers to implement stronger security.

    I think about this all the time as I look at the challenges of security that many organizations face. In my decades of working in different situations, the one thing I know is that pressure to move fast often creates security shortcuts. Much of the early security problems with SQL Server could easily be traced to a) allowing installs with no password, b) developers not understanding the security model and granting sysadmin (or using sa) in their applications, and c) management agreeing that this was OK because a deadline needed to be met.

    In all these situations, workers had the best of intentions to go fix things later, but there was rarely the time or energy to do so. As someone who forced apps to change away from sa in a large org, and required separate accounts and passwords for servers, I can tell you no one liked me and I got a lot of pressure to leave things alone. That is until I could explain the risks to managers with security people present. Even then there was no shortage of people who wanted me to assume the risk of apps using sa and let things be.

    Cloud security is pretty good from the major vendors. Most of the smaller co-location facilities I’ve worked with in the past also had good security. It was the clients that caused problems, often because of a lack of knowledge or the desire to hurry.

    Joey says Entra is a great system. I agree, and I love the SSO capabilities I’ve seen implemented. I also know that trying to set things up and configure them has been difficult for me, and I think I can be pretty sharp about lots of technology. Joey has patiently answered many of my ignorant questions because I didn’t understand how some part of Entra (AAD) works. I think lots of tech people don’t understand this and don’ t necessarily have a Joey to call on.

    Most of us working in technology need more security education, better habits, and the patience to implement strong security. That includes management. At the same time, I wish that the solutions out there were easier to understand, or maybe better explained for those of us who need to use some portion of the authorization and authentication systems in our software.

    Unfortunately, security software is still software and those vendors push out changes and updates quickly as well, leaving education and documentation to the user. This is a bad situation that continually arises with regular issues in many organizations.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Everything is Code

    I posted a note on Twitter/X with this quote: “The content updates had not previously been treated as code because they were strictly configuration information.” This is from testimony given by Crowdstrike to a US Congressional committee in trying to explain how they grounded much of the airline industry a few months ago. That was a mess of a situation, and apparently, the vendor didn’t think their configuration was part of their code.

    That’s an amazing viewpoint to me. The fact that any developer or manager thinks that their configuration data isn’t a part of their code is worth testing. Yet, I see this attitude all the time, where developers, QA, managers, and more think that the code is the only thing that changes or doesn’t change, ignoring the fact that there are configuration items that affect the code and need to be managed appropriately. Certainly, if the config data were in enums rather than in a file or database they’d feel differently.

    I think part of the reason that people try to ignore config data is that it is hard to manage. Often config data might change between dev, test, and prod. Dealing with that, and testing appropriately is hard. I haven’t ever seen a good solution for getting data into an environment the first time. That’s the hard part. Once the data is there, you can use it as a token where it is needed, and hopefully, the value has already been tested. At the very least, you can test how that data affects that environment.

    I am glad to see Crowdstrike publicly recognize that they need to dogfood not only their code changes but also their config changes. However, for a company that hasn’t shown a rigorous engineering approach, I suspect they’ll test very simple and basic config changes and not necessarily do a good job of carefully testing a variety of potential problem vectors. That takes work, and excluding config data from testing is a sign (to me) of a technology group trying to avoid doing too much work. It’s likely more management and leadership than technology workers, but the entire organization is showing signs of shortcutting good engineering.

    My view is that developers should be free to experiment and try lots of things, and have a lot of freedom on how they build software. I think the same thing for infrastructure people as well. However, as we start to move our changes towards production, everything should be in code, version-controlled, and promoted through PRs. In other words, get everything stored as code and nothing gets changed outside of development. It only gets approved to move forward or rejected, after it’s well tested.

    That’s a tough process to implement, and one many companies don’t spend the time doing, but for those that do, they end up deploying many  fewer bugs.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • No Backdoors

    Every once in awhile I hear about someone in law enforcement sure that tech people can build in a safe, secure way for data to be unencrypted by the company or vendor. The latest appears to be from Australia, where the Security Intelligence Organization wants tech companies to build this into products.

    Backdoors never work. Anytime an encryption key is stored, it could be stolen. We see this all the time. Keys are just data, and companies lose data all the time. At scale. Governments are certainly not immune from this. One of the reasons that Azure allows a BYOK (bring your own key) for encryption mechanisms is that many organizations don’t want to trust Microsoft to store their keys. I’m guessing Microsoft doesn’t want the liability, either.

    Many of the data protections an organization might implement are outside the scope of data professionals, but we certainly have responsibilities in this area. We should certainly manage our keys appropriately, and my recommendation for many people is that they have a separate repo and pipeline for the deployment of privileged objects, such as encryption keys. Your organization also ought to have a process (and test it) for key rotation. Keys expire relatively infrequently and the person who last rotated the key might not be available when you realize this process needs to be completed.

    It’s especially important everyone knows how this works for those emergency situations where something expires and none of you realize there’s a problem until a client calls.

    There are plenty of other security mechanisms we ought to be using. Secure your backup files and limit access to shares or even processes that move these files around. In general, limit access to those who need access to everything. Using organizational groups is the best way to do this, along with regular audits to ensure that those who change jobs are removed from groups that aren’t needed. I haven’t seen any organization that has a good process for knowing what positions need what access. I often only see people given new access for a role change, giving them the new roles that match some other employee. This is usually done without any previous access removed when it is no longer needed. The most senior people often have the most access, not because they need it, but because they keep getting new roles.

    Managing security with roles too granularly can become a nightmare, though ensure you use roles everywhere you can while trying to limit the numbers of roles. For most databases, we are giving access to nothing or everything, but there are reasons to limit access for certain data. You might consider two roles by default in most places: one for privileged users and another for everyone.  Easy to ensure new objects get grants to one or both roles and let access be managed by role membership.

    Keep it simple, but keep it secure.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

    Note, podcasts are only available for a limited time online.

  • Analyzing Breached Data

    A few of you out there might be data scientists who profile data regularly. Probably a fair number of you do import/export work and learn to check data values, perhaps with counts, distincts, or other aggregates. I don’t know if the performance tuners out there look at the skew of data or the details of what is in a query that needs improvement. However, all of you are likely familiar with data and trying to query it for some type of meaning.

    One of the largest data breaches occurred with National Public Data. Troy Hunt analyzed the breach as a part of his work with haveIbeenpwned. The piece is an interesting analysis of the data, trying to determine both it’s legitimacy as well as what is actually included in the breach. It’s a fascinating read and I encourage you to look at it not just from the data analysis side, but also to be aware of what data about you is being aggregated and sold by companies.

    The read is interesting as it is a bit of a detective story, digging through data in a folder, which is something I’ve had to do. I’ve had people in previous jobs just dump a bunch of data on me and ask me to load it into a database. Or a table. Often without them knowing what type of data it is, what formats, do files relate to each other? Are there multiple tables worth of data in a file? All questions I’ve had to ask myself (and answer), and similar to what Troy did to analyze the breach.

    Data is very important to many of us, in different ways, but I’m often amazed at how few people actually understand how to organize data and ensure others can track the metadata about their data (what their data represents). I’m guessing this is why every person that gets an extract of data to load into Excel formats it in different ways.

    In many cases, people want the ability to query data, but they prefer to just focus on one table that contains a lot of information. They don’t want to know how to “join” data together. I think this might be the reason we see so many views in databases, and why we have views built on views. Each new client of the database needs their own view structure.

    The world of data is a mess, even inside an organization. Once we start moving data between organizations, it’s truly a mess. We might bemoan all the inefficiencies and work we do to move, change, and re-load data as custom, human ETL machines, but there is one great thing about this tangled web. It provides for steady, secure jobs for many of us with no end of work in sight.

    Steve Jones