Voice of the DBA

Tag: security

  • Reducing Debt and Increasing Security

    Many of us know that reducing financial debt in our lives leads to a bit more security in our minds. We are better able to cope with unexpected expenses when we have the ability to get credit or pay for something in cash. If you’ve ever had a major house repair, car calamity, or (hopefully not) medical issue, you know what I mean. At least in the US. That last one is less of an issue in many other countries.

    We know that our code has technical debt, and often we fight a battle with project managers and business people over whether we should spend time paying down that technical debt or developing new features. We certainly can do both, but often that’s not a choice we are given. If we do get the chance to reduce technical debt, it’s a tiny fraction of the time that is allocated to adding something new.

    There might be a better argument for reducing some technical debt: security. There’s an article that talks about the security risk posed by moving too fast and not ensuring there is rigorous security testing. This is a common problem in many companies that adopt some parts of DevOps, but forget that quickly deploying new code isn’t the only principle. We also need to learn from our mistakes and quickly correct them.

    I am a big fan of releasing often, especially for security purposes, but those releases need to build on each other with not only new code but refactored old code that doesn’t perform well or has a security issue. More importantly, we need to ensure when we find issues in code, that all developers are aware of the problem and don’t repeat the issue. We also need to update our testing to ensure that humans don’t continue to write code in a style that causes known issues. We want to avoid antipatterns in our code.

    This is less of a developer issue than a managerial one. The C-suite and all management that are involved in software need to prioritize security so that known issues are fixed quickly. This also should involve ensuring that technical debt is kept under control as a lot of this poorly written code increases the chances that a vulnerability is in the code. Either because the code isn’t well written or because developers avoid touching the code in the future.

    The best solutions, to me, involve DevOps. Make small changes, deploy them, instrument them, and learn if they work well. If not, fix them quickly. While this might end up delaying some features, ultimately it ensures that our software is in better shape, with a reduced risk of simple security mistakes upsetting our customers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Republish: It’s Not Yours

    I’m still at Techorama.

    Check out It’s Not Yours from the past. For some reason I didn’t get it on this blog, so here it is:

    It’s Not Yours

    For most of my career, I’ve felt a tie to the systems I’ve managed or the code I’ve written. I’ve often said that a part of the code was mine, because it was something I’d written. Having some ownership of your work has often meant more pride in ensuring things are well implemented. The same thing goes for administering databases. When an employee feels personally responsible and accountable for  the database and its contents, usually they do a better job.

    Most of us know that the systems we manage and the data they contain aren’t really “ours”, no matter how we might refer to them in conversation. We know that ultimately the data, applications, and hardware belong to our employers. Or, we should know that.

    Apparently not everyone does. A contractor just pleaded guilty to stealing classified information from the US National Security Agency, about 50GB worth of data. This isn’t a case of spying or malicious intent. Instead, the report is that the employee was a hoarder, just keeping copies of data for some unknown reason. Hopefully that’s true.

    Many of us will have copies of data that we use for different purposes. We might be building software, or testing queries, or some other task that requires data. We might want to conduct more in depth analysis in our spare time. Those are all positive goals, but none of them mean that we should have copies of data without authorization. We don’t own the data, and we should always be sure if we are moving data to other systems, that those systems are secure and someone else has approved the action.

    This becomes more important all the time. We have enough issues with data security and keeping our information away from hackers. We make plenty of mistakes with configuration in authorized development and test systems. There’s no reason to add to the issues by treating data as our own personal asset, no matter how good your intentions.

    Steve Jones

  • Project Zero for Better Security

    Google started Project Zero to study zero-day vulnerabilities in systems. They want to improve security and safety, at least from a software and hardware standpoint. Its mission is to make it more difficult to find and exploit issues in software. They published an update recently that showed vendors patching their software quicker, at least according to data from the last few years.

    The metrics show a change in patching from an average of 80 days a few years ago to 52 days in 2021. There were also patches that missed the 90-day deadline, though only one missed the 90-day time and also a 14-day grace period. They are also trying to help with the understanding of how well vendors are adapting to new challenges. I hope this pressures and inspires smaller vendors and even individual organizations to take security more seriously.

    I think that security is becoming *slightly* more important to vendors, especially as competition grows in any particular space. Vendors aren’t necessarily looking to spend time and energy where they don’t see a problem impacting revenue, but they do worry about customers abandoning their software, and a security issue is one reason for customers to look elsewhere. In the 2020s, there are plenty of customers that would consider changing software over security issues that don’t get fixed.

    I saw another note that showed more and more hackers are exploiting zero-day vulnerabilities. Even if you think you have strong security, who knows when someone will misconfigure a firewall and expose your system. It’s worth staying up to date, at least within the last patch for your critical systems. I certainly consider a database a critical system, and I’d include helper databases like Redis and Elasticsearch as critical systems.

    It’s easy to delay patching when nothing seems to be broken. However, patches are like oil changes. When the system lets you know that maintenance is overdue, it’s usually a catastrophic event.

    I like the disclosure and openness of groups like Project Zero and hope they start to pressure more software developers (and project managers) to promote secure coding along with quick patches for vulnerabilities.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Advanced Incident Response

    Early in my career, I worked at a few smaller companies where a problem with the computer systems meant I went into the office and stayed until things were working. As I went to a few larger companies, I wasn’t alone when a system went down, but the process was mostly the same. We figured out what was wrong and found a way to fix or replace things, occasionally with help from a vendor. Those were the (not necessarily) good days before our internal networks were connected to a public Internet.

    Companies developed formal incident response plans to deal with various issues, whether these were problems we caused ourselves or failures of an application. I had the fortune, or misfortune, to be involved in more than a few issues and learned a great deal in how to solve problems as well as how to manage the impact to a large number of employees.

    As email and Internet use grew, so did the attacks with viruses and other sorts of malware. Antivirus software helped a great deal, but these days ransomware seems to be a common problem that isn’t as preventable as I would have hoped. Quite a few friends have dealt with ransomware issues, most of which have not been widely reported in the news.

    I saw an article about a few things that you might want to consider adding to your incident response plan. While some of these items might be unique to the ransomware threat, the thing that struck me was that there is a need to react quickly, in real-time, in response to any detection of an issue. I can only assume this means that there needs to be some advanced monitoring of nodes to detect issues, and I’m not sure how many organizations would adopt this, but in today’s world where we want systems available 24/7, perhaps they will.

    Being on call is a part of working in many IT departments. Having a response plan, even the general outline of one, helps to coordinate resources and ensure that we use people effectively. Tools are important, especially in today’s complex world, and it is important that one of those tools is a simple backup, preferably air-gapped from your main systems. If you don’t have these things in place, you might suggest someone start assembling them. These days it seems it’s not if you will get attacked, but when.

    Lastly, I don’t often see this addressed in plans, but make sure you have spelled out some guidelines on rotating staff and getting rest. Far too many companies want “all hands on deck” and forget that normal business still needs to occur. Any incident could last longer than a day, and you want to ensure that some of your staff is fresh and ready to take over from those that do need rest. Don’t be afraid to send some people home, or better yet, don’t call them in the first place.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.