Voice of the DBA

Category: Editorial

  • Reassess Your Configuration

    It’s the start of a new year, and while it might feel like COVID-19 has us stuck back in March 2020, I hope that will start to fade as we move into the new year. Certainly our employers want to move forward, and I suspect that there is some hope that organizations will start to thrive in 2021.

    I saw a post from Kenneth Fisher recently that talked about an end of year, new year checklist. It’s a basic set of things that you might do as a type of review and cleanup of some of the instances and databases that you manage. Checking things like end of year processes and performing some review of potential items that might no longer be needed.

    In the past, I didn’t do this type of work at the end of the year, or even that often, but when there were slow times, I did have a list of things to periodically review. We had canned queries and reports to help us evaluate resources. Many were automated to ensure the data about systems was available if I had time.

    One area that I was check once or twice a year was the configuration, trying to determine if any hardware had changed, the workload had altered, or there were proactive configuration changes we could make. This might include removing unused security logins or turning off services that weren’t being used.

    Certainly I was always cognizant of the load that managing lots of databases can have on a DBA. Every additional database is a potential customer that can call or file a ticket. I couldn’t often remove any, but I did find ways to remove unused objects and archive data at times. Reducing resource usage often pays itself back over time, especially when the need for more storage arises, which was more of a battle than it needed to be.

    I might add one thing to Ken’s list. I would ensure that you should check on cloud resources that might not be used. One thing I see too often is that cloud resources are easy to spin up and leave up, even if they become unused. Finding a few of these might earn you some points with the people paying the bills.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Responsibility for Bugs

    Most software has bugs. Actually, maybe all software has bugs, but often we can live with some of them. Over time, vendors regularly issue patches to fix things, and despite these issues, many of us continue to use software productively, if not happily. We live with some workarounds or issues, and continue to get work done, entertain ourselves, and the world keeps moving forward.

    Recently a software developer issued a full refund to customers that had been running into issues with their software, in this case a game. Apparently this is more of a platform issue, as the early reviewers saw the game on a PC, and many of the complaints are on dedicated consoles. The developer of the game has promised patches to fix the main issues, but if customers want a refund, they can get one.

    I’ve had issues with plenty of software in the past, as vendors release a new version that might not have been tested as well as it should. I’ve also seen plenty of scale issues, where software is tested or architected for a certain scale, but customers use it at larger scales. My work with large enterprises, showed the latter to be the case more often than I’d like.

    In the case of the game, the public outcry can cause lots of changes, mostly because the goodwill and reputation of the vendor are at stake. However, in many enterprise software sales, there isn’t a public disclosure of issues that prevent other customers from learning of issues. Part of this is a lack of detailed public reviews of software, and part of this is that users don’t often share specific issues outside of their organization.

    I wish more vendors were held accountable for bugs, and with responsibility to patch and mitigate them. However, I also see that software is used in so many different ways, with many configurations, that it can be hard to comprehensively test everything. I also know that there are bugs that very few customers hit, and the cost to deal with these can be high.

    Ultimately, I think I’d like to have more transparency required about which bugs customers hit and in which environments. I’d also like to have some requirement that some percentage of bugs must be fixed within a timeframe or customers entitled to some compensation. I know this might lead to many simple bugs fixed, but perhaps it would also pressure software vendors to improve their quality a little more than they do today.

    Steve Jones

  • The Best Way to Protect Sensitive Data

    I was listening to someone talk about data privacy recently, and the ways that you can protect the sensitive information in your databases. They had a great quote about something you might consider. They said, “The best way to protect data is not hang onto the raw data at all.”

    If we don’t have sensitive data, then a loss of data can’t occur. Hacks won’t cause issues, we can’t accidentally send out data or leave it lying around. There’s a good case to be made that keeping less sensitive data around is a good idea.

    For some applications, we can’t avoid keeping sensitive data. Medical databases keep private health information. E-commerce systems likely need financial information. Many of us will definitely have to deal with some sensitive data, and protect it, but we can minimize our struggles.

    We often don’t have a good reason for keeping lots of data around. Lots of queries run by users end up looking at only a small portion of data. Often recent data is needed, and some aggregates for older data, but we don’t actually look at the details of old data often. We may even have older data around that we’ve forgotten about, and our users don’t even know is available.

    We certainly don’t often need sensitive data in non-production environments. Plenty of people use scripts or tooling to obfuscate, anonymize, generate, or otherwise ensure sensitive data isn’t in unprotected environments. We can archive, or even change, old data to ensure it isn’t a liability. We can even do this in production, preserving metrics, but delinking data from any individual.

    I’ve always been someone that kept more data than necessary, just in case I needed it. However, over time, I find that the costs, and the potential risks, just aren’t worth it. Moving forward, archival, anonymization, and other strategies need to be a part of any system I manage.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • A Need for Monitoring without Administration

    There was a report recently that a number of US government agencies were hacked through a network management system. Apparently Solarwinds had their code hacked, and this resulted in a backdoor being distributed to customers via software updates.

    There is a lot that went wrong here, and this ought to make many system management software vendors very nervous. Attacks on your software developers, designed to allow a hacker to put backdoors into source code repositories is a wild second (or third) order attack. I would certainly be nervous to be a software developer right now, and be extra cautious about any sort of potential phishing email sent to me. Yes, that’s a thing.

    The bigger issue, to me, is that monitoring most systems ought to be possible without requiring escalated privileges. While there are some ways to get metrics without requiring administrative rights, most OSes and most administrative and monitoring software expects to have complete rights to all resources.

    That’s a hole in design. There are plenty of cases where we want monitoring data (and alerts/notifications) distributed to other automated systems or to interested individuals, but we don’t want to expand the number of administrators. Every additional individual or system that can potentially change something as an administrator is another potential attack vector.

    We have built our core operating systems with the idea that someone needs complete control of the system to work with it. For some things, that’s true, but for resource usage, especially in the way that many of us need to watch at scale, I’m not sure that this needs to be the case. My view is that Windows, MacOS, and Linux ought to undertake fundamental design reviews to determine if they can further shrink the scope of privileges for monitoring systems.

    In the meantime, granting privileged access to an automated system for monitoring ought to be done very carefully, even more carefully than for human sysadmins. This account will run by itself, and someone might not notice if it is compromised. Set strong, very long passwords, change them periodically, and audit the account to be sure it is only accessing what you think it should access.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.