Voice of the DBA

Category: Editorial

  • The Responsibility for Bugs

    Most software has bugs. Actually, maybe all software has bugs, but often we can live with some of them. Over time, vendors regularly issue patches to fix things, and despite these issues, many of us continue to use software productively, if not happily. We live with some workarounds or issues, and continue to get work done, entertain ourselves, and the world keeps moving forward.

    Recently a software developer issued a full refund to customers that had been running into issues with their software, in this case a game. Apparently this is more of a platform issue, as the early reviewers saw the game on a PC, and many of the complaints are on dedicated consoles. The developer of the game has promised patches to fix the main issues, but if customers want a refund, they can get one.

    I’ve had issues with plenty of software in the past, as vendors release a new version that might not have been tested as well as it should. I’ve also seen plenty of scale issues, where software is tested or architected for a certain scale, but customers use it at larger scales. My work with large enterprises, showed the latter to be the case more often than I’d like.

    In the case of the game, the public outcry can cause lots of changes, mostly because the goodwill and reputation of the vendor are at stake. However, in many enterprise software sales, there isn’t a public disclosure of issues that prevent other customers from learning of issues. Part of this is a lack of detailed public reviews of software, and part of this is that users don’t often share specific issues outside of their organization.

    I wish more vendors were held accountable for bugs, and with responsibility to patch and mitigate them. However, I also see that software is used in so many different ways, with many configurations, that it can be hard to comprehensively test everything. I also know that there are bugs that very few customers hit, and the cost to deal with these can be high.

    Ultimately, I think I’d like to have more transparency required about which bugs customers hit and in which environments. I’d also like to have some requirement that some percentage of bugs must be fixed within a timeframe or customers entitled to some compensation. I know this might lead to many simple bugs fixed, but perhaps it would also pressure software vendors to improve their quality a little more than they do today.

    Steve Jones

  • The Best Way to Protect Sensitive Data

    I was listening to someone talk about data privacy recently, and the ways that you can protect the sensitive information in your databases. They had a great quote about something you might consider. They said, “The best way to protect data is not hang onto the raw data at all.”

    If we don’t have sensitive data, then a loss of data can’t occur. Hacks won’t cause issues, we can’t accidentally send out data or leave it lying around. There’s a good case to be made that keeping less sensitive data around is a good idea.

    For some applications, we can’t avoid keeping sensitive data. Medical databases keep private health information. E-commerce systems likely need financial information. Many of us will definitely have to deal with some sensitive data, and protect it, but we can minimize our struggles.

    We often don’t have a good reason for keeping lots of data around. Lots of queries run by users end up looking at only a small portion of data. Often recent data is needed, and some aggregates for older data, but we don’t actually look at the details of old data often. We may even have older data around that we’ve forgotten about, and our users don’t even know is available.

    We certainly don’t often need sensitive data in non-production environments. Plenty of people use scripts or tooling to obfuscate, anonymize, generate, or otherwise ensure sensitive data isn’t in unprotected environments. We can archive, or even change, old data to ensure it isn’t a liability. We can even do this in production, preserving metrics, but delinking data from any individual.

    I’ve always been someone that kept more data than necessary, just in case I needed it. However, over time, I find that the costs, and the potential risks, just aren’t worth it. Moving forward, archival, anonymization, and other strategies need to be a part of any system I manage.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • A Need for Monitoring without Administration

    There was a report recently that a number of US government agencies were hacked through a network management system. Apparently Solarwinds had their code hacked, and this resulted in a backdoor being distributed to customers via software updates.

    There is a lot that went wrong here, and this ought to make many system management software vendors very nervous. Attacks on your software developers, designed to allow a hacker to put backdoors into source code repositories is a wild second (or third) order attack. I would certainly be nervous to be a software developer right now, and be extra cautious about any sort of potential phishing email sent to me. Yes, that’s a thing.

    The bigger issue, to me, is that monitoring most systems ought to be possible without requiring escalated privileges. While there are some ways to get metrics without requiring administrative rights, most OSes and most administrative and monitoring software expects to have complete rights to all resources.

    That’s a hole in design. There are plenty of cases where we want monitoring data (and alerts/notifications) distributed to other automated systems or to interested individuals, but we don’t want to expand the number of administrators. Every additional individual or system that can potentially change something as an administrator is another potential attack vector.

    We have built our core operating systems with the idea that someone needs complete control of the system to work with it. For some things, that’s true, but for resource usage, especially in the way that many of us need to watch at scale, I’m not sure that this needs to be the case. My view is that Windows, MacOS, and Linux ought to undertake fundamental design reviews to determine if they can further shrink the scope of privileges for monitoring systems.

    In the meantime, granting privileged access to an automated system for monitoring ought to be done very carefully, even more carefully than for human sysadmins. This account will run by itself, and someone might not notice if it is compromised. Set strong, very long passwords, change them periodically, and audit the account to be sure it is only accessing what you think it should access.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Devil is in the Details

    Some of us have run into perplexing technology  problems, where we had to dig deep into an application to solve a problem. We might need to work with our own staff, vendor support, perhaps even coordinate people across multiple different organizations. This can be even more challenging when we don’t have access into the internals of all the code.

    I ran across a neat story from Netflix, where an engineer had to dig into an issue with one of their partners. In this case, there were hardware and software components, and four different companies involved. The problem involved a playback issue, with deadlines and finger pointing over the issue.

    Eventually the root cause discovered was a thread level issue in the Android OS, but the tale of how the engineer asks questions, looks at good, sets up tests, and more is a good examine of how to dive into an issue. While many of us wouldn’t get involved in threads on the data platform, we do need to understand the code and use metrics and data to narrow down the issues. It’s certainly possible we could discover a bug, but most of my experience is that I’ve found a problem in developer code or insufficient hardware resources.

    The story is a nice read, and the final paragraph made me smile It had this quote: ” This story really exemplifies an aspect of my job I love:…” It’s always inspirational to find someone truly enjoying their job, and I find myself thinking about the things I love about my job, with problem solving being one of the best parts.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.