Voice of the DBA

Category: Editorial

  • Product Platform Teams

    Is infrastructure a product? Should an Ops team start to feel like they are a team involved in building things rather than maintaining them? It’s an interesting idea that is put forth in this blog from Forrester. The piece postulates that the Ops group should be thinking like a product team, just like developers working on an application.

    When a development group moves to a DevOps style of building things, and starts to take some responsibility for the support of their work, what does the Ops group do? If we push infrastructure as code into our VCS, is there much for the Ops group to do?

    I’d argue there still is. From the DBA side, operational staff can help determine what infrastructure is needed, or what changes need to be made. It’s easy to say that devs should add RAM or CPUs, or even another replica, but do they really know when to add them? Or do they just scale everything up? The latter might be easy on people used to writing code, but it’s not necessarily a good idea for the organizational budget.

    Ops staff can still tune systems, add indexes and rewrite code, and give feedback and knowledge to developers to help them understand how to write better code, or infra-as-code code, the first time. There’s also the need to practice and be ready for various DR situations, including the “whoops, I deleted some data”. There are skills that help handle these situations, which are far different from developer skills.

    This is in addition to designing new systems and setting templates for that infrastructure as code that the developers keep in their VCS. There’s also the need to build self-service tools, which might be where Ops becomes a product team. I think that will be necessary, as tooling is important everywhere. I think Operations might even develop other things, like reports requirements that developers don’t have time to build. Ultimately, though, there is still a need for some Ops work, albeit work that ought to be closely aligned, and shared, with developers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Data for Defense

    It’s a dangerous world out there. Many of us hear about data breaches on a regular basis, and we often blame poor security at various organizations. Certainly there are a lot of silly mistakes made, whether in configuration or the mishandling of data in insecure environments. Security is a tough business, however, and most organizations don’t have the budget to combat all the threats they face, at least not as effectively as a larger organization could.

    I have more sympathy for organizations after reading about the work of some Microsoft security teams that go up against the world’s best hackers on a daily basis. To truly understand the vulnerabilities in software, someone must also understand how to attack it. Those of us that only have a defensive mindset are inherently limited in how we design protective measures.

    There is a group, the Microsoft Threat Intelligence Center (MSTIC), that looks for and tracks hackers around the world, trying to understand how they attack organizations. This is a group of multiple teams looking at nation state attacks, as well as complex software tricks, using the tremendous amount of data they gather from telemetry to analyze the different techniques and vectors used to exploit holes in our systems. They notify customers, letting them know when a customer need to be prepared to defend themselves.

    This is a fascinating read, and one that makes me think that most of our organizations are going to be at cyberwar for the foreseeable future. Already we know that any system on the Internet is likely to be probed and attacked if it has vulnerabilities. Most of us are careful about what we expose, but we also know the weakest link in our organization could cause us issues.

    We do the best we can, but often we do depend on other organizations, like Microsoft, to actively be identifying and helping us understand what to patch or change to protect ourselves. I assume Google and Apple have similar groups, notifying their customers of potential threats. These large vendors are likely the best positioned to help, given all the data they collect about their platforms. Certainly I worry about data privacy with these vendors, but I do think that their massive troves of data on how their software works can also benefit all of us.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • What’s a forgivable mistake?

    Most of us try our best to do the best job they can each day at work. We never try to make mistakes. We might be lazy at times, or avoid some work, but we don’t try to do anything incorrectly. We might shortcut something, hoping a partial completion or quick patch is enough to satisfy a requirement, but we don’t actively do things incorrectly.

    At least I hope that’s the case for all of you. I assume there are some BOFH people out there, but they are few and far between.

    That being said, we’re human. We will have accidents, we’ll do something without thinking, or we’ll click (or type) the wrong item. Sometimes these errors go unnoticed until we can correct them. Perhaps they go unnoticed for years, and we completely forget about them.

    Other little “whoops” mistakes have widespread consequences. I’ve seen incorrect firewall rules deployed that took down entire swaths of a business. There are incorrect deployments by developers, putting out old, or even future, code that breaks things down. DBAs might change, or even truncate, the wrong data at times. That’s something I’ve certainly been guilty of in the past.

    In your job, are there mistakes you make that are forgivable? I’d hope that most are, and you don’t worry about certain tasks affecting your employment status. I’m sure there are potential mistakes that might require termination, but I’d hope these are repeated mistakes and not one time events.

    Let us know today what mistakes might be forgivable in your organization. These might not be something you’ve done, but perhaps something you’ve witnessed. How much mercy does your management have for the humans trying to manage their database systems?

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.

  • Expect the Unexpected with DiRT

    Disaster recovery is one of the core tasks that many DBAs think about on a regular basis. Ensuring that we can get our data back online, available, accessible, and intact is important. More than a few DBAs that haven’t been able to recover systems, find themselves seeking new employment.

    That’s not to say that most DBAs perform perfectly under pressure. Plenty make mistakes, and there may be times when they can’t recover all data. There does seem to be a correlation between how often DBAs practice recovery skills and how well they perform in an actual emergency. I know that at a few companies, we scheduled regular disaster tests, though often with simulated recovery of a systems that didn’t expect to actually take over a workload. Arguably not a good test, but better than nothing.

    Google takes things a step further. They have annual, company wide, multi-day DiRT (Disaster Recovery Testing) events. These are across many departments and can be substantial in terms of the disruption that the these events cause to their infrastructure. This is a way for the various individuals responsible for infrastructure to actually evaluate if they are prepared for potential issues.

    If you read the article, you find that Google started small with these and progressed them to larger, more inclusive tests, like taking down a data center. They also whitelist some servers, knowing they cannot pass a test, so there is no reason to actually take them down. After all, business still needs to work.

    It’s good to have tests and walk through actual events, like call lists and bridges to be sure that communication and documentation work. This might be especially important when teams often expect that all their written procedures are available. I went through an audit with one company, where we failed immediately when all our DR plans were on a network share. In this simulation, we had experienced a network failure and servers had crashed. We were supposed to bring up the systems on spare hardware, but some critical documentation wasn’t available without a network. We started printing things out right away so that we could continue on with the simulation (as well as have this in a binder in our office).

    Not everyone can schedule large scale tests, and certainly many managers don’t see the point. They’ll often want to gamble that staff will “figure things out” if there is an incident. That doesn’t mean that DBAs and sysadmins can afford to wait for a disaster to practice some skills. Be sure that everyone on your team can recover databases, they know where backups are (or how to determine this), and multiple people have access to resources. The last thing you want is a disaster to occur during your vacation and have managers calling you to cut short your holiday because you’re the only one that knows where something is or has the authority to access a resource.

    Think about this ahead of time and prepare.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher or iTunes.