Voice of the DBA

Tag: DevOps

  • Rogue Colleagues

    The economy might be good or bad for you right now. Some of that depends on where you live, what your employment situation is like, what your habits dictate about how you live life, and more. No matter what your situation, likely there are people around you that complain about the world and others who think things are fine. There are likely more of the former than the latter, but that’s because humans tend to complain out loud more than they praise.

    When people think there is an economic downtown for themselves, they may be more likely to engage in malicious activities. While I don’t think most data professionals will start to hack other systems, or even their own employer’s systems, there is evidence to support the idea that some might be susceptible to recruitment by bad actors. This piece references some research and warns security groups to be wary.

    There is no shortage of books, or television and movie scripts that might show creative ways to access information, but how can you tell if a colleague makes a simple mistake or they are a bad actor? Clicking on a phishing email could be either one. Not removing anonymous access to an S3 bucket could be either. Losing their credentials through social engineering is something that happens every day. Who’s to say that this happened purposefully?

    I don’t want to second guess the people I work with making mistakes, but I also think these possibilities are why we want to use our computer systems with strong auditing and multiple groups reviewing logs. We might not necessarily stop all activity, but we can often detect it quickly and mitigate the issues. It’s also why DevOps and automated deployments with logging are a good idea. They can limit the problems from both accidents and malicious actors.

    My employer has started to do more education around security and how individuals can avoid accidentally causing issues. We use a lot of automation, and more all the time, that ensures once we know how we ought to patch and update systems, we can do it regularly and confidently. Repeatable, reliable deployments of changes are what we aim for.

    We know they’ll be some mistakes, but we also know that we can quickly identify issues (MTTD) and fix them (MTTR). Even if we get a bad patch from a vendor, we can quickly deploy a “fix” if we get one, or even reinstall and re-patch to lower levels, if needed.

    DevOps, GitOps, and other xxOps aren’t just about getting new features out quickly. They also include the ability to fix problems when the need arises. They don’t prevent rogue actors from causing issues, but they should help you detect and recover quicker than you might expect.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Shift Right

    Earlier this year I was watching someone present on DevOps and IaaC (Infrastructure as Code). The speaker was showing how they had worked with clients to implement tests and checks that evaluated whether their systems were deployed and the code was working as expected. They did this as the CI/QA environments weren’t maintained, but rather spun up as needed when code was committed or tested.

    What I found fascinating was the part in the talk where the speaker said they were “shifting right.”

    That had me sitting up and paying closer attention. After all, for ten years I’ve been working in DevOps where we try to shift everything left. Shift testing to the developers, shift infrastructure into the repository, shift pen testing into some automated CI stage. We want to know about issues sooner. What is this shift right stuff?

    It turns out that the speaker was using this as a way of smoke testing. They didn’t completely trust that CI/QA were configured the same as production. They were performing many of the same tests in production after a deployment. In a few instances, they’d spin up a copy of production, deploy changes, and then run tests against that. They wanted to get a real-life view of how the system worked.

    For some parts of the infrastructure, this seemed like overkill. After all, if I’m updating an app and perhaps adding a web server to check, verifying this was the same size as a previously deployed web server seems silly. However, they included other checks like security. Was the new, or even old, server configured with the same security that was required by the org? They had policies around various authentication and authorization items and these were constantly rechecked. Someone alter a file share or enable anonymous access? If this was against policy, someone was notified when the next deployment caught this.

    What was interesting about this methodology was that deployments weren’t often failed. Instead, when something failed, there was a report sent to a group and the incident was logged, but the deployment still occurred. That’s an approach that I actually like. Don’t stop things, but make sure someone follows up. Of course, like many nagging problems, if this happens constantly or it can’t get fixed, then this isn’t helpful information.

    I haven’t thought much about the idea of shifting things right, but I have advocated for the same ideas. Create smoke tests that evaluate if things still work after deployment. Get feedback to people right away. Much like a unit test, I’d want to know if simple things are broken. I’d also approach these tests the same way. Don’t write a lot to start, but every time there is an issue, add a unit test and a smoke test to prevent regressions.

    And be careful of which frameworks you use. Some database unit testing frameworks alter tables, which can’t happen in production.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The 2023 Redgate UK DevOps Roadshow

    Years ago Redgate did some traveling events under the SQL in the City brand. These were a lot of fun and kind of amazing. One of the longer tours also made me realize I would hate being in a musical band and touring.

    However, I enjoyed the events and while we’ve moved on from that brand, we have a new one

    The Redgate DevOps Roadshow

    The UK edition of this kicks off in September 2023 and I’ll be in these cities:

    I’ll be traveling with one of our Sales Engineers (Chris Hawkins, I hope), and doing will day sessions for customers on the Flyway suite of tools. You’ll get some practice, learn how things work, and get answers to your questions. Hopefully that day, but if Chris and I don’t know, we’ll query the brilliant engineers back in Cambridge.

    If you’ve ever attended a Redgate event, you know we put on a nice show, we take care of you, you’ll have plenty to eat and drink, and we have fun.

    Hopefully I’ll see some of you in September in the UK.

    Keep watching as well as we’re working on a US edition of this. I’ll do some of these, while Grant and Ryan will do others. I’ll keep my schedule updated as I get more info.

  • Talking about the New Open-Source SQLCMD on Data Exposed

    I was on the Data Exposed: MVP Edition show recently, talking about SQLCMD. I’ve written a few articles on the topics as well, and a blog post about setting up a node HTTP server, which I show in the demo.

    Check out the show below:

    Read these articles for more info: