Voice of the DBA

Tag: DevOps

  • The Test Mistake

    Last year a company was testing their software in a dev/test environment and accidentally sent out an “Integration email” to their customers.  They blamed this on an intern, though a little jokingly, and plenty of people shared their own stories about making mistakes.

    I did this once, and only once. I was doing some software testing on a copy of the production database and accidentally sent out thousands of emails to customers. I hadn’t scrubbed the email addresses in the copy, but since I also hadn’t scrubbed the data, customers got an “extra” email for their account and few complained. My boss, however, complained quite a bit and with good reason.

    These days using production data in dev/test environments isn’t really an acceptable practice. Plenty of organizations do it, and hopefully, they are taking precautions. Kendra Little wrote a little about what happened and what should be done, including scrubbing out sensitive data. Not enough people do this, though I am seeing more and more companies looking for solutions that prevent data release.

    This isn’t an intern’s issue. I appreciate someone making a bit of a joke here, and hopefully, there were engineers that taught the intern something. The bigger issue is that we make mistakes as humans and our protocols and processes should expect that. This is one reason why adopting automation in our software process is important. Whether that’s deploying code to the right production server or refreshing the non-production databases, we want to be sure that we limit the number of human mistakes.

    There are always places where we will use humans to perform a process. Whether that’s typing code or clicking something. It’s entirely possible someone clicks the wrong button or types the wrong item. As much as possible, we can use automation to provide a safety net to prevent these issues from reaching customers. We can’t prevent every mistake, but we can continue to improve our process but adjusting it over time. At the very least, we ought to prevent the same mistake from happening over and over.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Speedy Break Fix

    I received a report at SSC of a missing toolbar in the forums. I checked, and sure enough, the toolbar for formatting code and text in responses was missing. I submitted a high-priority ticket, mostly because I hate unformatted code. The site was still usable, but people would be posting stuff that just didn’t look good. The developers got to work and had a fix, but it still wasn’t right. The wrong toolbar was showing. This was a generic toolbar, not the one that we have had for years (this one). They produced a second fix that contained the right toolbar.

    The developers are in and out of the SQL Server Central codebase, working for a few days on open issues, then busy with other work for a week or two. They work in a DevOps style environment, with changes run through a Pull request/peer-review process, then CI, then an automated deployment. They often can make a change and deploy it in hours, or even minutes.

    Is this a bad process?

    It would be easy to say that the developers were moving too fast in their deployments and not testing things. However, the issue in production wasn’t easily reproduced locally, and we found a few places where local dev environments don’t quite match live. That’s been an ongoing problem for as long as we’ve been developing software outside of production. The DevOps solution here is to adjust the environment setup in code so that dev does match production closer. I’d say this is my solution, but likely we’ll find something else in the future that isn’t the same and we will continue to need to adjust something to get the environments in sync.

    What’s the alternative? We could go slower and batch up a bunch of changes, testing them all, moving at a more waterfall-ish cadence. Would we catch this? Perhaps. However, we might not, and if we are batching changes, then bugs live for a longer time in production until we perform a new release.

    Or we could build a quick patch, in one of two ways. One is by moving quickly, at an agile/DevOps speed, to quickly build a patch. That’s a hacky, not-thought-through flow that many people use to fix a deployment, usually without the process and bounds that a good DevOps process provides.

    The other option is to bundle a fix in the next release, which could be next week? Next month? Who knows. Slow processes don’t necessarily make the final code better and can prevent rapid fixes.

    Ultimately my feedback from here to the project manager is that we need a test, a process, or something that checks for these issues before release. While I know something else could break, I prefer these small deployments taking place often, with bumpers to limit the regressions we know about.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Defining DevOps

    DevOps has become somewhat mainstream in that many companies, perhaps most, are talking about it or trying to implement some of the principles behind this software development approach. However, even as I talk with people that practice DevOps, I find the idea of what DevOps actually includes can vary widely among developers, even inside the same company.

    There are lots of definitions, and this one caught my eye recently. It says, among other things, that there should be “increased collaboration among not only developers and operations staff but users.” I like that there is some emphasis on working together, which is often the hardest part of changing how we build software. I find that many customers struggle because they end up adopting the technology tools, but fail to change how they communicate with others. Culture change is hard, and DevOps requires a culture change.

    The article is high level, but it does highlight some of the complexities of DevOps, mainly that lots of stuff is happening at once. Good reporting, tracking, and management are important, and unfortunately, lots of managers don’t want to change their approaches to managing teams, which further complicates the process of building better software faster. Certainly, the need to have developers work with ops people, including supporting production systems, is important, but it never happens without management requiring and supporting this collaboration.

    The drawbacks to DevOps are many. Plenty of teams slap together code, getting the “faster” part but forget the “better” part, which means increasing and higher quality code over time. They embrace new tools, but they sometimes forget to run small experiments and change their approaches across time when they find better ways of building and operating code. Or they refuse to stick with current techniques or software because they crave change. Developers also find that some groups, like security and compliance departments, aren’t ready for the pace of change, which can be frustrating.

    Ultimately, DevOps is better collaboration, getting all teams on board working together. Like many people rowing a boat, every group needs to align their work with others to ensure a smooth and successful journey. Actually creating that alignment requires strong leadership and culture, two of the hardest, and rarest, things to build inside many organizations.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Data and DevOps Predictions for 2022

    All the Advocates at Redgate (Kathi, Grant, myself) got asked about some of the future predictions we might have, based on some of the data from The 2021 State of Database DevOps and The 2021 State of Database Monitoring. We each came up with a few things and then someone summarized them. Grant has an official blog post, but I thought I’d glance through the summary, see what the others wrote, and then come up with a few things.

    Side note: I hope to review this next January and see how I did.

    I didn’t do this last year, but I’m going to separate the DevOps and monitoring topics and give a few thoughts on each.

    DevOps

    DevOps continues to grow and really become almost a mainstream idea. I don’t know if I think the majority of companies are really adopting much of it, but many are adopting something. I rarely need to explain DevOps, though I do often need to define it for me and level set how I view it. It still seems everyone things about this differently.

    The main challenges for DevOps are adding automation, moving to the cloud (or hybrid) and writing better code. DevOps can help with all these things, but I think far too many organizations still want to tackle these are projects, not an ongoing habit.

    Well, all but the last one. It seems no one really prioritizing better code in most management I encounter.

    Automation is the easiest of these, and I find that more and more DBAs are embracing automation in different ways. Some still want to do everything manually, but more of them are starting to accept that a framework (like Flyway) or even a simple automated script runner can be used without them needing to connect with SSMS.

    I see this as a trend that will continue to increase, and eventually we’ll expect 95% of all deployment code to run with an automated process of some sort. Bt process I mean you aren’t connecting to a production server from your workstation.

    My prediction for 2022 is also that I won’t see anyone create new repos in a version system other than git. I do tend to work in the MS stack or the Flyway area, and I don’t expect to encounter any clients still trying to work with SVN, CVS, Vault, Perforce. or even TFVC. Any why should they?

    I also expect that we will see more cross platform work from everyone, which means that we will want a fairly consistent way of managing development, whether this is on SQL Server, Oracle, PostgreSQL, or MongoDB. I know that deploying code here means more chances for mistakes, so having a consistent high level process is important. This might mean that you use one type of framework for SQL Server and another for PostgreSQL, but the flow will be the same. Save code, run CI, deploy with automation. Hopefully testing in there somewhere.

    The cloud encourages automation, but not completely. The portals are complex, the offerings changing, and no one wants to worry about which version of a CLI you have on which machines. Still, I hope to see more GitOps or Infrastructure as Code used by clients.

    Monitoring

    I think two things here. First, people are going to rely more and more on monitoring systems to manage the disparate and ever-growing estates of data stores they have. This is going to come about with the problem for many people of not only managing different platforms, but also different monitoring tools.

    People already struggle with SQL Server and Oracle (or any other combination of platforms). I expect they will have AWS and Azure to contend with as well, though likely each organization will lean more heavily on one than the other. However, they’re also going to be dealing with different products for different platforms.

    I predict some stress, confusion, and anxiety about incidents and employees trying to decide how to find and use information.

    The second prediction is that we are going to see another security issue with a monitoring and/or management platform. We had a big one last year, but I bet hackers are working out how to attack other platforms, especially those that require sysadmin privileges and also allow the tools to change the remote systems.

    Security is always a problem and after last year’s incident, I expect other software to be targeted in this same way.

    Looking Forward

    If you are trying to find a new job, or improve your career, here’s what I suggest. First, learn to write and speak. However you can, but working with others, advocating your position, and improving your showing at interviews will matter.

    In terms of tech, get good at something but learn a bit of many things. Pick a cloud and learn how the general infrastructure of IaaS, PaaS, and SaaS work. Things will vary a bit from Azure to AWS to GCP, but the core remains the same. Learn how to spin up and work with a variety of technologies related to data. Be comfortable with version control, with repos, with CI, with CLI tools, with finding answers on how to accomplish tasks. Knowing a bit and how to find out more might be a good way to answer some of those interview questions.