Voice of the DBA

Tag: DevOps

  • Improving DevOps Automation

    We use a lot of automation at Redgate. As a company that builds software, we want to ensure that all the changes from our teams get integrated and tested in a timely manner. We have multiple teams on some products, and while we do have regular meetings between them, it can be easy for a developer to miss some update on a change others are making and break something with their own work. Worse, they could cause a regression or security issue, which we work hard to avoid.

    Recently I saw a post from one of our lead software engineers about the build process. While we have lots of pipelines, the general process hasn’t changed in many years. In some sense that’s good, because we focus on building better software, not the process. We don’t change just for the sake of change, or because a new team or division lead likes one tool over another. I’ve seen customers where they move from Jenkins to Azure DevOps because someone in charge “likes one better.” Not a good use of time.

    However, you do need to evaluate whether your process is meeting your needs. In our case, we surveyed lots of developers to get their thoughts on pain points and issues. The build and release process was consistently listed as a pain point, with releases often requiring a dev to manage it from their workstation, preventing them from spending time building software for customers.

    That might be the big lesson I saw in the write-up. We realized that a non-negligible amount of time was being spent by developers on the process of moving bits rather than building the software. This led us to re-examine how things are done. In particular, we looked at the agent OS (Linux v Windows) and containers. Both of these are more viable technologies now than in years past, and they can reduce costs while smoothing the process. As a result, we are developing a general contract that helps us decide how to change our process. This is similar to an API, that doesn’t specify the tool to use, but rather the inputs, outputs, and what the effect should be. From this, we’ll start to help teams move forward is changing their process as we have time.

    The big takeaway right now is that this is pushing us to use containers, which simplify the steps and allow us to easily move from Azure to AWS to on-premises, or really any environment. We can switch builds across different platforms and more easily scale up or down as needed. It will take some time, and some of our software will be more challenging in containers. However, we are also seeing benefits from customers, and at some point, I expect we’ll provide containers for certain functions that make it easier for the end-users of our software to also deploy and upgrade their tools.

    DevOps is an ongoing process. Not a set of tools that you change just because, or a way of building software that matches what another organization does. Instead, it’s learning, experimenting, and evaluating how you can be more effective. Then adopting what you’ve learned and repeating the process. Keep improving, and you’ll find that you can produce software quicker and at a higher level of quality while improving the skills of your engineers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Enterprise Software

    “I’ll go in the store; it’s built for that.”

    That is a quote from Jessica Kerr, who is a developer. She has a great blog on software development, called The Enterprise Eats Software. It starts with a poor experience for an online order for Lowe’s. Interestingly enough, as I read this, I was starting a home renovation project, but I didn’t have a bad experience, I had a great one. Read Jessica’s blog, and see what you think.

    For my short story, we had contracted for a bathroom project, but we forgot to get some of the fixtures. The contractor came a day early to talk about things, and we realized we needed a shower diverter valve and trim kit. We looked for the trim, which is what was important to my wife, and found a valve to fit it. While the contractor called two local supply houses, I looked at Amazon. The suppliers didn’t have the valve, but Amazon did, noting delivery in two days. I ordered it around 10:30 am that day. It arrived around 11:00 am the next day, a day early and before the existing shower had been demoed.

    That was fantastic. It made me think why would I want to use any service that wasn’t that amazing and quick. I got an email at my desk with a picture, telling me the package had been delivered and showed me where it was. I walked it upstairs thinking that calling multiple suppliers and then driving around town would have been a pain,  not to mention a time sink.

    I don’t often find a lot of large companies do a good job with their software integrating into real world operations. A few do, and apart from Amazon, I’ll say build.com was incredible for us to get a tub and Wayfair got us a vanity very quickly (too quickly, actually). Both have built great systems that not only took the order but updated us and handled the complexities of shipping large items. Not to mention they both had an incredible selection available to peruse easily and quickly. Just finding a place to look at something like a tub in person is incredibly difficult.

    Apart from the retail challenges, just the general experience from many enterprises leaves something to be desired. They just aren’t good at being agile, flexible, and maybe more importantly, constantly improving. They get caught up in some of the hassles, like bureaucracy, power struggles, too many rules, etc. They don’t know how to operate in a flexible, agile, constant change environment.

    I’m reading Project to Product, which in many ways sees a lot of the same problems. Software is disconnected from the goals of the business, and too often there are individuals and processes that get in the way of becoming more effective across teams and partners. BMW is one of the success stories here, and I still think they’re behind Tesla in their industry. They might catch up, and if they do, it’s because they are learning to be a better software company, not a better manufacturer.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Titles and DevOps Confusion

    Alex Yates has become quite the DevOps consultant in the last few years. I used to work with Alex at Redgate before he left to start his own consulting firm. I hear nothing but good things about his work, and if you are looking for someone to help guide your database development team, he’d be a good choice.

    He wrote a piece on the Octopus Deploy blog about the title of DevOps Engineer and why that doesn’t quite make sense. He essentially notes that this doesn’t really describe what a person does and it might not be a good title. I tend to agree, since DevOps isn’t a thing, per se, but rather a set of guiding principles. While this might include building software, or deploying it, there are titles for those things (build engineer or release engineer). It could include Infrastructure-as-code, but that’s the domain of sysadmins.

    There are any number of titles that have come about in this business, and likely more being created all the time. If you have a new job, often you can finagle your way around some pay structures because after all, if you invented the term, the HR department doesn’t have a pay scale to limit you. I suspect that’s part of the reason someone started calling themselves a DevOps Engineer in the first place and got someone to hire them under that title.

    I suspect titles used to be more descriptive and important in the past, when we had fewer of them and the work that each of us did was more tightly scoped. Throughout my career, I’ve started to see more people doing more types of work all the time. Especially in technology where we usually do the work that needs to be done, often crossing over the duties between job titles when something is broken. Getting a system working often takes precedence over any claims of some task being “my work” or “your work.”

    Titles ought to be somewhat consistent, if for no other reason than to easily allow us to compare the work we do in different organizations and allow us to easily promote ourselves to a new employer. If every organization had different titles, we’d waste a lot of time trying to decide if we wanted a position, as would hiring managers trying to decide if we could do the work. Certainly the we find that a DBA or a Developer moving organizations might have some different tasks, but we have a general idea of what work they should be able to complete.

    I’ve never been too concerned about my titles, though I have worked to become “senior” at different employers. That’s usually a function of proving you can do the work well, across a period of time. Of course, if you get work done effectively and efficiently, the title likely doesn’t matter. Someone, likely many someones, will want to hire you.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • The Tech Blame Game

    Last year Solarwinds was hacked and blamed an intern for a security lapse. When Equifax was hacked, in testimony to the US Congress, the former CEO blamed a specific, though unnamed, person for not patching a system. British Airways blamed their USD$200+mm IT issue on an engineer that rebooted a system too quickly.

    I don’t know that any large company from my younger days, say before 1990, would have blamed a massive failure on a single person. While any single person can influence more systems in the age of technology, no one should have the power to cause such a massive failure. If they do, I think I’d look towards poor system design, rather than individuals.

    These aren’t the only examples of management trying to scapegoat an IT worker, and I suspect we’ll see more examples in the future. However, I hope that governments and shareholders start to demand better management from management. If you don’t understand how IT works, get auditors or consultants to evaluate things and explain them to you. If you don’t think that your systems are well put together without single points of failure, address that. If you worry about security, make that a priority. Microsoft did after the Slammer worm, and arguably they have a difficult job where most employees want to control their laptops and workstations entirely and run them in their individual manner. Microsoft built better controls into infrastructure and software development, and everyone else should as well. Management needs to own their responsibility for failures.

    We should expect mistakes in security, in design, in coding, and more. We should also be placing guardrails, tests, and limits inside our environments to ensure that we catch most of the issues. Software development and system design have improved dramatically the last decade to help us improve quality and security, but we have to embrace the knowledge that’s been gained, as well as ensure we have circuit breakers to prevent runaway failures. If a sysadmin can alter a Chef script to set the max memory in SQL Server to 1MB, this shouldn’t get deployed to all instances. Moreover, we ought to be testing for all sorts of potential changes that can cause issues.

    To me, this is the area that DevOps, GitOps, anything Ops, automated, or at scale, needs to mature. We need to allow for, expect, and assume mistakes and failures will happen and build in controls to our build and test systems. Once we start to better understand how someone can make simple mistakes, we can attach more checks and balances to ensure that we continue to improve quality, without sacrificing speed, or lowering security.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.