Voice of the DBA

Tag: DevOps

  • DevOps is Culture

    DevOps is a culture shift in your organization. Technology enables things inside of this culture, but the culture is important. If you don’t know how to build a new culture and support individuals as they adopt it, often you’re just renaming an existing broken process inside your organization.

    Often, I think the failure to reach the efficient levels of a Spotify, Capital One, or any other DevOps success story is because of a failure of management to change culture.

    There are any number of anti-patterns for organizations wanting to adopt DevOps. Renaming teams without changing how they work together, including incentives and punishments, doesn’t help. Just buying technology without process and culture is just spending money on new toys. Keeping your existing organizational constraints in place without encouraging and supporting teamwork, changing incentives, eliminating silos/fiefdoms for managers will ensure that you will struggle to make a digital transformation that enables your organization to realize a return on their efforts.

    What things do help with culture change? Certainly, many changes start with a team or two, learning about how to build a better software pipeline, both with better code and more automation. They have to be allowed to fail fast, make some mistakes, and review issues without blame. Management must support this.

    We need to optimize systems, meaning the flow of code from developer to production system, without optimizing the effort in any team. People need to put theie name on work, owning failures, and celebrating the success of others. Collaborate and pitch in to work together, regardless of the title, position, or responsibility you have been assigned.  If someone needs help, they need to be supported and not punished. We need psychological safety to do the best we can, but understand we all need to ask for, or be ready to give, help at some point.

    DevOps is teamwork. It’s coordinating and collaborating to agree on the best code and process that gets code to production quickly. Safe, secure, performant, and high-quality code of which everyone is proud. If we’re not, then we ought to be working together to make the code better or agree an exception is allowed for some reason.

    Above all, we need to respect the autonomy of individuals and teams. Trust them to get the work done their way and take responsibility for it. Most people are accountable, especially when they can accept and help fix their mistakes without punishment. We all make mistakes, so let’s do our best to move forward and help everyone avoid repeating mistakes with learning, not yelling.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • How Hard is Kubernetes?

    We’ve run Kubernetes inside Redgate for some research projects (like Spawn) and we are building some skills running this orchestrator. At the same time, we’ve had no shortage of challenges in keeping the clusters up at times, patching, fixing issues, upgrading to new configurations, etc. Like any software, there is work involved with managing the orchestrator.

    I’ve watched Andrew Pruski and Anthony Nocentino write about containers and Kubernetes and overall they’ve made me view the clusters like email. It’s useful and I want to use it, but I don’t want to manage or administer it. I’d want some service like AKS or EKS instead. Let someone else build expertise.

    If you use containers, do you have an orchestrator running in your data center? Mercedes does, with over 900 clusters. They found value early on with container technologies and built in-house expertise within their research arm. I think a large organization like Mercedes likely can make this investment pay off, especially as they likely don’t depend on any one person to understand and manage Kubernetes. They can afford someone like Andrew or Anthony quitting and taking another position.

    The rest of us can’t really do that, certainly not without our organization feeling containers and orchestration is a core competency.

    The key for Mercedes is automation. They note that if they added 500 more clusters, they’d need just one more engineer. That’s a key for any of us that want to manage growing numbers of systems without spending a lot of our time reviewing resumes and hoping we can hire good staff. Hiring is hard, and finding good people even harder. When you find them, set them free codifying their knowledge using DevOps, scripting, automation, and more.

    Then educate others and teach them what your talented engineer is doing. Mercedes notes that finding people is hard, and educating existing people is easier. DevOps, better coding, understanding APIs and declarative scripting are not hard skills, but they are something people need to practice to develop familiarity and skill. We want staffers to be able to easily pick up the work of another, understand it, and extend or improve it. We don’t want to depend on the person that wrote it.

    The way Mercedes has attacked this technology is the way I’d have developers and administrators tackle DevOps. Take advantage of the power of modern software development and infrastructure tools and empower your staff to make things better. They are likely to enjoy their jobs more and remain employed, reducing your need to struggle with the vagaries of finding and hiring good people, a problem no one has solved well.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Slides from SQL Saturday Los Angeles 2022

    If you attended my talk at SQL Saturday Los Angeles 2022, then you can get my slides here: Adopting a DevOps Process for your Database.

  • Reducing Debt and Increasing Security

    Many of us know that reducing financial debt in our lives leads to a bit more security in our minds. We are better able to cope with unexpected expenses when we have the ability to get credit or pay for something in cash. If you’ve ever had a major house repair, car calamity, or (hopefully not) medical issue, you know what I mean. At least in the US. That last one is less of an issue in many other countries.

    We know that our code has technical debt, and often we fight a battle with project managers and business people over whether we should spend time paying down that technical debt or developing new features. We certainly can do both, but often that’s not a choice we are given. If we do get the chance to reduce technical debt, it’s a tiny fraction of the time that is allocated to adding something new.

    There might be a better argument for reducing some technical debt: security. There’s an article that talks about the security risk posed by moving too fast and not ensuring there is rigorous security testing. This is a common problem in many companies that adopt some parts of DevOps, but forget that quickly deploying new code isn’t the only principle. We also need to learn from our mistakes and quickly correct them.

    I am a big fan of releasing often, especially for security purposes, but those releases need to build on each other with not only new code but refactored old code that doesn’t perform well or has a security issue. More importantly, we need to ensure when we find issues in code, that all developers are aware of the problem and don’t repeat the issue. We also need to update our testing to ensure that humans don’t continue to write code in a style that causes known issues. We want to avoid antipatterns in our code.

    This is less of a developer issue than a managerial one. The C-suite and all management that are involved in software need to prioritize security so that known issues are fixed quickly. This also should involve ensuring that technical debt is kept under control as a lot of this poorly written code increases the chances that a vulnerability is in the code. Either because the code isn’t well written or because developers avoid touching the code in the future.

    The best solutions, to me, involve DevOps. Make small changes, deploy them, instrument them, and learn if they work well. If not, fix them quickly. While this might end up delaying some features, ultimately it ensures that our software is in better shape, with a reduced risk of simple security mistakes upsetting our customers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.