Voice of the DBA

Tag: DevOps

  • What’s the Edge Case?

    Quite a few of the bugs we see in production systems come from data that isn’t handled well. Perhaps the developer never considered this data, or another bug lets data into a system that should never be recorded. These are often NULL values, but they could be other data values that are far out of the ordinary.

    Where do we draw the line for edge cases? Is it anything that doesn’t fit 95% of the data range? I see this number used in many fields, often manufacturing and other “physical endeavors”. Is it the 80% rule, where we ensure 80% of data cases are covered, but 20% represent special handling?

    It’s an interesting thought because drawing this line helps us decide what level of data we need in our dev and test environments. We need enough data to represent what exists in production, but not much more. The less data we have, the faster everything moves, with much less friction in setting up, resetting, and moving these databases around.

    However, the more bugs that slip through, the more we might need to ensure that we add more data to our development environments to mimic what is in production. Often we have used copies of production data, but there are plenty of issues with this. First, we often have less security in non-production environments and no shortage of data breaches comes from these systems. Therefore we might need to apply masking/obfuscation/pseudomization to values. Second, production databases are growing larger, often over 1 TB. While storage and bandwidth are cheap, they aren’t free, and moving around 1 TB of data regularly, or even restoring it, can present resource challenges.

    My preference is a representative set of data from production, masked and without PII, along with some randomness that might catch edge cases before we deploy changes to production. With that in mind, what’s the edge case? I think I’d lean towards the 95% value, but ready to lower that if we discover many bugs.

    How many are many bugs? I might apply the same standard. If more than 5% of bugs filed are data issues, we need better dev/test data.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Psychological Safety

    One of the biggest challenges to becoming better at building, deploying, and operating software is the culture changes required. DevOps requires we work as a team, which can be hard to do. Often people have competing interests and goals. They don’t trust each other as much as we’d like, and maybe more importantly, management doesn’t trust the people doing the work.

    There is plenty of blame thrown around when things don’t work as expected (or aren’t delivered). That’s a poor culture in which to be creative or experimental. In reality, those things are a lot of what we do in software development. We aren’t solving the same problem over and over, building a bridge or house that is mostly constructed as many others are. Instead, we are assembling more complex systems in different ways.

    Most of us aren’t building airplanes, but the situations and problems in this article could easily be applied to many software development projects. Ambitious sales, targets set by management, not engineers. A lack of listening to feedback and unstated pressures to just push things through the system, regardless of potential issues. Perhaps the only thing I agree with from Boeing’s side is that they can’t find all defects, just like we can’t find all bugs. However, we can often find many and minimize or eliminate the impact of the critical ones.

    The pressure on Boeing engineers and line workers is similar to those in software, both on developers and operations. Development needs to deliver features, whether well-tested or not. Operations need to get these out, regardless of the impact on the system. It’s a gross generalization, but one repeated over and over in a few places I’ve worked.

    Culture is critical. Part of DevOps is frequent deployments from a faster speed of deployment. However, that’s only one part. The other is that we continually learn and improve how we do things. We raise the bar of quality. If we are only focused on moving forward without getting better, we get into the position that Boeing was in. No trust, no psychological safety, and no attempt by management to implement DevOps culture. Instead, they’re just pushing for more features with (hopefully) more automated testing, but no real impetus to ensure this is the case.

    More and more of the world is run by software. Whether this is in cars, restaurants, or government. When software doesn’t work well, there can be substantial problems that not only impact a bottom line, but can prevent people from getting goods or services they depend on. I know many of you that build and operate software try to do a good job, and I hope you know this means polishing your craft and improving your abilities. I also hope that management will learn that their support in improving quality is as important as the drive for more features in software.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • DevOps is Culture

    DevOps is a culture shift in your organization. Technology enables things inside of this culture, but the culture is important. If you don’t know how to build a new culture and support individuals as they adopt it, often you’re just renaming an existing broken process inside your organization.

    Often, I think the failure to reach the efficient levels of a Spotify, Capital One, or any other DevOps success story is because of a failure of management to change culture.

    There are any number of anti-patterns for organizations wanting to adopt DevOps. Renaming teams without changing how they work together, including incentives and punishments, doesn’t help. Just buying technology without process and culture is just spending money on new toys. Keeping your existing organizational constraints in place without encouraging and supporting teamwork, changing incentives, eliminating silos/fiefdoms for managers will ensure that you will struggle to make a digital transformation that enables your organization to realize a return on their efforts.

    What things do help with culture change? Certainly, many changes start with a team or two, learning about how to build a better software pipeline, both with better code and more automation. They have to be allowed to fail fast, make some mistakes, and review issues without blame. Management must support this.

    We need to optimize systems, meaning the flow of code from developer to production system, without optimizing the effort in any team. People need to put theie name on work, owning failures, and celebrating the success of others. Collaborate and pitch in to work together, regardless of the title, position, or responsibility you have been assigned.  If someone needs help, they need to be supported and not punished. We need psychological safety to do the best we can, but understand we all need to ask for, or be ready to give, help at some point.

    DevOps is teamwork. It’s coordinating and collaborating to agree on the best code and process that gets code to production quickly. Safe, secure, performant, and high-quality code of which everyone is proud. If we’re not, then we ought to be working together to make the code better or agree an exception is allowed for some reason.

    Above all, we need to respect the autonomy of individuals and teams. Trust them to get the work done their way and take responsibility for it. Most people are accountable, especially when they can accept and help fix their mistakes without punishment. We all make mistakes, so let’s do our best to move forward and help everyone avoid repeating mistakes with learning, not yelling.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • How Hard is Kubernetes?

    We’ve run Kubernetes inside Redgate for some research projects (like Spawn) and we are building some skills running this orchestrator. At the same time, we’ve had no shortage of challenges in keeping the clusters up at times, patching, fixing issues, upgrading to new configurations, etc. Like any software, there is work involved with managing the orchestrator.

    I’ve watched Andrew Pruski and Anthony Nocentino write about containers and Kubernetes and overall they’ve made me view the clusters like email. It’s useful and I want to use it, but I don’t want to manage or administer it. I’d want some service like AKS or EKS instead. Let someone else build expertise.

    If you use containers, do you have an orchestrator running in your data center? Mercedes does, with over 900 clusters. They found value early on with container technologies and built in-house expertise within their research arm. I think a large organization like Mercedes likely can make this investment pay off, especially as they likely don’t depend on any one person to understand and manage Kubernetes. They can afford someone like Andrew or Anthony quitting and taking another position.

    The rest of us can’t really do that, certainly not without our organization feeling containers and orchestration is a core competency.

    The key for Mercedes is automation. They note that if they added 500 more clusters, they’d need just one more engineer. That’s a key for any of us that want to manage growing numbers of systems without spending a lot of our time reviewing resumes and hoping we can hire good staff. Hiring is hard, and finding good people even harder. When you find them, set them free codifying their knowledge using DevOps, scripting, automation, and more.

    Then educate others and teach them what your talented engineer is doing. Mercedes notes that finding people is hard, and educating existing people is easier. DevOps, better coding, understanding APIs and declarative scripting are not hard skills, but they are something people need to practice to develop familiarity and skill. We want staffers to be able to easily pick up the work of another, understand it, and extend or improve it. We don’t want to depend on the person that wrote it.

    The way Mercedes has attacked this technology is the way I’d have developers and administrators tackle DevOps. Take advantage of the power of modern software development and infrastructure tools and empower your staff to make things better. They are likely to enjoy their jobs more and remain employed, reducing your need to struggle with the vagaries of finding and hiring good people, a problem no one has solved well.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.