Voice of the DBA

Category: Editorial

  • What’s the Edge Case?

    Quite a few of the bugs we see in production systems come from data that isn’t handled well. Perhaps the developer never considered this data, or another bug lets data into a system that should never be recorded. These are often NULL values, but they could be other data values that are far out of the ordinary.

    Where do we draw the line for edge cases? Is it anything that doesn’t fit 95% of the data range? I see this number used in many fields, often manufacturing and other “physical endeavors”. Is it the 80% rule, where we ensure 80% of data cases are covered, but 20% represent special handling?

    It’s an interesting thought because drawing this line helps us decide what level of data we need in our dev and test environments. We need enough data to represent what exists in production, but not much more. The less data we have, the faster everything moves, with much less friction in setting up, resetting, and moving these databases around.

    However, the more bugs that slip through, the more we might need to ensure that we add more data to our development environments to mimic what is in production. Often we have used copies of production data, but there are plenty of issues with this. First, we often have less security in non-production environments and no shortage of data breaches comes from these systems. Therefore we might need to apply masking/obfuscation/pseudomization to values. Second, production databases are growing larger, often over 1 TB. While storage and bandwidth are cheap, they aren’t free, and moving around 1 TB of data regularly, or even restoring it, can present resource challenges.

    My preference is a representative set of data from production, masked and without PII, along with some randomness that might catch edge cases before we deploy changes to production. With that in mind, what’s the edge case? I think I’d lean towards the 95% value, but ready to lower that if we discover many bugs.

    How many are many bugs? I might apply the same standard. If more than 5% of bugs filed are data issues, we need better dev/test data.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Losing Skills Because of Automation

    When I was younger, it seemed that everyone I worked with in technology knew how to build a computer. Most knew how to work with a BIOS, were comfortable with command lines, and could assemble complex compiler directives into a Make file. Over time, it seems many people, especially Windows and MacOS users, became focused on the things they needed for their job, lacking a lot of knowledge about how computers process instructions and the low-level operations they perform.

    There are plenty of very talented developers out there, and many great data modelers, but as I work with many of them that try to make the transition to DevOps, I see lots of uncertainty and tentative behavior. They often approach builds, automated tests, and deployments as though these are completely new skills they need to learn. New software that is foreign to them.

    I wonder if the increased use of automation will make this worse in the future? Already plenty of companies are looking to low-code and no-code solutions as a way to handle the lack of staff to perform development. Will that be exacerbated in the future as more and more automation is put in place that mocks up a shell (or more) of a project and handles work for the developers and even operations staff? Will they struggle to debug complex problems, which are the more likely problems to occur in modern software?

    There is an interesting article about surgeons that might be losing some skills with the advent of robots helping in surgery. While there are some worrisome aspects to this for me, as someone getting older and possibly needing medical care, I find that some of this is applicable to the world of software. How well do we apprentice people new to our environment and give them the chance to build new skills? Often we have senior people taking on interesting work, making data modeling decisions, troubleshooting issues and more. Do we allow more junior people the chance to get hands-on experience and perhaps take charge to lead others? To learn to actually be the one to make the decision?

    In a few places, I’ve seen senior people fixing bugs and junior people developing code. That seemed strange at first, until I realized that fixing problems is more than likely something I want the better developer dealing with, not the worse one. These don’t even have to be issues in production. In a DevOps world, I might have the senior people looking at and fixing the bugs that are caught in CI. While I appreciate giving someone the chance to correct their mistakes, I also think that a “refactoring” or improving of code might be something senior people are better positioned to tackle.

    Of course, I think rotating people in and out of roles, giving them a chance to experience difference sides of our industry, is a good idea. This might include learning how dev works (for Ops people) and how Ops works (for developers) can bring about empathy, understanding, some skills, but likely to create a better culture of collaboration. A tenet of DevOps.

    What skills are we worried about learning? Or losing? To what extent ought we try to ensure others grow wide and deeper outside of their core skills? I find this to be a area that the best DevOps companies do well. They have champions that can provide assistance, knowledge, and teaching, not just do the work. Companies that pretend to adopt DevOps aim for specialization, letting others do the work when they can. Over time, I think they’ll end up like some of today’s surgeons: only practicing their craft for the first time (with no training) when someone else isn’t available.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Psychological Safety

    One of the biggest challenges to becoming better at building, deploying, and operating software is the culture changes required. DevOps requires we work as a team, which can be hard to do. Often people have competing interests and goals. They don’t trust each other as much as we’d like, and maybe more importantly, management doesn’t trust the people doing the work.

    There is plenty of blame thrown around when things don’t work as expected (or aren’t delivered). That’s a poor culture in which to be creative or experimental. In reality, those things are a lot of what we do in software development. We aren’t solving the same problem over and over, building a bridge or house that is mostly constructed as many others are. Instead, we are assembling more complex systems in different ways.

    Most of us aren’t building airplanes, but the situations and problems in this article could easily be applied to many software development projects. Ambitious sales, targets set by management, not engineers. A lack of listening to feedback and unstated pressures to just push things through the system, regardless of potential issues. Perhaps the only thing I agree with from Boeing’s side is that they can’t find all defects, just like we can’t find all bugs. However, we can often find many and minimize or eliminate the impact of the critical ones.

    The pressure on Boeing engineers and line workers is similar to those in software, both on developers and operations. Development needs to deliver features, whether well-tested or not. Operations need to get these out, regardless of the impact on the system. It’s a gross generalization, but one repeated over and over in a few places I’ve worked.

    Culture is critical. Part of DevOps is frequent deployments from a faster speed of deployment. However, that’s only one part. The other is that we continually learn and improve how we do things. We raise the bar of quality. If we are only focused on moving forward without getting better, we get into the position that Boeing was in. No trust, no psychological safety, and no attempt by management to implement DevOps culture. Instead, they’re just pushing for more features with (hopefully) more automated testing, but no real impetus to ensure this is the case.

    More and more of the world is run by software. Whether this is in cars, restaurants, or government. When software doesn’t work well, there can be substantial problems that not only impact a bottom line, but can prevent people from getting goods or services they depend on. I know many of you that build and operate software try to do a good job, and I hope you know this means polishing your craft and improving your abilities. I also hope that management will learn that their support in improving quality is as important as the drive for more features in software.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Challenging Yourself

    I saw this posted on Twitter recently: “If you’re always right, you’re not learning. If you’re never failing, you’re not reaching. The objective is to be right. The objective is to succeed. But if you’re always winning, you’re undershooting your potential.” –@JamesClear

    I found that to be a very interesting view, especially as I think about moving my career, life, business, or anything else forward. I think about this in terms of the goals I’ve set for myself each year for work, or each season of coaching. In the past, I’ve sometimes tried to pick those items that I think I can accomplish.

    After all, it’s nice to check off everything at the end of the quarter/year and note I did everything I planned to do.

    However, is that playing it safe? Since I know roughly what my workload is, how busy I expect to be, am I just picking things that aren’t pushing me to do better? Or perhaps more importantly, should I be optimistic, push, and if I fall short, examine reasons why. Life sometimes changes in unexpected ways, and I don’t always have control over what things might come up in my schedule across a year. Heck, I might not anticipate everything that comes up this week, or even today.

    When I set my goals, I try to review these monthly and examine what I’ve done. Usually, across a year, I realize that some of my goals feel less important across time, or that they are such a low priority (nice-to-have’s) that other work or life constantly is more important. There also is the case that my mood changes and sometimes I just don’t want to do some things and keep procrastinating.

    My life often looks different in May than it did in January. It looks even more different in November, and as I write this, I think back to the decisions I made in January. I can barely remember the why, which is probably another hole in how I challenge myself. I ought to set out a basis at the start to remind myself of the reasons for these goals when I look back periodically when I assess my progress.

    I ask others to push themselves, so I ought to do more of that myself. Taking on a challenge is a good thing, and since life is more than work, I am looking forward to finding a better balance of challenging how I look to advance my career, as well as my personal life, and pick a variety of things that are both important to me, but are also in keeping with the spirit of balance. Time with my wife, hobbies, friends, and even myself are important as well. I don’t look to schedule those, but I do want to remember that while I want to challenge myself, I also need to balance that with living and enjoying my life.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.