Voice of the DBA

Category: Editorial

  • Self-Service Databases

    When I first started work as a software developer, I knew that getting an environment set up where I could compile a project might take a few hours or a few days. The complexities of how people built software projects, the dependencies, and more were handled in a very immature manner. These days I can often download a project from GitHub and compile it in minutes, with many of the issues handled automatically with various tools such as complex config files and package managers.

    In the world of database development, I often find people struggle to get a database set up with a project. Whether you are coding in SQL, C#, or Java, many organizations haven’t got a standard way of bringing up a database environment for developers. Some have a shared database for a team, which can be better, but can also lead to collisions of code as the pressure to get software built grows and team size increases.

    This leaves out the challenges of data, which can be changed by others, or lack the size/shape/skew to properly reflect what the production code has to handle. There are no shortage of bugs that come from incomplete data sets used in development that are unrepresentative of the live database.

    Do you have a way for developers to self-service the database assets they need? Whether this is a new dev database, a refreshed one from prod, or even a point in time from development.  Is there a way developers can get a database into the state that matters for them. When developers are ready to work on a problem, the last thing we want is a significant delay to them getting started.

    One of the things DevOps stresses is that we want to reduce friction and tedious, manual work were possible. Let the computer manage the tedious things. This should be how we manage those database environments. We also want some consistency among all developers, QA, CI, and any other environment, to ensure we can reproduce problems to troubleshoot issues.

    The customers I work with that have the smoothest development processes, and best ROI from the resources spent on development, have a way to hydrate a consistent database environment for their developers. More importantly, the developers can self-service this. When a single developer, or a team, decides to reset their environment, they can do so, applying in-flight code changes from version control to bring back the database to the state it was previously in.

    Application developers do this all the time, as they restart their application. Removing their abliity to change code, restart the app, test it, and observe results easily would be crippling. Yet, we rarely reset dev databases, and often, don’t have consistency between environments.

    Self-service is one of the key indicators of a mature software development organization. This is easily achievable for many app projects, but not simple for databases. Relatively few groups have figured out how to do this well, but those that do are much more productive and efficient than everyone else. Traits that are becoming more important in this highly competitive world with increasingly fickle customers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • What’s the Edge Case?

    Quite a few of the bugs we see in production systems come from data that isn’t handled well. Perhaps the developer never considered this data, or another bug lets data into a system that should never be recorded. These are often NULL values, but they could be other data values that are far out of the ordinary.

    Where do we draw the line for edge cases? Is it anything that doesn’t fit 95% of the data range? I see this number used in many fields, often manufacturing and other “physical endeavors”. Is it the 80% rule, where we ensure 80% of data cases are covered, but 20% represent special handling?

    It’s an interesting thought because drawing this line helps us decide what level of data we need in our dev and test environments. We need enough data to represent what exists in production, but not much more. The less data we have, the faster everything moves, with much less friction in setting up, resetting, and moving these databases around.

    However, the more bugs that slip through, the more we might need to ensure that we add more data to our development environments to mimic what is in production. Often we have used copies of production data, but there are plenty of issues with this. First, we often have less security in non-production environments and no shortage of data breaches comes from these systems. Therefore we might need to apply masking/obfuscation/pseudomization to values. Second, production databases are growing larger, often over 1 TB. While storage and bandwidth are cheap, they aren’t free, and moving around 1 TB of data regularly, or even restoring it, can present resource challenges.

    My preference is a representative set of data from production, masked and without PII, along with some randomness that might catch edge cases before we deploy changes to production. With that in mind, what’s the edge case? I think I’d lean towards the 95% value, but ready to lower that if we discover many bugs.

    How many are many bugs? I might apply the same standard. If more than 5% of bugs filed are data issues, we need better dev/test data.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Losing Skills Because of Automation

    When I was younger, it seemed that everyone I worked with in technology knew how to build a computer. Most knew how to work with a BIOS, were comfortable with command lines, and could assemble complex compiler directives into a Make file. Over time, it seems many people, especially Windows and MacOS users, became focused on the things they needed for their job, lacking a lot of knowledge about how computers process instructions and the low-level operations they perform.

    There are plenty of very talented developers out there, and many great data modelers, but as I work with many of them that try to make the transition to DevOps, I see lots of uncertainty and tentative behavior. They often approach builds, automated tests, and deployments as though these are completely new skills they need to learn. New software that is foreign to them.

    I wonder if the increased use of automation will make this worse in the future? Already plenty of companies are looking to low-code and no-code solutions as a way to handle the lack of staff to perform development. Will that be exacerbated in the future as more and more automation is put in place that mocks up a shell (or more) of a project and handles work for the developers and even operations staff? Will they struggle to debug complex problems, which are the more likely problems to occur in modern software?

    There is an interesting article about surgeons that might be losing some skills with the advent of robots helping in surgery. While there are some worrisome aspects to this for me, as someone getting older and possibly needing medical care, I find that some of this is applicable to the world of software. How well do we apprentice people new to our environment and give them the chance to build new skills? Often we have senior people taking on interesting work, making data modeling decisions, troubleshooting issues and more. Do we allow more junior people the chance to get hands-on experience and perhaps take charge to lead others? To learn to actually be the one to make the decision?

    In a few places, I’ve seen senior people fixing bugs and junior people developing code. That seemed strange at first, until I realized that fixing problems is more than likely something I want the better developer dealing with, not the worse one. These don’t even have to be issues in production. In a DevOps world, I might have the senior people looking at and fixing the bugs that are caught in CI. While I appreciate giving someone the chance to correct their mistakes, I also think that a “refactoring” or improving of code might be something senior people are better positioned to tackle.

    Of course, I think rotating people in and out of roles, giving them a chance to experience difference sides of our industry, is a good idea. This might include learning how dev works (for Ops people) and how Ops works (for developers) can bring about empathy, understanding, some skills, but likely to create a better culture of collaboration. A tenet of DevOps.

    What skills are we worried about learning? Or losing? To what extent ought we try to ensure others grow wide and deeper outside of their core skills? I find this to be a area that the best DevOps companies do well. They have champions that can provide assistance, knowledge, and teaching, not just do the work. Companies that pretend to adopt DevOps aim for specialization, letting others do the work when they can. Over time, I think they’ll end up like some of today’s surgeons: only practicing their craft for the first time (with no training) when someone else isn’t available.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Psychological Safety

    One of the biggest challenges to becoming better at building, deploying, and operating software is the culture changes required. DevOps requires we work as a team, which can be hard to do. Often people have competing interests and goals. They don’t trust each other as much as we’d like, and maybe more importantly, management doesn’t trust the people doing the work.

    There is plenty of blame thrown around when things don’t work as expected (or aren’t delivered). That’s a poor culture in which to be creative or experimental. In reality, those things are a lot of what we do in software development. We aren’t solving the same problem over and over, building a bridge or house that is mostly constructed as many others are. Instead, we are assembling more complex systems in different ways.

    Most of us aren’t building airplanes, but the situations and problems in this article could easily be applied to many software development projects. Ambitious sales, targets set by management, not engineers. A lack of listening to feedback and unstated pressures to just push things through the system, regardless of potential issues. Perhaps the only thing I agree with from Boeing’s side is that they can’t find all defects, just like we can’t find all bugs. However, we can often find many and minimize or eliminate the impact of the critical ones.

    The pressure on Boeing engineers and line workers is similar to those in software, both on developers and operations. Development needs to deliver features, whether well-tested or not. Operations need to get these out, regardless of the impact on the system. It’s a gross generalization, but one repeated over and over in a few places I’ve worked.

    Culture is critical. Part of DevOps is frequent deployments from a faster speed of deployment. However, that’s only one part. The other is that we continually learn and improve how we do things. We raise the bar of quality. If we are only focused on moving forward without getting better, we get into the position that Boeing was in. No trust, no psychological safety, and no attempt by management to implement DevOps culture. Instead, they’re just pushing for more features with (hopefully) more automated testing, but no real impetus to ensure this is the case.

    More and more of the world is run by software. Whether this is in cars, restaurants, or government. When software doesn’t work well, there can be substantial problems that not only impact a bottom line, but can prevent people from getting goods or services they depend on. I know many of you that build and operate software try to do a good job, and I hope you know this means polishing your craft and improving your abilities. I also hope that management will learn that their support in improving quality is as important as the drive for more features in software.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.