Voice of the DBA

Tag: software development

  • Self-Service Databases

    When I first started work as a software developer, I knew that getting an environment set up where I could compile a project might take a few hours or a few days. The complexities of how people built software projects, the dependencies, and more were handled in a very immature manner. These days I can often download a project from GitHub and compile it in minutes, with many of the issues handled automatically with various tools such as complex config files and package managers.

    In the world of database development, I often find people struggle to get a database set up with a project. Whether you are coding in SQL, C#, or Java, many organizations haven’t got a standard way of bringing up a database environment for developers. Some have a shared database for a team, which can be better, but can also lead to collisions of code as the pressure to get software built grows and team size increases.

    This leaves out the challenges of data, which can be changed by others, or lack the size/shape/skew to properly reflect what the production code has to handle. There are no shortage of bugs that come from incomplete data sets used in development that are unrepresentative of the live database.

    Do you have a way for developers to self-service the database assets they need? Whether this is a new dev database, a refreshed one from prod, or even a point in time from development.  Is there a way developers can get a database into the state that matters for them. When developers are ready to work on a problem, the last thing we want is a significant delay to them getting started.

    One of the things DevOps stresses is that we want to reduce friction and tedious, manual work were possible. Let the computer manage the tedious things. This should be how we manage those database environments. We also want some consistency among all developers, QA, CI, and any other environment, to ensure we can reproduce problems to troubleshoot issues.

    The customers I work with that have the smoothest development processes, and best ROI from the resources spent on development, have a way to hydrate a consistent database environment for their developers. More importantly, the developers can self-service this. When a single developer, or a team, decides to reset their environment, they can do so, applying in-flight code changes from version control to bring back the database to the state it was previously in.

    Application developers do this all the time, as they restart their application. Removing their abliity to change code, restart the app, test it, and observe results easily would be crippling. Yet, we rarely reset dev databases, and often, don’t have consistency between environments.

    Self-service is one of the key indicators of a mature software development organization. This is easily achievable for many app projects, but not simple for databases. Relatively few groups have figured out how to do this well, but those that do are much more productive and efficient than everyone else. Traits that are becoming more important in this highly competitive world with increasingly fickle customers.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • What’s the Edge Case?

    Quite a few of the bugs we see in production systems come from data that isn’t handled well. Perhaps the developer never considered this data, or another bug lets data into a system that should never be recorded. These are often NULL values, but they could be other data values that are far out of the ordinary.

    Where do we draw the line for edge cases? Is it anything that doesn’t fit 95% of the data range? I see this number used in many fields, often manufacturing and other “physical endeavors”. Is it the 80% rule, where we ensure 80% of data cases are covered, but 20% represent special handling?

    It’s an interesting thought because drawing this line helps us decide what level of data we need in our dev and test environments. We need enough data to represent what exists in production, but not much more. The less data we have, the faster everything moves, with much less friction in setting up, resetting, and moving these databases around.

    However, the more bugs that slip through, the more we might need to ensure that we add more data to our development environments to mimic what is in production. Often we have used copies of production data, but there are plenty of issues with this. First, we often have less security in non-production environments and no shortage of data breaches comes from these systems. Therefore we might need to apply masking/obfuscation/pseudomization to values. Second, production databases are growing larger, often over 1 TB. While storage and bandwidth are cheap, they aren’t free, and moving around 1 TB of data regularly, or even restoring it, can present resource challenges.

    My preference is a representative set of data from production, masked and without PII, along with some randomness that might catch edge cases before we deploy changes to production. With that in mind, what’s the edge case? I think I’d lean towards the 95% value, but ready to lower that if we discover many bugs.

    How many are many bugs? I might apply the same standard. If more than 5% of bugs filed are data issues, we need better dev/test data.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Losing Skills Because of Automation

    When I was younger, it seemed that everyone I worked with in technology knew how to build a computer. Most knew how to work with a BIOS, were comfortable with command lines, and could assemble complex compiler directives into a Make file. Over time, it seems many people, especially Windows and MacOS users, became focused on the things they needed for their job, lacking a lot of knowledge about how computers process instructions and the low-level operations they perform.

    There are plenty of very talented developers out there, and many great data modelers, but as I work with many of them that try to make the transition to DevOps, I see lots of uncertainty and tentative behavior. They often approach builds, automated tests, and deployments as though these are completely new skills they need to learn. New software that is foreign to them.

    I wonder if the increased use of automation will make this worse in the future? Already plenty of companies are looking to low-code and no-code solutions as a way to handle the lack of staff to perform development. Will that be exacerbated in the future as more and more automation is put in place that mocks up a shell (or more) of a project and handles work for the developers and even operations staff? Will they struggle to debug complex problems, which are the more likely problems to occur in modern software?

    There is an interesting article about surgeons that might be losing some skills with the advent of robots helping in surgery. While there are some worrisome aspects to this for me, as someone getting older and possibly needing medical care, I find that some of this is applicable to the world of software. How well do we apprentice people new to our environment and give them the chance to build new skills? Often we have senior people taking on interesting work, making data modeling decisions, troubleshooting issues and more. Do we allow more junior people the chance to get hands-on experience and perhaps take charge to lead others? To learn to actually be the one to make the decision?

    In a few places, I’ve seen senior people fixing bugs and junior people developing code. That seemed strange at first, until I realized that fixing problems is more than likely something I want the better developer dealing with, not the worse one. These don’t even have to be issues in production. In a DevOps world, I might have the senior people looking at and fixing the bugs that are caught in CI. While I appreciate giving someone the chance to correct their mistakes, I also think that a “refactoring” or improving of code might be something senior people are better positioned to tackle.

    Of course, I think rotating people in and out of roles, giving them a chance to experience difference sides of our industry, is a good idea. This might include learning how dev works (for Ops people) and how Ops works (for developers) can bring about empathy, understanding, some skills, but likely to create a better culture of collaboration. A tenet of DevOps.

    What skills are we worried about learning? Or losing? To what extent ought we try to ensure others grow wide and deeper outside of their core skills? I find this to be a area that the best DevOps companies do well. They have champions that can provide assistance, knowledge, and teaching, not just do the work. Companies that pretend to adopt DevOps aim for specialization, letting others do the work when they can. Over time, I think they’ll end up like some of today’s surgeons: only practicing their craft for the first time (with no training) when someone else isn’t available.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Good Enough

    I find that many of us that work in technology want a solution to be great. Often perfect. Computers are so good at following instructions that we expect them to work as we intend all the time. We aim for being perfect in our code and we find “good enough” to be a low bar.

    I think this is because many times we deal with code that barely works and gets deployed because it’s good enough. Maybe the code can’t handle a load or corner cases and that’s caused us some sort of stress or pain. So we don’t want “good enough”. because we’ve been burned in the past, either by our code or someone else’s.

    However, good enough means to me that it is good enough for most situations. It’s not leaving something unfinished or undone, but finishing something that works well. I often tackle small projects or minor tasks in this manner, usually because of time crunches. I consider the performance impacts, but in a realistic sense. Not all of my code will be accessed by thousands of users trying to process millions of rows. There are plenty of systems where we know the code will be thousands of rows and dozens of users. We should write the best code we can, and spend time improving our coding skills over time, but we don’t need to over-engineer code too early.

    At the same time, we should not be satisfied with code that we think will not meet the needs of our customers. Whether in function or performance. Sometimes we make quick decisions, knowing we will need to rework the code. In that case, we need to allocate time, file a bug ourselves, or do something else to ensure we revisit this code. That has been the case often with administrative code that I need to run today, but I know this will need to be run regularly over time. I might shortcut code with a cursor or loop, but plan on finding a better solution that performs better and is more robust. If the task can be automated, this means I need to include error handling, logging, and more.

    Life is a series of trade-offs. I rarely get something done as perfectly as I’d like, but that doesn’t stop me from trying to be better over time. I can chase perfection, especially in code. If I regularly learn and change how I code, I find that my “good enough” becomes better and better every month.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.