Voice of the DBA

Tag: software development

  • What’s the Edge Case?

    Quite a few of the bugs we see in production systems come from data that isn’t handled well. Perhaps the developer never considered this data, or another bug lets data into a system that should never be recorded. These are often NULL values, but they could be other data values that are far out of the ordinary.

    Where do we draw the line for edge cases? Is it anything that doesn’t fit 95% of the data range? I see this number used in many fields, often manufacturing and other “physical endeavors”. Is it the 80% rule, where we ensure 80% of data cases are covered, but 20% represent special handling?

    It’s an interesting thought because drawing this line helps us decide what level of data we need in our dev and test environments. We need enough data to represent what exists in production, but not much more. The less data we have, the faster everything moves, with much less friction in setting up, resetting, and moving these databases around.

    However, the more bugs that slip through, the more we might need to ensure that we add more data to our development environments to mimic what is in production. Often we have used copies of production data, but there are plenty of issues with this. First, we often have less security in non-production environments and no shortage of data breaches comes from these systems. Therefore we might need to apply masking/obfuscation/pseudomization to values. Second, production databases are growing larger, often over 1 TB. While storage and bandwidth are cheap, they aren’t free, and moving around 1 TB of data regularly, or even restoring it, can present resource challenges.

    My preference is a representative set of data from production, masked and without PII, along with some randomness that might catch edge cases before we deploy changes to production. With that in mind, what’s the edge case? I think I’d lean towards the 95% value, but ready to lower that if we discover many bugs.

    How many are many bugs? I might apply the same standard. If more than 5% of bugs filed are data issues, we need better dev/test data.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Losing Skills Because of Automation

    When I was younger, it seemed that everyone I worked with in technology knew how to build a computer. Most knew how to work with a BIOS, were comfortable with command lines, and could assemble complex compiler directives into a Make file. Over time, it seems many people, especially Windows and MacOS users, became focused on the things they needed for their job, lacking a lot of knowledge about how computers process instructions and the low-level operations they perform.

    There are plenty of very talented developers out there, and many great data modelers, but as I work with many of them that try to make the transition to DevOps, I see lots of uncertainty and tentative behavior. They often approach builds, automated tests, and deployments as though these are completely new skills they need to learn. New software that is foreign to them.

    I wonder if the increased use of automation will make this worse in the future? Already plenty of companies are looking to low-code and no-code solutions as a way to handle the lack of staff to perform development. Will that be exacerbated in the future as more and more automation is put in place that mocks up a shell (or more) of a project and handles work for the developers and even operations staff? Will they struggle to debug complex problems, which are the more likely problems to occur in modern software?

    There is an interesting article about surgeons that might be losing some skills with the advent of robots helping in surgery. While there are some worrisome aspects to this for me, as someone getting older and possibly needing medical care, I find that some of this is applicable to the world of software. How well do we apprentice people new to our environment and give them the chance to build new skills? Often we have senior people taking on interesting work, making data modeling decisions, troubleshooting issues and more. Do we allow more junior people the chance to get hands-on experience and perhaps take charge to lead others? To learn to actually be the one to make the decision?

    In a few places, I’ve seen senior people fixing bugs and junior people developing code. That seemed strange at first, until I realized that fixing problems is more than likely something I want the better developer dealing with, not the worse one. These don’t even have to be issues in production. In a DevOps world, I might have the senior people looking at and fixing the bugs that are caught in CI. While I appreciate giving someone the chance to correct their mistakes, I also think that a “refactoring” or improving of code might be something senior people are better positioned to tackle.

    Of course, I think rotating people in and out of roles, giving them a chance to experience difference sides of our industry, is a good idea. This might include learning how dev works (for Ops people) and how Ops works (for developers) can bring about empathy, understanding, some skills, but likely to create a better culture of collaboration. A tenet of DevOps.

    What skills are we worried about learning? Or losing? To what extent ought we try to ensure others grow wide and deeper outside of their core skills? I find this to be a area that the best DevOps companies do well. They have champions that can provide assistance, knowledge, and teaching, not just do the work. Companies that pretend to adopt DevOps aim for specialization, letting others do the work when they can. Over time, I think they’ll end up like some of today’s surgeons: only practicing their craft for the first time (with no training) when someone else isn’t available.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Good Enough

    I find that many of us that work in technology want a solution to be great. Often perfect. Computers are so good at following instructions that we expect them to work as we intend all the time. We aim for being perfect in our code and we find “good enough” to be a low bar.

    I think this is because many times we deal with code that barely works and gets deployed because it’s good enough. Maybe the code can’t handle a load or corner cases and that’s caused us some sort of stress or pain. So we don’t want “good enough”. because we’ve been burned in the past, either by our code or someone else’s.

    However, good enough means to me that it is good enough for most situations. It’s not leaving something unfinished or undone, but finishing something that works well. I often tackle small projects or minor tasks in this manner, usually because of time crunches. I consider the performance impacts, but in a realistic sense. Not all of my code will be accessed by thousands of users trying to process millions of rows. There are plenty of systems where we know the code will be thousands of rows and dozens of users. We should write the best code we can, and spend time improving our coding skills over time, but we don’t need to over-engineer code too early.

    At the same time, we should not be satisfied with code that we think will not meet the needs of our customers. Whether in function or performance. Sometimes we make quick decisions, knowing we will need to rework the code. In that case, we need to allocate time, file a bug ourselves, or do something else to ensure we revisit this code. That has been the case often with administrative code that I need to run today, but I know this will need to be run regularly over time. I might shortcut code with a cursor or loop, but plan on finding a better solution that performs better and is more robust. If the task can be automated, this means I need to include error handling, logging, and more.

    Life is a series of trade-offs. I rarely get something done as perfectly as I’d like, but that doesn’t stop me from trying to be better over time. I can chase perfection, especially in code. If I regularly learn and change how I code, I find that my “good enough” becomes better and better every month.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Code Supply Chain Security

    There have been a number of attacks in the last few years on source code. In fact, I saw a new one this week for an e-commerce WordPress plugin. This time hackers got access to the distribution server for the company, Fishpig, and altered the plug-ins that their customers download.

    A few years ago this was big news, with the SolarWinds exploit. There was also an attack on PyPy, a popular Python package that many people include in their code.  There have been no shortages of problems in npm packages as well. I’m sure this has happened in other software packages, which is scary. In the days of DevOps where we publish code from a repository, an exploit against your developers might go unnoticed. Then again, maybe not.

    Would any of us notice new code in a file share or a folder on our system? We might just compile a large project without realizing it. At least with DevOps, we have the opportunity to include security scans and code analysis checks, some of which could look for known patterns of exploits. I know some companies use these, and often compromised or vulnerable packages are stopped by the automated pipelines.

    In the US, various security agencies have released a set of recommendations, as has the Open Source Security Foundation. Both of these are designed to help developers secure their supply chain against attacks. This is likely going to be a continuous problem for software vendors in the future as it’s much easier to attack one vendor whose software many people use than each individual company. I shudder to think about what happens if someone manages to get a ransomware package into a vendor’s codebase.

    Ultimately, there will still be problems. Many new projects begin with poor practices precisely because they’re experiments and the authors don’t know if others will find the software valuable.  While we can have good templates and security controls, I’m not hopeful. To me, the best solution for stopping code is to have patterns detectable by security checks in the pipeline. Checks that can be expanded and enhanced as new issues are determined.

    Of course, that means the makers of security software need to ensure their supply chain is protected as well.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.