Voice of the DBA

Tag: software development

  • When Work Isn’t Done

    Software development can be a challenge for each of us with lots of demands and the need to ensure your code solves a problem correctly, efficiently, and completely. Juggling the workload by yourself is one thing, but add in a team of developers, and the complexity quickly grows.

    The real world is chaotic and despite the best efforts of project managers and scrum masters, our software development life cycle doesn’t always proceed smoothly. I wonder how many of you run into this situation and how you deal with it.

    Developer 1 gets a piece of work, let’s call this A. They complete this and send it to the QA team. Somewhere during this process, Developer 2 get’s a different piece of work (B) and writes code. They send this to QA before A is completely tested.

    Now, Developer 1 finds a mistake. Something doesn’t work or they realize their solution is incomplete. QA is A + B, but A doesn’t work and needs revision. B passes testing and needs to be deployed. If your codebase and QA have both A and B in them, how do you strip out A or B and ensure B is deployed to production but A isn’t?

    If this is C# or Java, you might have one solution, even if both changes are in the same class. If this is database code, you might have a different set of issues to deal with.

    Really, the question is can you reorder work in your deployment process? I find many customers don’t consider this when they are evaluating their software pipeline. They somehow assume if code gets to QA that it’s good, which is nicely optimistic but not realistic. At some point, we’ll deploy code to QA that doesn’t work. The more developers we have, the more likely this is, and the more demands on our time, the more likely we need to reorder work and release one thing but not another.

    As a DB developer and DBA in a company 20 years ago, I built a process and forced us to reset the QA environment and redeploy B only (using a branch of code that stripped out A) for re-testing. This ensured that we tested what was going to be deployed. However, I find a lot of organizations can’t do this or don’t want to. They want to hope that a human and either extract all of B or strip out all of A and release partially tested code without issues.

    I find that to be a poor idea. In this era of regular staff changes, staff of varying quality, and the high complexity of software, this is asking for mistakes. With cheap hardware, virtualization, and the ability to provision copies of environments, we ought to do better.

    How do you handle this today? Depend on humans to not make mistakes? Hope for the best? Or follow a repeatable, reliable process that accounts for inexperience and human error?

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Bad Culture Bad Security

    I ran across a blog noting that Cisco has a vulnerability in a new product. The blog also lists two (one, two) articles showing that Cisco has had hard-coded credentials in the past. I understand that many times a known process is repeated, essentially copy-pasted between people, and we have similar issues as we have had in the past. However, in 2022 or 2023, it’s unacceptable to hard-code credentials in digital systems that will be used in today’s world.

    What’s worse than having this issue is stating that the fix is “an upgrade”. Their verbiage for those without a service contract is: “Customers should have the product serial number available and be prepared to provide the URL of this advisory as evidence of entitlement to a free upgrade.” Which, to me, is not only bad for the world, but it’s equivalent to the stuff that bulls leave behind in the fields.

    I suspect that this product and software were based on something that already exists, and it was rushed out without a good security evaluation. Or perhaps there are developers and managers who don’t think that hard-coding credentials will compromise security.

    That’s a cultural problem. Either you’re going too fast, or you don’t take this seriously, either way, you don’t have a good culture that values quality and protection. Certainly, their disclaimer about needing valid contracts or proving you have a device to get a security patch for a flaw they built is also a sign of a poor culture that doesn’t really understand the problems they are creating, nor are they taking responsibility for the issues.

    At the very least, fixing their poor security ought to be free and easy. I get that there are likely some software upgrades included in this patch, given the nature of software development and limited branches under support. However, there are other ways to ensure those features aren’t enabled for customers who shouldn’t have them. Making the entire world of computer network infrastructure less secure because you want customers to pay for your mistakes isn’t a model I’d want to adapt.

    I know many people don’t want more regulation or guidelines from governments (or even from insurance), but if I were going to accept some universal restriction, this would be the place. If you use hard-coded passwords, your product can’t be sold and no insurance claims apply if you are sued.

    Maybe that would change the way Cisco and others build software.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • You Always Have a Software Pipeline

    I was working with a customer recently and they said they didn’t want to set up a pipeline. Somehow they wanted magically to see changes that developers made to their database appear in their production database without making any effort to build or configure a pipeline in a tool like Octopus Deploy.

    As I was thinking about how to find them a solution they would accept, I realized that I’ve always had a pipeline for software. Early on, this was a copy of all the files in a folder sent to the client for Clipper and dBase apps. Later, my pipeline was copying around all the VB6 files from a .zip file and executing a few SQL scripts that were sent to me in emails.

    At one point, I had developers just tell me the software pipeline was them copying files to a server while I had to move all the changes from the QA server to production. I didn’t like that one for two reasons. One was that the developers would sometimes forget to move a change from dev to QA, despite claiming they’d tested the application. Two was that the developers sometimes moved too much to QA, and had changes to the schema that they hadn’t finished in their app. SQL Compare couldn’t help me in either of those cases.

    Over the years, I learned that a slightly thought-out pipeline is better than nothing. I wrote my scripts to simulate what Flyway does today, executing the scripts I’d placed in a folder. I added logging that would capture output results and save them for later review. I learned how tools such as Team City or Jenkins could execute the same thing I could but in a more repeatable, reliable manner. After all, my reliability at 4:55 p.m. on Friday isn’t as good as it is at 2 pm on a Tuesday.

    DevOps is about learning and taking advantage of tooling. Look at what works in your pipeline and find repeatable, reliable ways of accomplishing those tasks without humans doing more than clicking a button to approve something. It’s about using your creativity to improve the process and codifying it in a way. It’s about being better than we were last week while ensuring the process isn’t dependent on any particular person. After all, I might leave for vacation.

    Or find another job.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.

  • Modern Development

    I was browsing online the other day during a break and stumbled upon an article on the modern architecture stack for a startup. It’s not a bad read, and as someone who’s worked in startups, it’s interesting to see what others think. As with many of these articles, it has a lot of practical advice, and it’s also not relevant for many of us because we don’t often work in greenfield development.

    However, we could view adding something new to our application as greenfield. It’s not completely the same, but there are similarities when we start a new feature that doesn’t exist anywhere. Some of their advice, like docker-izing everything, won’t apply, but there is one thing that does apply: data.

    There is a quote, which I really like: “…what is the point of an API running on top of an empty DB? Manually entering necessary data shortly leads to depression (and the risk of increasing the duration of development cycles). Hence, we prepared a curated dataset that was inserted into the local DB to be able to play with.”

    That’s similar to the advice I give when speaking on DevOps. Invest in a curated data set for developers. As this group learned, you can start to use this in testing, CI, etc. and it makes life better for developers. Heck, if you’re using version control and you create a data set for yourself, save the insert statements, put them in a folder, name the script for the table, and share it with others.

    It’s a small change, but it’s one that pays big dividends over time. If others use your INSERTS, and you use theirs, all of a sudden you have a good curated set. If you maintain this as you find bugs or strange things customers enter into production, you don’t need a restore of production; you can add more data and run just those statements. Heck, if you add columns, add some data. You’ll do that anyway to test your new column so maintain this as a script. Or update your data and use SQL Prompt to create INSERT statements for all the data and replace the entirety of your data script.

    A big part of development is maintaining data. While there are virtualization solutions, like Redgate Clone, those can be cumbersome and expensive. They do solve problems, and they might be a good fit for your situation. However, I really like smaller sets of data that duplicate our problem domain. For those of you dealing with time-based problems, include scripts that “update” dates and times to simulate problems from today or yesterday rather than last week. The limit is the creativity of your team, and as you maintain this dataset, everyone benefits from small changes made over time.

    Version control has been a boon to sharing software projects between developers, but underutilized as a way of sharing data as well. Add some data scripts to your project and you might be surprised how much easier it is to work as a team.

    Steve Jones

    Listen to the podcast at Libsyn, Spotify, or iTunes.