Voice of the DBA

Tag: software development

  • Large PRs are Bad

    I heard a joke years ago that went something like this. When a developer gets a pull request for code review that’s 100 lines long, they will open the file(s), look at the code, and ensure standards are being followed. They might run some code, they’d walk through the algorithm in their mind, and spend 10, 20, or more minutes examining how the change is built. If you give them a code review of 1000 lines, they’ll just assume it’s great and approve it in seconds.

    I know that many developers don’t find that funny. Often I meet people that think they’re professional and they spend the time needed to examine the changes and ensure they work. I’m sure that many people do this often, and some people are very meticulous about their evaluation of the change. However, everyone gets busy and I know that often depending on who submitted the PR and how busy the reviewers are, the joke does reflect reality. The longer the PR, the less likely someone will either a) spend the time to carefully review it, or b) catch small mistakes.

    There have been a few large profile outages in 2023, one of which was an Azure DevOps outage in Brazil. I’m not picking on Microsoft as AWS, GitHub, GCP, and others have had issues. I know GitHub is part of Microsoft, but it’s also a separate enterprise that really runs on its own in many ways. The point is more that there will be issues, and some of these are related to the rapid changes of a DevOps or GitOps workflow where PRs aren’t always reviewed clearly and cleanly.

    In this case, there was a typo in how a process worked. A cleanup process was supposed to delete databases in Azure, but the typo had it deleting the logical servers. Those had many databases, not just old ones that needed cleanup. This PR, however, contained a lot of changes, as there was an upgrade to swap out older Azure Manager packages with Resource Manager packages. I don’t know if the cleanup job was related here or included in a large PR, but in any case, the PR was reviewed as in the joke above. It was approved and things started failing.

    This wasn’t caught in testing as there wasn’t great test coverage. You can say MS should have more tests, and they should, but there will never be enough test coverage. There also weren’t any systems in their ring 0 (first) deployment that triggered this typo, so no one realized there was an issue. Again, ring 0 systems might not be representative of larger rings. Another reality that we aren’t likely to fix in every situation.

    Microsoft recovered the data, but it took a long time. I don’t know how easy or feasible it is to create smaller PRs with something like this when you are upgrading packages in many files. I just know that the time that I make large-scale changes in code, with large PRs, often I find some problem somewhere. Especially if there are changes that aren’t all the related. If too many different things get included, the potential for mistakes and problems goes up.

    I think this is actually a good DevOps story. They ran their process and there was a mistake. They fixed it and have started to adjust their process to add more testing in this area and potentially ensure this doesn’t happen again. The logging helped them diagnose the issue quickly once it was reported. Their ability to deploy on-demand meant that once the problem was understood, a fix could be quickly deployed. That’s what DevOps is: it’s not perfect, but it does allow us to understand, learn, and adapt quickly.

    Now we just need to ensure that humans use the process in a way that other humans can more easily understand, with smaller PRs.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Creative Development

    I was working with a customer recently that has a development process that both made me cringe and struck me as very creative. In this case, the customer has software they have written when spawning a few databases for each new project that is created. There are three types of databases created for each project, with unique names for the project. The DDL and DML to create these initial databases are stored in a central database as a set of rows in a table.

    To make changes to their software, developers create a project (with new databases) and then alter the application and the database to meet the requirements. These changes then need to be captured and applied to a development template database. There is a template for each of the three different databases created for projects. Changes made to these are processed and then added to the central database for new projects, and to upgrade existing ones.

    This creates a complex development process with lots of potential for mistakes and simple human error. However, that’s the state of the software, and so I’ve been trying to help them find ways to simplify this as well as make it more robust across time (and staff changes).

    The situation got me thinking, however. While they or I might not like the process, I do admire the creativity it took to set this up and build a system that allows custom software to meet their needs for project tracking work. It’s a solution that works, albeit one that now looks overly complicated. However, I wasn’t part of the initial design or the various evolutions since then. Perhaps I’d have ended up in a similar place, given the knowledge and requirements known at each point in time.

    I’m sure many of you have an architecture or a process that is unusual in some way. Perhaps you designed it, or perhaps someone else did, but there was some creativity in building a solution to a problem. Today I’m looking to hear the stories of where you’ve seen creative solutions in development, either to a programming problem or maybe to a process that manages or deploys your software. Maybe you deal with remote systems that aren’t connected. Maybe you work with large numbers of sharded databases. Perhaps you have cultural challenges that require creativity to ensure you can update your software.

    Let us know today what sorts of creative development solutions you’ve seen implemented.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • A Paste Policy

    Most of the code I’ve written for various employers was something that lived inside the organization. This was for internal use and no one outside of the organization ever used it. It was almost always code I’d written, so I wasn’t ever worried about the provenance of the code (where it came from).

    However, I’m sure I’ve copied code from somewhere online and used that in some way. I might have taken this function from Jeff Moden and dropped it on a production server where it was used to process a string. I’m sure I’ve taken code from various places and used it in administrative functions, like Ginger’s code to manage a log file.

    I don’t know that any of those authors would have minded using their code. After all, that’s why many of us have written articles and published things at SQL Server Central. We are looking to help others grow their knowledge, as well as provide them short solutions they can incorporate into their own work. I assume most of the people answering questions at SQL Server Central or Stack Overflow feel the same way. Use the code.

    In this new age of AI, perhaps that’s OK, or perhaps it’s not. There is concern about who owns code and who has rights. At Redgate, we build software that’s sold and installed on other machines, so we have stricter guidelines. Our policy (in general) is “You MUST NOT include source code of unknown provenance in your application.” I saw a note recently that said this also applies to any AI-generated code, even if we provide the prompts.

    I doubt many organizations have policies about using code from the Internet, but I wonder if they should. Possibly there might be ownership or usage issues with AI code. If someone posts code on SSC that’s one thing, but getting code from an AI that might have been trained (and regurgitates) code that is not freely posted could be a problem.

    Maybe it doesn’t matter. After all, in your corporate CRUD application or some DBA maintenance script, it’s unlikely anyone will be auditing the code for any sort of misuse or copyright infringement.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

  • Database APIs

    Creating an API and using an API server of some sort has become a very common way of working with modern software, especially distributed systems. Whether developers move to microservices or something else, they often make more use of APIs than ever before. At Redgate, I’ve watched us work on new products, often beginning with a basic API at the command line and evolving from there.

    However, does the database need an API? This article thinks so, though I don’t love the piece. It seems to wander a bit amongst a few topics and doesn’t always fit together. It’s also trying to promote the Stargate API gateway for Cassandra. I don’t know if this is a good idea, or if it really helps make developers more productive, but the idea of having an API for developers makes sense.

    In fact, when I work with clients I often try to get them to think of their tables as an API of sorts. With APIs, we want to rev them, but carefully. After all, we know there are dependencies from various other pieces of software, including other items inside the database. We want to be sure we don’t break anything. At least not for long, which is why DevOps and being able to release work quickly matters.

    Communicating changes and coordinating work is a challenge. I was surprised by a client who used SQL Doc for this reason, to ensure everyone had up-to-date knowledge of what their schema meant. It’s also a reason I’ve taken some of the work I do with clients and built my Architecting Zero Downtime talk. Many developers think narrowly about the requirement in front of them and not about all the other dependencies and impacts. I think a bit more unit testing in databases would help here. In fact, I demonstrate sometimes how to write a test that treats your table like an API to prevent breakage.

    I do agree that databases, especially relational ones can be complex. If we can simplify things for developers, especially developers new to our system, we can get more work done and hopefully, raise code quality. I don’t know that Stargate or GraphQL or anything else is better, though I have started to think LINQ is a better implementation than SQL for working with relational databases. That’s not going to change anytime soon, and plus, I need SQL to remain popular until I retire 😉

    Whether you formally build an API or not, it is helpful to treat your schema like an API and make changes in the same way that you would if there were other teams depending on your interface. That’s because there likely are others who depend on what you build.

    Steve Jones

    Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.