Tag: software development

I was browsing online the other day during a break and stumbled upon an article on the modern architecture stack for a startup. It’s not a bad read, and as someone who’s worked in startups, it’s interesting to see what others think. As with many of these articles, it has a lot of practical advice, and it’s also not relevant for many of us because we don’t often work in greenfield development.

However, we could view adding something new to our application as greenfield. It’s not completely the same, but there are similarities when we start a new feature that doesn’t exist anywhere. Some of their advice, like docker-izing everything, won’t apply, but there is one thing that does apply: data.

There is a quote, which I really like: “…what is the point of an API running on top of an empty DB? Manually entering necessary data shortly leads to depression (and the risk of increasing the duration of development cycles). Hence, we prepared a curated dataset that was inserted into the local DB to be able to play with.”

That’s similar to the advice I give when speaking on DevOps. Invest in a curated data set for developers. As this group learned, you can start to use this in testing, CI, etc. and it makes life better for developers. Heck, if you’re using version control and you create a data set for yourself, save the insert statements, put them in a folder, name the script for the table, and share it with others.

It’s a small change, but it’s one that pays big dividends over time. If others use your INSERTS, and you use theirs, all of a sudden you have a good curated set. If you maintain this as you find bugs or strange things customers enter into production, you don’t need a restore of production; you can add more data and run just those statements. Heck, if you add columns, add some data. You’ll do that anyway to test your new column so maintain this as a script. Or update your data and use SQL Prompt to create INSERT statements for all the data and replace the entirety of your data script.

A big part of development is maintaining data. While there are virtualization solutions, like Redgate Clone, those can be cumbersome and expensive. They do solve problems, and they might be a good fit for your situation. However, I really like smaller sets of data that duplicate our problem domain. For those of you dealing with time-based problems, include scripts that “update” dates and times to simulate problems from today or yesterday rather than last week. The limit is the creativity of your team, and as you maintain this dataset, everyone benefits from small changes made over time.

Version control has been a boon to sharing software projects between developers, but underutilized as a way of sharing data as well. Add some data scripts to your project and you might be surprised how much easier it is to work as a team.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

I heard a joke years ago that went something like this. When a developer gets a pull request for code review that’s 100 lines long, they will open the file(s), look at the code, and ensure standards are being followed. They might run some code, they’d walk through the algorithm in their mind, and spend 10, 20, or more minutes examining how the change is built. If you give them a code review of 1000 lines, they’ll just assume it’s great and approve it in seconds.

I know that many developers don’t find that funny. Often I meet people that think they’re professional and they spend the time needed to examine the changes and ensure they work. I’m sure that many people do this often, and some people are very meticulous about their evaluation of the change. However, everyone gets busy and I know that often depending on who submitted the PR and how busy the reviewers are, the joke does reflect reality. The longer the PR, the less likely someone will either a) spend the time to carefully review it, or b) catch small mistakes.

There have been a few large profile outages in 2023, one of which was an Azure DevOps outage in Brazil. I’m not picking on Microsoft as AWS, GitHub, GCP, and others have had issues. I know GitHub is part of Microsoft, but it’s also a separate enterprise that really runs on its own in many ways. The point is more that there will be issues, and some of these are related to the rapid changes of a DevOps or GitOps workflow where PRs aren’t always reviewed clearly and cleanly.

In this case, there was a typo in how a process worked. A cleanup process was supposed to delete databases in Azure, but the typo had it deleting the logical servers. Those had many databases, not just old ones that needed cleanup. This PR, however, contained a lot of changes, as there was an upgrade to swap out older Azure Manager packages with Resource Manager packages. I don’t know if the cleanup job was related here or included in a large PR, but in any case, the PR was reviewed as in the joke above. It was approved and things started failing.

This wasn’t caught in testing as there wasn’t great test coverage. You can say MS should have more tests, and they should, but there will never be enough test coverage. There also weren’t any systems in their ring 0 (first) deployment that triggered this typo, so no one realized there was an issue. Again, ring 0 systems might not be representative of larger rings. Another reality that we aren’t likely to fix in every situation.

Microsoft recovered the data, but it took a long time. I don’t know how easy or feasible it is to create smaller PRs with something like this when you are upgrading packages in many files. I just know that the time that I make large-scale changes in code, with large PRs, often I find some problem somewhere. Especially if there are changes that aren’t all the related. If too many different things get included, the potential for mistakes and problems goes up.

I think this is actually a good DevOps story. They ran their process and there was a mistake. They fixed it and have started to adjust their process to add more testing in this area and potentially ensure this doesn’t happen again. The logging helped them diagnose the issue quickly once it was reported. Their ability to deploy on-demand meant that once the problem was understood, a fix could be quickly deployed. That’s what DevOps is: it’s not perfect, but it does allow us to understand, learn, and adapt quickly.

Now we just need to ensure that humans use the process in a way that other humans can more easily understand, with smaller PRs.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

Most of the code I’ve written for various employers was something that lived inside the organization. This was for internal use and no one outside of the organization ever used it. It was almost always code I’d written, so I wasn’t ever worried about the provenance of the code (where it came from).

However, I’m sure I’ve copied code from somewhere online and used that in some way. I might have taken this function from Jeff Moden and dropped it on a production server where it was used to process a string. I’m sure I’ve taken code from various places and used it in administrative functions, like Ginger’s code to manage a log file.

I don’t know that any of those authors would have minded using their code. After all, that’s why many of us have written articles and published things at SQL Server Central. We are looking to help others grow their knowledge, as well as provide them short solutions they can incorporate into their own work. I assume most of the people answering questions at SQL Server Central or Stack Overflow feel the same way. Use the code.

In this new age of AI, perhaps that’s OK, or perhaps it’s not. There is concern about who owns code and who has rights. At Redgate, we build software that’s sold and installed on other machines, so we have stricter guidelines. Our policy (in general) is “You MUST NOT include source code of unknown provenance in your application.” I saw a note recently that said this also applies to any AI-generated code, even if we provide the prompts.

I doubt many organizations have policies about using code from the Internet, but I wonder if they should. Possibly there might be ownership or usage issues with AI code. If someone posts code on SSC that’s one thing, but getting code from an AI that might have been trained (and regurgitates) code that is not freely posted could be a problem.

Maybe it doesn’t matter. After all, in your corporate CRUD application or some DBA maintenance script, it’s unlikely anyone will be auditing the code for any sort of misuse or copyright infringement.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.