Tag: software development

I’ve spent quite a bit of my career as a DBA/sysadmin/Operations person. However, I’ve had my share of development positions as well. As I work with customers who look to mature their database development to be more like other software development, I’ve noticed that PRs sometimes don’t get handled as smoothly as we might like. In some sense, they are like help desk trouble tickets that never get closed.

One of the first things I caution people about is specifying specific reviewers, especially DBAs. There are often DBAs who are the gatekeepers for code, but if we require them to be the only ones to review code before a CI or test process, we really slow things down. This often happens in smaller environments where one DBA wants to avoid anything impacting their job. They want to review everything before it commits.

That’s a bottleneck that will slow you down. Train others, agree on the types of changes that can be approved by multiple people, and learn to handle the problems, not try to prevent them all.

Another issue is that lots of people are often overloaded with their workload. It’s a big reason people want to adopt a more DevOps view of the world and try to reduce time spent on handoffs and the friction in the process. However, until things get better, they can get worse. PRs can languish when people are overloaded. Like the first item above, choosing specific people for a review can make this worse, but even if you have a group to review requests, you might find everyone expects someone else to look at the request.

There’s no good solution here, but establishing a rota for reviews or setting aside time in a calendar can help. There will still be cases when you need immediate review, so there needs to be some escalation process, at least until everyone acts like a team and tries to ensure the others aren’t waiting a long time for their work to be reviewed.

The last thing about PRs that I find similar to trouble tickets is the content. I have had tickets where I have MB of multiple text logs, and at the same time, a bare-bones description like “this doesn’t work.” PRs can be the same way, with too many changes and too sparse a description. Your commit message(s) might not be good for a PR. Use the PR space to give the reviewer some context. Try to limit the changes, though I know sometimes this can be hard. As much as possible, focus on making non-breaking changes in code, tying schema alterations to feature flags so that you can merge in smaller bits of work without breaking the system.

Learning to create a more mature database development process takes work. Software developers have gotten better, though they still face some struggles in managing their PRs. Learn from them, ask questions, take advice, and above all, work as a team.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

At an event recently, I had a chat with someone after one of my sessions. I had been speaking on DevOps and ways to better structure your team and build software. After the session, one person asked me if I’d read The Mythical Man Month and if I felt we’d gotten a lot better at building software since that book was published.

I do think we have gotten better, way better, in fact. I caught another review of the book a while back from the Pragmatic Engineer. That review looked at what’s changed in 50 years since the first edition, as well as contrasting the world today. You have to subscribe to read that one, but I’ll give you a few thoughts from me on the book itself and the review.

Perhaps the most famous part of the book is the notion that adding more people to a late project makes it later. That doesn’t always happen in other fields, where people can tackle separate tasks and get things done. Certainly feeding hay to horses goes quicker with more people. However, in software, things don’t work that way. Perhaps this can work better today if we use microservices and separate architectural sections, but for any single piece of software, this often holds true. Even for microservices I’m not sure it’s true because developers take time to get productive.

And yet, people still try to add more staff to projects in the hope they’ll complete the application sooner.

Another great quote from the book is that programmers (developers today) can build castles in the sky. We can use our imagination to create, polish, and re-work in a way that isn’t easy in the real world. That, along with the joy, complexity, and opportunity to learn, are why people program. The Pragmatic Engineer’s view notes that there are other reasons, such as so much of the world uses software that people want to be a part of that, and the career is lucrative. I agree with that for sure. Working with code is a good job in many ways.

The book notes several challenges to building large systems, such as precise coding, lack of control, dependencies, debugging, and obsolescence from delays. Of these, some are true, but we have tools, like SQL Prompt, that help us avoid poor typing of code (and CI for checking). We also have moved to a much faster pace of delivering parts of a system and evolving them, so we are  often targeting just what our customers need, at an apparent faster pace. I’d also say that being able to deliver something quickly is important, as customers are quick to move on if you cannot.

One interesting part of the book that isn’t quite so relevant is the discussion about why projects are late. While I don’t know we estimate better, we certainly are better at getting smaller pieces of work done, which might hide some of the delays for larger features. We have so many more ways to track work and ensure we aren’t forgetting what we have planned. There are also many more good managers today that empower their people. These are also the things in DevOps that have vastly improved how we build software.

Onboarding is a major issue in the book, as finding staff who knew tools/techniques/whatever was hard back then. There just weren’t many people. Today, we struggle to onboard some people, but I’d like to think that it’s not the same as in the past. However, I certainly think database techies in many cases haven’t kept up or learned a lot of core software dev practices, such as version control, CI, and lots of things we bundle into DevOps. At the same time, I think we’ve burdened many people with full-stack development when they don’t have a good understanding of core technologies, like databases and networking. I still think onboarding people quickly is a major advantage for any company that needs to build software and compete with others in their industry.

In the book, Brooks talks about the 10x engineer, a hotly debated topic in the modern software world. Are some people much more productive? I do think so, though if they are 10x as skilled as many others, I’d argue the others aren’t earning their salaries.

If you’ve never read the Mythical Man Month, pick up a copy. It’s worth your time as a developer or a DBA.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

The use of feature flags in software development has become more and more prevalent over time, especially as teams move to DevOps-style development with frequent releases. I’ve often thought that using feature flags allows technical people to separate out the deployment of some feature or change from the release of that to users. There are a number of articles on this style of work (feature flag driven development, Why Use Feature Flags?) as well as a discussion at Reddit.

I am a big believer in feature flags helping with improving your software in many ways. These articles (and others) highlight the advantages that a software organization gains by using feature flags. Failed releases become less of an issue, as the specific change that doesn’t work can be turned off. This can even work with databases. I can deploy a database change and at a later time have the code (or new table/column) start being used when a feature flag is set. If there is an issue, I can turn off the feature flag and stop using the code (or populating the schema). I can then clean things up, even saving data before I make a change.

I don’t love the idea of using feature flags to handle security access to features, which is pointed out in a few places. If this is for testing or evaluation by customers, perhaps. If this is to get access to data from a security standpoint, this is a bad idea. I hope most of you are savvy enough to realize this.

Feature flags are not a panacea for preventing issues. They do clutter up code and make it harder to read. Once a feature is done and permanently enabled, the code for switching flags should be removed. It is also hard to stack versions of features up behind one flag, which can increase coding mistakes. Adding flags in stored procedures or functions also can wreak havoc on query optimizers, so I’d recommend you don’t do that. Instead handle feature enablement in the application code and use multiple procs/functions for the different functionality you might need.

To use feature flags appropriately with database changes, you also need to be able to dark deploy those changes, with your application code able to handle additive changes to the database. A new column, a new table, or a new parameter should be easy to add without breaking the app code. This requires the use of defaults as well as good coding practices (no select *, inserts with column lists), but it can be done. Once you are in this place, life becomes a lot less stressful and feature flags work amazingly well.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.