Tag: software development

The use of feature flags in software development has become more and more prevalent over time, especially as teams move to DevOps-style development with frequent releases. I’ve often thought that using feature flags allows technical people to separate out the deployment of some feature or change from the release of that to users. There are a number of articles on this style of work (feature flag driven development, Why Use Feature Flags?) as well as a discussion at Reddit.

I am a big believer in feature flags helping with improving your software in many ways. These articles (and others) highlight the advantages that a software organization gains by using feature flags. Failed releases become less of an issue, as the specific change that doesn’t work can be turned off. This can even work with databases. I can deploy a database change and at a later time have the code (or new table/column) start being used when a feature flag is set. If there is an issue, I can turn off the feature flag and stop using the code (or populating the schema). I can then clean things up, even saving data before I make a change.

I don’t love the idea of using feature flags to handle security access to features, which is pointed out in a few places. If this is for testing or evaluation by customers, perhaps. If this is to get access to data from a security standpoint, this is a bad idea. I hope most of you are savvy enough to realize this.

Feature flags are not a panacea for preventing issues. They do clutter up code and make it harder to read. Once a feature is done and permanently enabled, the code for switching flags should be removed. It is also hard to stack versions of features up behind one flag, which can increase coding mistakes. Adding flags in stored procedures or functions also can wreak havoc on query optimizers, so I’d recommend you don’t do that. Instead handle feature enablement in the application code and use multiple procs/functions for the different functionality you might need.

To use feature flags appropriately with database changes, you also need to be able to dark deploy those changes, with your application code able to handle additive changes to the database. A new column, a new table, or a new parameter should be easy to add without breaking the app code. This requires the use of defaults as well as good coding practices (no select *, inserts with column lists), but it can be done. Once you are in this place, life becomes a lot less stressful and feature flags work amazingly well.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

I have run into a lot of people in the last few years that love decoupled software and microservices. It seems many people are aiming to move their work in this direction, and while I see some appeal, I also see tremendous additional complexity that has moved out of the software into the operations and debugging space. As I read the book Observability Engineering, I found myself thinking the complexity of setting up more logging and instrumentation in an observability framework as well as the costs of managing a system start. That caused me to think this is overkill for most software.

To be clear, I don’t think that Uber could have been built as a few monoliths, and there are other examples of such systems, but most of us don’t work at that scale. There are lessons to be learned about large, real-time software systems, and certainly Google, Amazon, Spotify, Netflix, etc. can help us understand how certain techniques work better at scale, but most of us don’t work at that scale.

Scale to me is thousands of connections and terabytes of data. Those companies tend to work at a couple orders of magnitude above that. Those are also companies that must have real time systems up to survive. The vast majority of companies I’ve worked at might suffer a loss with an outage of a system, but honestly, we could survive a day or two of recovery.

Heck, look at all of the companies that have had portions of their digital infrastructure knocked offline from ransomware the last few years. Some failed, but most didn’t. The incident sucked for IT staff, but those companies survived. I still remember the SQL Slammer worm forcing us to take our entire network offline for multiple days at a large software company. We still ran sales, and support, and other systems independently or from paper. Granted that was 20 years ago, but I’m sure Redgate and many other organizations could survive a few weeks with no network.

Uber on the other hand, that would be a disaster for them. They would likely survive, but at a high cost. How many people would jump to a new service and never look back?

I was reading a piece on coupling and complexity, which I’ll discuss a bit in another article, but it got my thinking of how often I see people overcomplicating their work by trying to move to a decoupled world instead of hiring and training people to adopt better architectures and communicate better. After all, moving to microservices isn’t going to avoid the training issue. You’ll still have to teach people more.

And while you’re at it, force every developer to pass a SQL test every year. That might help you build better systems as well. SQL isn’t going away and better SQL coding in will result in much better applications everywhere.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

If your job as a developer or DBA has been like mine, it’s a constant stream of requests to change something, often without enough information and short deadlines that create a bit of stress. There’s always more work to be done, and while it might be a great job, you’re often trying to finish something quickly enough to get to the next thing.

In this mode, how often do you think about creating (or modifying) the thing you’re working on for today vs maintaining it for tomorrow. In other words, do you consider how easily your work can be understood, is documented, is designed to allow for flexibility, and can be enhanced without many (any?) side effects, or anything else.

In other words, is it maintainable?

We often build things to solve a problem and we can be very creative. We design a solution that works well, solves a problem, and may work very efficiently. However, I know I often haven’t thought about future maintenance. I haven’t considered how difficult it may be to understand the context around what I put together by anyone else. Sometimes I’ve built things that things that were very clever, but weren’t easily understood by my team or able to easily adapt to new requirements when those arise.

Often I’ve been looking at the problem from only a “it’s this tree that’s important” perspective, and forgetting that my particular thing is part of a wider system (a forest). I hadn’t been considering future maintenance, which often led someone in the future (often a future-me) to tear it down and rebuild something new.

That’s technical debt.

When some structure isn’t maintainable, it’s debt. It’s a burden for the team in the future. Designing things quickly and building them within a deadline, while making them maintainable requires some knowledge, experience, and also discipline to work with the patterns, and avoiding the anti-patterns, that make code difficult. The same thing applies to managing systems. Custom jobs on every server and separate configurations make life hard. At the same time, a one-size-for-everyone approach also isn’t maintainable. We need a balance of well-written solutions that solve our problems, but are easily maintainable.

Part of becoming a better engineer or admin is learning how to build maintainable things that others will continue to use for a long time, not because they have to, but because they want to because the code works well.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Starbucks has too many order combinations. This article about the challenges for their new CEO notes there are over 170,000 combinations of things customers can order. As someone who has visited a store often and stood behind someone placing an order, that sounds like an accurate number. Sometimes I hear people order things to drink that I had no idea were even possible.

When there are too many choices, sometimes there can be a delay as the person tries to process the information and make a decision. Some people can’t make a decision. Others enjoy the variety and keep making different decisions each time. That’s fine if you’re visiting a restaurant, but it’s less useful if you’re building software.

I have worked with people who struggle to decide on an approach to solving a problem. In the C++ world, I’d see people struggling to organize classes and members, sometimes spending too much time researching and comparing different methods. They debate when is private or public appropriate, when do we need a superclass or subclass? We would see some paralysis by analysis from certain developers who are endlessly searching for the optimal approach.

Other developers might change their approach constantly, resulting in a lack of consistency across the codebase. In the modern world, I sometimes see users experimenting with different libraries or packages (or data stores) with each new project. They might be consistent within a project for a period of time, but since many teams often shift to different projects over time (or maintain them), how helpful is this? If each new project looks different, we spend a lot of time on developers getting familiar with the technology.

I have tended to adopt the philosophy of strong opinions, loosely held. In other words, I’ll pick an approach/platform/technology/etc. and stick with it. Moreover, I’ll ask everyone on the team to stick to a similar method of coding. If we need to change, then we can evaluate the proposed change, performance test it, and if we decide it’s better, we all adopt this and slowly work to refactor old code as we have time. We never have time, so we only refactor when we have to touch old code for some fix or enhancement. However, when we do touch old code, let’s improve it.

My preference is to allow a limited set of technologies in our company at any point in time. We need a process to add or remove something, but this ought to be a decision made to limit our range of technologies and required expertise. I’m not against Python or PowerShell or MongoDB or any other technology, but I don’t want things adopted because an individual prefers that technology or thinks it works better. They need to prove that there is a good reason to others, and as a group, we agree that the value of adding a new technology is worth the burden.

Otherwise, we end up in the situation I see often in job descriptions: with a requirement that new hires know dozens of technologies at an expert level because we don’t have that person on staff.

And we’ll never find them.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.