Tag: software development

I have run into a lot of people in the last few years that love decoupled software and microservices. It seems many people are aiming to move their work in this direction, and while I see some appeal, I also see tremendous additional complexity that has moved out of the software into the operations and debugging space. As I read the book Observability Engineering, I found myself thinking the complexity of setting up more logging and instrumentation in an observability framework as well as the costs of managing a system start. That caused me to think this is overkill for most software.

To be clear, I don’t think that Uber could have been built as a few monoliths, and there are other examples of such systems, but most of us don’t work at that scale. There are lessons to be learned about large, real-time software systems, and certainly Google, Amazon, Spotify, Netflix, etc. can help us understand how certain techniques work better at scale, but most of us don’t work at that scale.

Scale to me is thousands of connections and terabytes of data. Those companies tend to work at a couple orders of magnitude above that. Those are also companies that must have real time systems up to survive. The vast majority of companies I’ve worked at might suffer a loss with an outage of a system, but honestly, we could survive a day or two of recovery.

Heck, look at all of the companies that have had portions of their digital infrastructure knocked offline from ransomware the last few years. Some failed, but most didn’t. The incident sucked for IT staff, but those companies survived. I still remember the SQL Slammer worm forcing us to take our entire network offline for multiple days at a large software company. We still ran sales, and support, and other systems independently or from paper. Granted that was 20 years ago, but I’m sure Redgate and many other organizations could survive a few weeks with no network.

Uber on the other hand, that would be a disaster for them. They would likely survive, but at a high cost. How many people would jump to a new service and never look back?

I was reading a piece on coupling and complexity, which I’ll discuss a bit in another article, but it got my thinking of how often I see people overcomplicating their work by trying to move to a decoupled world instead of hiring and training people to adopt better architectures and communicate better. After all, moving to microservices isn’t going to avoid the training issue. You’ll still have to teach people more.

And while you’re at it, force every developer to pass a SQL test every year. That might help you build better systems as well. SQL isn’t going away and better SQL coding in will result in much better applications everywhere.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

If your job as a developer or DBA has been like mine, it’s a constant stream of requests to change something, often without enough information and short deadlines that create a bit of stress. There’s always more work to be done, and while it might be a great job, you’re often trying to finish something quickly enough to get to the next thing.

In this mode, how often do you think about creating (or modifying) the thing you’re working on for today vs maintaining it for tomorrow. In other words, do you consider how easily your work can be understood, is documented, is designed to allow for flexibility, and can be enhanced without many (any?) side effects, or anything else.

In other words, is it maintainable?

We often build things to solve a problem and we can be very creative. We design a solution that works well, solves a problem, and may work very efficiently. However, I know I often haven’t thought about future maintenance. I haven’t considered how difficult it may be to understand the context around what I put together by anyone else. Sometimes I’ve built things that things that were very clever, but weren’t easily understood by my team or able to easily adapt to new requirements when those arise.

Often I’ve been looking at the problem from only a “it’s this tree that’s important” perspective, and forgetting that my particular thing is part of a wider system (a forest). I hadn’t been considering future maintenance, which often led someone in the future (often a future-me) to tear it down and rebuild something new.

That’s technical debt.

When some structure isn’t maintainable, it’s debt. It’s a burden for the team in the future. Designing things quickly and building them within a deadline, while making them maintainable requires some knowledge, experience, and also discipline to work with the patterns, and avoiding the anti-patterns, that make code difficult. The same thing applies to managing systems. Custom jobs on every server and separate configurations make life hard. At the same time, a one-size-for-everyone approach also isn’t maintainable. We need a balance of well-written solutions that solve our problems, but are easily maintainable.

Part of becoming a better engineer or admin is learning how to build maintainable things that others will continue to use for a long time, not because they have to, but because they want to because the code works well.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

Starbucks has too many order combinations. This article about the challenges for their new CEO notes there are over 170,000 combinations of things customers can order. As someone who has visited a store often and stood behind someone placing an order, that sounds like an accurate number. Sometimes I hear people order things to drink that I had no idea were even possible.

When there are too many choices, sometimes there can be a delay as the person tries to process the information and make a decision. Some people can’t make a decision. Others enjoy the variety and keep making different decisions each time. That’s fine if you’re visiting a restaurant, but it’s less useful if you’re building software.

I have worked with people who struggle to decide on an approach to solving a problem. In the C++ world, I’d see people struggling to organize classes and members, sometimes spending too much time researching and comparing different methods. They debate when is private or public appropriate, when do we need a superclass or subclass? We would see some paralysis by analysis from certain developers who are endlessly searching for the optimal approach.

Other developers might change their approach constantly, resulting in a lack of consistency across the codebase. In the modern world, I sometimes see users experimenting with different libraries or packages (or data stores) with each new project. They might be consistent within a project for a period of time, but since many teams often shift to different projects over time (or maintain them), how helpful is this? If each new project looks different, we spend a lot of time on developers getting familiar with the technology.

I have tended to adopt the philosophy of strong opinions, loosely held. In other words, I’ll pick an approach/platform/technology/etc. and stick with it. Moreover, I’ll ask everyone on the team to stick to a similar method of coding. If we need to change, then we can evaluate the proposed change, performance test it, and if we decide it’s better, we all adopt this and slowly work to refactor old code as we have time. We never have time, so we only refactor when we have to touch old code for some fix or enhancement. However, when we do touch old code, let’s improve it.

My preference is to allow a limited set of technologies in our company at any point in time. We need a process to add or remove something, but this ought to be a decision made to limit our range of technologies and required expertise. I’m not against Python or PowerShell or MongoDB or any other technology, but I don’t want things adopted because an individual prefers that technology or thinks it works better. They need to prove that there is a good reason to others, and as a group, we agree that the value of adding a new technology is worth the burden.

Otherwise, we end up in the situation I see often in job descriptions: with a requirement that new hires know dozens of technologies at an expert level because we don’t have that person on staff.

And we’ll never find them.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.

I was listening to someone at Microsoft talk about their product recently. I can’t disclose which one it was, but lots of people use this product and are impacted by changes. The particular thing that caught my attention was that the presenter noted there was a breaking change in the new version for some people. This wasn’t a huge change or one that would affect a lot of people, but it was a breaking change for a few.

Another attendee asked this question: I would tend to do xxx instead of what you showed, so why would you do this?

It wasn’t an antagonistic question, but more curiosity. However, it was a good question since any changes that cause breakage can be disruptive to other people. The thing I noticed was that the questioner’s frame of reference was completely different than the presenter’s. The questioner couldn’t imagine writing code in this way.

As I work with more and more customers, I find that many of them get tunnel vision in that they approach their work or code in one of a few ways and don’t think widely about other approaches. In general, I think that’s a good way to work as we ought to use patterns and avoid anti-patterns in our work. The more we all code in a similar structure and style with the same patterns, the easier it is for any person to maintain the codebase (or infrastructure).

Of course, we have to be willing to add new patterns when we find them and drop older patterns that no longer work (and perhaps refactor the code). That’s another hard thing for humans to do. We rarely want to let go of patterns and who has time for refactoring?

At Microsoft, they have to consider lots of different ways to look at code. I saw awhile back one of them writing that building Windows was like ordering pizza for a billion people. You’ll never get it perfect and someone will always be upset. I get that. I couldn’t imagine making changes to something like Windows, where every little thing upsets some group of people and sometimes those are a very loud minority.

Many of us approach the world in similar ways, and we can appreciate or understand small differences. When we meet someone who sees things very differently, then it becomes hard to reconcile their view with our own. In code, this often means we want to use different patterns than others, which creates a less maintainable codebase.

Communication continues to be one of the hardest parts of building software (and most endeavors). Part of our communication should be to try and understand others’ points of view, get them to understand ours, and come to a shared understanding of how to approach problems. That shared understanding is what helps us build better teams.

Steve Jones

Listen to the podcast at Libsyn, Spotify, or iTunes.

Note, podcasts are only available for a limited time online.