Tag: DevOps

When I watched Star Trek as a kid, I was amazed by the technology. Talking to the computer, the touch screens, the handheld communicators. We have most of those devices now, without the space travel. Hopefully that will start to change with all the efforts being made by various organizations.

One of the things that always bothered me was the chance for mistakes. A mis-spoken (or mis-heard) command to a computer that didn’t verify things as a human might. The chance to hit the wrong part of the screen as the starship moved. It seemed as though soft buttons would have allowed more mistakes than hard ones. Certainly humans make mistakes with physical switches, but I think I make more mistakes trying to hit a part of the screen in my Tesla than using one of the (few) buttons or wheels to change something. Interestingly enough, my 23 year old decided on a slightly older car because it had more physical buttons and fewer soft ones.

We are human. Frail and faulty. We make mistakes. Some are small (I ran a SELECT query on the wrong database), some are bigger. A mistaken copy paste error sent US$36 million away. That is the type of mistake that could happen to any of us, though hopefully not at this scale of financial loss.

This type of mistake is a main reason why I think DevOps and automated flows for development, testing, and even for production updates are a good idea. This doesn’t prevent human error, but it does serve to limit it and reduce silly mistakes. Often these types of errors are caught when we force someone to work through a bit of a process. An easy and quick process, but still a process.

In the case in the article, you would hope that someone looking to make changes would write an update that can be tested in a second environment before applying to the production blockchain. Perhaps with some idempotent wrapper and a pre-check that verifies the target. That might seem like overkill, but it’s the type of care that most of us take when we know we aren’t going to be the one executing the code. If you submitted a script to a DevOps process, you’d want to be sure the process running your code made the proper decision of whether to run the code or stop because of some error.

We won’t prevent all errors, but a lot of automation and light DevOps process is designed to limit simple, silly human errors because we are tired, distracted, or otherwise unfocused. I am a proponent of having humans design systems and processes, but then letting the computer handle the drudgery of following the process over and over on a regular basis.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.

I received a report at SSC of a missing toolbar in the forums. I checked, and sure enough, the toolbar for formatting code and text in responses was missing. I submitted a high-priority ticket, mostly because I hate unformatted code. The site was still usable, but people would be posting stuff that just didn’t look good. The developers got to work and had a fix, but it still wasn’t right. The wrong toolbar was showing. This was a generic toolbar, not the one that we have had for years (this one). They produced a second fix that contained the right toolbar.

The developers are in and out of the SQL Server Central codebase, working for a few days on open issues, then busy with other work for a week or two. They work in a DevOps style environment, with changes run through a Pull request/peer-review process, then CI, then an automated deployment. They often can make a change and deploy it in hours, or even minutes.

Is this a bad process?

It would be easy to say that the developers were moving too fast in their deployments and not testing things. However, the issue in production wasn’t easily reproduced locally, and we found a few places where local dev environments don’t quite match live. That’s been an ongoing problem for as long as we’ve been developing software outside of production. The DevOps solution here is to adjust the environment setup in code so that dev does match production closer. I’d say this is my solution, but likely we’ll find something else in the future that isn’t the same and we will continue to need to adjust something to get the environments in sync.

What’s the alternative? We could go slower and batch up a bunch of changes, testing them all, moving at a more waterfall-ish cadence. Would we catch this? Perhaps. However, we might not, and if we are batching changes, then bugs live for a longer time in production until we perform a new release.

Or we could build a quick patch, in one of two ways. One is by moving quickly, at an agile/DevOps speed, to quickly build a patch. That’s a hacky, not-thought-through flow that many people use to fix a deployment, usually without the process and bounds that a good DevOps process provides.

The other option is to bundle a fix in the next release, which could be next week? Next month? Who knows. Slow processes don’t necessarily make the final code better and can prevent rapid fixes.

Ultimately my feedback from here to the project manager is that we need a test, a process, or something that checks for these issues before release. While I know something else could break, I prefer these small deployments taking place often, with bumpers to limit the regressions we know about.

Steve Jones

Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.