Earlier this year I was watching someone present on DevOps and IaaC (Infrastructure as Code). The speaker was showing how they had worked with clients to implement tests and checks that evaluated whether their systems were deployed and the code was working as expected. They did this as the CI/QA environments weren’t maintained, but rather spun up as needed when code was committed or tested.
What I found fascinating was the part in the talk where the speaker said they were “shifting right.”
That had me sitting up and paying closer attention. After all, for ten years I’ve been working in DevOps where we try to shift everything left. Shift testing to the developers, shift infrastructure into the repository, shift pen testing into some automated CI stage. We want to know about issues sooner. What is this shift right stuff?
It turns out that the speaker was using this as a way of smoke testing. They didn’t completely trust that CI/QA were configured the same as production. They were performing many of the same tests in production after a deployment. In a few instances, they’d spin up a copy of production, deploy changes, and then run tests against that. They wanted to get a real-life view of how the system worked.
For some parts of the infrastructure, this seemed like overkill. After all, if I’m updating an app and perhaps adding a web server to check, verifying this was the same size as a previously deployed web server seems silly. However, they included other checks like security. Was the new, or even old, server configured with the same security that was required by the org? They had policies around various authentication and authorization items and these were constantly rechecked. Someone alter a file share or enable anonymous access? If this was against policy, someone was notified when the next deployment caught this.
What was interesting about this methodology was that deployments weren’t often failed. Instead, when something failed, there was a report sent to a group and the incident was logged, but the deployment still occurred. That’s an approach that I actually like. Don’t stop things, but make sure someone follows up. Of course, like many nagging problems, if this happens constantly or it can’t get fixed, then this isn’t helpful information.
I haven’t thought much about the idea of shifting things right, but I have advocated for the same ideas. Create smoke tests that evaluate if things still work after deployment. Get feedback to people right away. Much like a unit test, I’d want to know if simple things are broken. I’d also approach these tests the same way. Don’t write a lot to start, but every time there is an issue, add a unit test and a smoke test to prevent regressions.
And be careful of which frameworks you use. Some database unit testing frameworks alter tables, which can’t happen in production.
Steve Jones
Listen to the podcast at Libsyn, Stitcher, Spotify, or iTunes.
