A Buggy Release

I definitely believe in a DevOps process, though a thoughtful, incremental one. I think this is the best way to develop software, whether you release every day or every year. Yes, you can implement DevOps and release once a year. You just end up tracking, testing, communicating, and being ready for that once a year release. Of course, I bet you don’t release once a year since I’m sure you’ll patch the system at least once.

One of the core principles of DevOps is to use automation where you can. Remove humans and ensure that repeatability is possible for moving software from one machine to the other. Communicate, test, and then alter your process to work better. This requires the monitoring and input of humans to examine the process, but they shouldn’t be involved in deployments other than approving them. It’s too easy for an individual to make a mistake.

However, DevOps isn’t a panacea for building better software. Witness the issues at Knight Capital, where they went from having $364mm in assets to losing $460mm in 45 minutes. Mostly because of a problem deployment, where an engineer didn’t deploy code to all the servers in their farm. Certainly a clean deployment to every system might have prevented this, but the reuse of old flags in code is problematic, as is leaving old code around that could be executed.

In addition to moving to a DevOps mindset, I’d also say that you should be sure that you follow good software development practices as well. Clean out old code (including database code) and be very, very careful about reusing any part of your software, including flags, for a new purpose. It’s far, far too easy to make mistakes here.

Steve Jones

The Voice of the DBA Podcast

Listen to the MP3 Audio ( 3.5MB) podcast or subscribe to the feed at iTunes and Mevio .

About way0utwest

Editor, SQLServerCentral
This entry was posted in Editorial and tagged , , . Bookmark the permalink.

3 Responses to A Buggy Release

  1. Ilian Iliev says:

    Hi Steve, I agree with you that “DevOps isn’t a panacea for building better software.” but I think you are missing part of the point here. If they had an automated deployment that would make sure that all servers run the same code the issue with Knight will be avoided.

    This wouldn’t solve the potential danger of the old code, but still the issue will be avoided.

    • way0utwest says:

      Certainly a DevOps process here would have (we hope) alleviated the chance of not deploying to one server. That’s an issue. What I wanted to emphasize is there are other issues here. Old code can potentially be called, which may or may not occur here, but refactoring an application and ensuring all calls and paths to cold code are removed is tough. If you remove the old methods/procs/etc, then you’ll get errors, but not old code executing.

      I probably didn’t explain that well.

    • Ilian Iliev says:

      Yeah, that makes sense and I totally agree with you )

Comments are closed.