Voice of the DBA

Tag: software development

  • Mission Critical Deployments

    Talk about a lot of pressure to get a software deployment correct. A software install on an Airbus airplane resulted in a file containing parameters being wiped. This error caused (apparently) an airplane to crash when three engines cut off in flight and four people were killed.

    I don’t know if this is the final report on this, but the fact this is a possibility concerns me as a technologist. Certainly we have probably had similar mechanical failures and installation issues in the past, but there are some scary issues here with regards to software. There was a faulty software installation (yikes!), a poor architecture (not assuming more then 2 engines would stop), too much tolerance for software errors (the review letting this pass), and poor overall design (no alerts on the ground).

    I can’t decide if I think that software makes issues like this more or less likely. Certainly checks of physical systems are skipped regularly by people. It’s far easier, and more reliable, to automate checks of software systems, especially with deployments, than it might be for complex mechanical changes. However, maybe that’s not true. Perhaps mechanics are more likely to notice a loose bolt than a misconfigured software menu. Or maybe we need a new type of mechanic that’s savvy with technology.

    Ultimately I think that any software that makes changes to systems, including through deployments, needs to have double checks by an independent process and clear alerting of any issue, not relying on someone to look for the success of a long series of steps. We also need to take review of potential software errors very seriously and ensure the tolerance for potential issues shrinks as the impact of those issues rises.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.2MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • How many environments?

    One of the things I’ve learned over time is that in order to develop software, you need multiple environments. I think that outside of school projects, I’ve always used at least two environments: development and production. At school there were times I worked on software that I also used, and would make changes on my live (and only) system. I had no shortage of regrets, and whenever I question using a Version Control System (VCS) or separate environments, I think back to those days.

    People I meet these days that develop software almost always have more than one environment, and most have at least 3. The most common model might be development, test, and production, but plenty of people work with even more. Maybe not as many as the seven Grant once had to deal with, but I have worked in environments where we had six database environments for one of our software pipelines.

    How many different environments do you work with?

    The poll this week asks for the count in your organization. I realize that you probably have multiple applications and test environments, but choose one or more and give us some answers.

    The most I’ve worked with was in a financial services company. We had a development database and a separate QA system for testing. There was a staging system to mimic production, a Beta database for users to actually test against and production. That worked well, and we were able to easily refresh environments as needed without impacting our development efforts. Mostly because we have a strong DevOps type process.

    Let us know this week how you’ve got your systems set up and what each one is used for in your software pipeline.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.8MB) podcast or subscribe to the feed at iTunes and LibSyn.

  • Deploy from VCS

    Are you confident in your version control practices and code backups? If so, are you willing to do this? Wipe your applications and database schema from your servers and redeploy things from your Version Control System (VCS).

    I’ve typically maintained code in a VCS, but I have to admit I’d be worried about being positive about which version of code was deployed to which server in many of my previous jobs. Only in one position would I have been positive about which code was deployed to every system.

    I was teaching a class recently with Ike Ellis and he mentioned a consultant he knew always asked this. The reason was that if you can’t be confident of the state of your development code, you can’t really have confidence in your refactoring. There will always be places in the code that no one is willing to allow changes, and your flexibility will be limited.

    That’s very true, though even if you know what code you have, I’m sure there will still be sections of your code that no one wants to change. However, our goal should be to get away from code that’s untouchable.

    We should be able to change any code in the system if we have a good reason. When we can make improvements, we should. We need the confidence that we can alter code when we see the need. That means that we need a repeatable, reliable, adaptable software development process. That’s something we can’t build without making the conscious decision to do so.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.0MB) podcast or subscribe to the feed at iTunes and LibSyn. feed

  • Five Minute Refactoring

    I’ve listened to, and been a part of, no shortage of arguments about the technical merits of a particular programming solution. I’ve seen two (or more) developers argue about the way to solve a particular problem for hours, even days at times. The Internet is full of discussions and debates in forums on the “best” way to write code for all sorts of specific issues.

    These arguments are sometimes referred to as religious wars for the devotion and belief that each proponent has in their solution. I tend to try and avoid becoming embroiled in any sort of zealotry as I am always cognizant of the fact that I need to be effective. I need to get things done and can’t spend a lot of time arguing about a solution, whether for or against it. i need to get work completed.

    When I was managing people I tried to limit the devolution of my staff into a never ending argument, but I ran across a better solution recently. Al Shalloway wrote a blog about limiting discussions to five minutes. He notes that two level-headed people ought to be able to come to a consensus in five minutes. Beyond that he says it’s likely that one of them is arguing about covering possibilities rather than current realities.

    That’s interesting, and like many rules, I’m sure there are exceptions. A manager can make a decision to continue a discussion if they really think that it is warranted. However as a default, the idea that a team should choose the easiest solution to implement if they can’t decide makes sense. It’s the least amount of work that might be wasted, and we want to avoid wasting time and work when we can.

    We won’t always agree, and despite our best intentions, we will make mistakes and we will make poor decisions. Don’t let indecision continue indefinitely and don’t rashly make decisions, but don’t let the fear of mistakes prevent you from choosing a path. Be decisive and move forward, adjusting as you learn more that helps you refine your solution.

    Steve Jones

    The Voice of the DBA Podcast

    Listen to the MP3 Audio ( 2.5MB) podcast or subscribe to the feed at iTunes and LibSyn.