Upgrades are Hard

I think the people that run StackOverflow are pretty sharp. They’ve built a well performing site, they’ve worked on some useful open source projects, and they think through their projects. They still have issues with upgrades.

Taryn Pratt, from Stack Exchange, has written a few nice posts about her experiences upgrading the Stack Overflow databases. The first one was last year, where their multiple server AG environment upgraded from SQL Server 2012 to SQL Server 2017. Just recently she published another post on their Windows Server upgrade, from 2012 to 2016.

If you’ve never done a complex upgrade, these posts are worth reading. Don’t second guess Taryn, but rather, just read as if you are following along. You have hindsight now, but in the middle of planning this, you will learn about things that might cause you issues. In the Windows upgrade, one of the interesting issues is a VM vs. physical machine issue with drivers. To me, this might be one reason to never bother with anything other than a VM, even if it’s the only one on the machine. A lightweight version of Hyper-V or Xen doesn’t eat much in the way of resources, but can provide some separation from these issues.

One other thing to note is that you really need your runbook. I constantly see people asking for a checklist for how to upgrade, and there are good general steps to follow, but your environment likely needs to have a custom runbook that covers your situation. The Stack environment is complex, but even I was surprised with 35 pages of steps and notes.

As with most plans, this one had issues when it was finally implemented. I think Mike Tyson sums it up nicely, which is why you practice your move. It’s also why we can’t necessarily upgrade every year. There’s a reason many companies still have old versions in production (Thanks, Brent Ozar).

Plan, practice, test, repeat again, and then be prepared to think on your feet. That’s if you upgrade your systems. It’s time consuming and expensive, and I can see why a lot of companies have looked at cloud services, like Managed Instances. Reducing the time and cost to change your OS and/or SQL version is something we should all be thinking about.

Steve Jones

Listen to the podcast at Libsyn, Stitcher or iTunes.

About way0utwest

Editor, SQLServerCentral
This entry was posted in Editorial and tagged . Bookmark the permalink.

2 Responses to Upgrades are Hard

  1. SQL-Matt says:

    The major problem we have is that the business here is heavily siloed and the projects to upgrade instances to newer versions or to the cloud are done by people who hand over the support tasks to us and then wash their hands of it.
    They also choose to make the first application to be transferred to the cloud to be the one that is more mission-critical than any other and they are making changes in the production implementation on the fly as they find issues in test.
    I’m not kidding.
    Add to that, there is no communication between the teams (we find out many implementations by accident or when they land on us) and we’re often told that the consultation process is over when issues are raised.
    Responses from management equate to most of these as being a “Nike Job”…. “Just do it!”
    If it didn’t leave my team in a mess, I’d be inclined to leave and find a nice position somewhere else.


Comments are closed.