Dealing with Failure

This past week was a dichotomy for Coloradans. On Tuesday, it was 60F (15C) for most of the day, with my daughter and I going to volleyball practice in t shirts and shorts. Our phones chimed during practice, noting that schools had been cancelled for Wednesday, in anticipation of a winter storm. Plenty of people laughed about that at the time, and no shortage of parents were annoyed by the premature decision, especially with little wind and rain falling at 7am Wednesday.

That changed with high winds and some snow all over the area. Cars were stuck on roads, first responders overwhelmed, the airport closed, and people stuck in places between home and work. We hunkered down here at the ranch, and while horse care was hard, we survived. Power flickered for me, but I was able to work Wed and Thur without worry. I also have a generator, but we never lots power longer than the 30s it takes to kick in.

Glenn Berry wasn’t quite as lucky, though he came through the storm fine. He did lose power, and wrote about his experiences. Glenn is about 12 miles S of me, and in a rural area. I thought his thoughts on the experience were interesting, and not a lot different from the ones I had recently with another power outage. I don’t mind storing gasoline, and we have multiple generators here because of the horses, but I appreciate Glenn’s thoughts on finding a different way of storing power.

We both realize that disaster can strike and it’s important to think through the issues ahead of time. While I depend on my phone in the event of losing our Internet connection, I could easily lose that as well. The local towers probably run off batteries and in an extended outage, they’d lose power before I would. I have a tractor and can plow out our driveway, but in this case, the county roads were closed Wednesday and Thursday due to the poor conditions and number of cars stranded. Apparently Wed night visibility was inches and cars were being abandoned in the middle of roads. I’d prepared for a few days outage, but if this were still ongoing today, I’d start feeling the pinch from a lack of supplies.

It’s not often we encounter a major disaster at work, but it’s entirely possible that the secondary support plans we have in place won’t work. You might have a supplier for more diesel fuel, but if everyone else needs more, will you still get some? What about issues with a complex process at work? If it fails and you end up rebuilding some ETL flow, do you have the staff or are you expecting a “Brent” to help explain things and do some of the work? I think plenty of us have that dependency, one which we don’t think about until that person is on holiday and we need their expertise.

Disasters come in all sizes, but often they seem to come in the size that is at the limit (or just beyond) our preparations. I suggest that during some down times, take a minute and think about your preparations. Think what could go wrong. Do a little brainstorming and gaming of situations. If your plan has an issue, do you have another idea of what to do? It might be good to at least have thought about the potential issues.

Steve Jones


About way0utwest

Editor, SQLServerCentral
This entry was posted in Editorial and tagged . Bookmark the permalink.