Ryan had a planes, trains, and automobiles situation a few weeks ago. On the same day, I didn’t. A delay for a few hours, but an easy trip home. Lots of other people didn’t have a smooth trip, and I had a few friends who spent an extra night somewhere or had flights canceled and decided not to take a trip. The Crowdstrike outage hit the entire world, causing problems everywhere. Just before that happened, Azure had an outage in the central region.
If you were affected by these, you have my sympathies. If your IT job included responding to these, you get all the virtual hugs from me (and Brent). I know what it’s like when someone calls you to handle an outage, and I know it can turn your life upside down. Hopefully, it hasn’t been too stressful to you.
In Brent’s post, he talks about reviewing your DR plans and presenting your findings to your boss. Good advice, and worth reviewing yearly as most of our environments will experience some amount of change. If you follow this advice, make sure you present this in a logical, dispassionate way. I see plenty of technology professionals who get upset when DR isn’t given enough priority. That doesn’t help the situation. Channel your inner Spock and present your results, allowing someone else to decide what to do and accepting their priorities. You can advocate for change, but keep in mind that there is often no shortage of work and limited time. If someone else decides you can accept the risk of poor plans, document that and move on.
Plenty of us will experience a disaster at some point. I’ve had more than a few occur at various positions, and I know many colleagues who have gone through a wide variety of failures. Even for companies that have extensive DR plans, there will be challenges when a disaster occurs. We’re certainly not at the point where any AIs can handle reading our DR document and implementing fixes. Instead, those of us responding will have a guide, maybe a great guide, for how to respond, but we will need to adapt our plans to the current situation. Having the mindset that our plans might not be perfect (as Mike says), will help you deal with any challenges that arise.
Aside from the technical situation, there are also mental challenges in a disaster. You will feel pressure from your employer, but also pressure at home. Your partner might not understand the extra hours. If you miss commitments made to your family, they will be disappointed, and I hope you are as well. You might have expected your schedule to include fun events, and you likely didn’t expect to get less sleep. Your diet might suffer in times of crisis and long hours.
All of these situations can be managed, but it helps to think about them when you’re calm and make plans. Warn friends and family. Think about how to limit unhealthy foods or situations, set expectations with yourself that help you manage stress, and most of all, treat yourself with kindness. Many of us want to help and support our co-workers but ensure that you (and them) get breaks. Imagine you’re in a Crowdstrike situation that lasts for multiple days. Talk about how that will look, even with just a few people and you’ll be better prepared when a disaster event occurs.
Hopefully, it won’t be a large-scale event, but better to prepare for that and experience a minor situation than the reverse.
Steve Jones
Listen to the podcast at Libsyn, Spotify, or iTunes.
Note, podcasts are only available for a limited time online.
