Planning for an outage: Phone numbers

Who ya gonna call?

If there’s somethin’ strange in your neighborhood
Who ya gonna call?
If it’s something weird and it won’t look good
Who ya gonna call?

 

This summer, they’ve been taping the new Ghostbusters movie in Boston. It’s pretty cool to see the Ghostbusters car rolling through the city.

Unfortunately, the Ghostbusters aren’t the right person to call when something goes boom in your data center. I’m not saying I haven’t used ghosts and gremlins as part of a root cause analysis before–but even Dan Aykroyd and Bill Murray can’t help you when there’s a storage or network issue. I’m skeptical that Kristen Wiig will be much better.

So, who do you call when there’s a problem?

At almost every company I’ve ever worked at there’s a contact list for those of us in charge of keeping the lights on. Sometimes there’s a document on a network share, sometimes a page on the wiki, or a directory on SharePoint. That’s nice when you need someone’s number, but I won’t depend on it in an emergency. In fact, you’re a fool if you depend on that list.

Harsh words, I know. If there’s a SAN failure, that document on the network share is probably unavailable. What happens if a database problem causes the wiki to go offline? And SharePoint isn’t exactly known for being problem-free. When VPN goes down, you won’t have a prayer of finding the network guy’s number to call him for help.

Even if all those systems are up, how long will it take you to open the directory to find Storage Guy’s cell number? In an emergency, that’s precious time. If your organization has a four-nines uptime goal, and you spend four minutes trying to find a phone number, you just burned through an entire month’s worth of downtime.

Simple problems have simple answers

Program everyone’s phone numbers into your phone. My first week at my job, I sat down and made a list of who seemed like a key player, and I put their numbers into Google contact list. Those sync down to my phone, and now I have everyone’s number. Every once in a while, I go back and make sure it’s accurate–update phone numbers, add new people, etc.
This seems like a silly blog post, but it’s not. “I didn’t have your number” or “Does anyone know Joe’s number?” are things I hear all the time. Having those phone numbers is a job responsibility. You don’t have to save them to your phone’s contact list. That works best for me, but maybe you have a different solution that works best for you. As long as you have a solution, that’s what’s important.
It’s seriously simple, but seriously important.