How do you communicate Enterprise IT outages?
June 26, 2012 12:56 PM   Subscribe

How do you communicate your internal IT outages?

We're responsible for communicating Enterprise outages to all internal IT staff globally. Right now, all we do is send an email template with the description and status updates and some text-based checkboxes for high-level categorization. This is a pain to report on and has me hunting for a new solution. We'd prefer something out-of-the-box that we can use, but I'm not averse to coding my own solution if that's the only option left.

What are your best practices? What tools are you using to communicate outages?
posted by multiphrenic to Technology (7 answers total)
I think I'm in a much smaller organization than yours, but we do what you describe. When things are broke we notify affected people. In the event that the outage affects email systems or utility power we have alternate phone numbers for off site staff. I'm open to hear of other approaches but it doesn't sound like you have a bad process.
posted by dgran at 1:12 PM on June 26, 2012

Well first of all, using email to communicate IT outages can be problematic if the outage can impact access to email. You should *always* have a backup channel. At my old job, our NOC had access to an SMS gateway that eas entirely independent of the network - as long as they could get a dial tone on a POTS line ( which was itself not dependent on the PBX), they could get the word out via text message.
posted by deadmessenger at 1:14 PM on June 26, 2012 [1 favorite]

Part of the problem with IT outages is that if you're experiencing one, it's hard to reach folks with updates because, you know, they're down.

If you use IM or something like Chatter, you can post it on the "Social Network" so word gets around.

An outgoing message on the IT Help Line is super-useful, "The Internet is down in the Boston Office and it affects e-mail, Intranet and access to all systems, estimated time of repair is 2:00PM Eastern." Something like that.

Another option would be to have a separate website that folks could reach with a Smartphone, where outages can be listed.

Those are some thoughts, from the top of my head.
posted by Ruthless Bunny at 1:15 PM on June 26, 2012

At my old IT job, we used pagers that sat outside our network and would go around a potentially downed email server. This turned into SMS messages sent to our cell phones.
posted by Blazecock Pileon at 1:41 PM on June 26, 2012 [1 favorite]

Every place I've worked at has used email to notify most outages, but then it brings up the obvious question of how do you notify people about an email outage?

The best method (IMO) is to have a prearranged telephone number -- independent of your PBX and certainly of your VOIP system, a true self-powered landline -- with phone-company supplied voicemail on it. In the case of an outage, you use the landline to set the message on the voicemail. Then you distribute this number to everyone ahead of time (stickers work well!).

People call the number, get the voicemail message, and know "d'oh, email's out, IT is working on it."
posted by Kadin2048 at 1:48 PM on June 26, 2012

It's been awhile since I worked in IT, but approaches other than text messaging might be a simple phone tree (if the person you're supposed to call doesn't pick up, call the people they're supposed to call, and so on), and possibly custom error pages. There's also an "alert now" system my school district uses that auto-dials and plays recorded messages to EVERYONE in the event of closures; maybe there's something like that you could use.
posted by alphanerd at 1:58 PM on June 26, 2012

We use mailing lists, and twitter at the current job. The previous job set up a blog to let staff and faculty know about unplanned outages.

And if you really want to know, the relevant people are usually in an IRC channel.
posted by pwnguin at 8:23 PM on June 26, 2012

« Older Attend Health Care Ruling   |   Agile & Scrum, SOAP, Huh? Newer »
This thread is closed to new comments.