Finding LiveJournal SysAdmins

January 15, 2005 11:42 AM

Getting a message to a sysadmin staff currently in hell? [MI]

As you know, Bob-Mefites, LiveJournal's been down. They're slowing coming back up, but me and a friend have this thought that when they do come up, they're going to get crushed by a few million LJ addicts trying to get a fix at once.

We kicked around ideas, and came up with a useful soultion (TWA used it once when they restarted during a storm) -- throttle the incoming bandwidth hard, and gradually dethrottle it as your confidence in the site grows.

The other idea was restart at 4AM CST, so there are few addicts ready to pounce.

Of course, telling them this is hard (and they may reject it out of hand for other reasons.)

Anyone have any ideas how to get a message through, esp. since they're no doubt inundated with "WHERE'S LJ" cries?

posted by eriko to computers & internet (14 comments total)

If their SAs don't know how do do the stuff you're talking about, they should probably be canned as incompetent. I'm not sure it's worth the effort of doing something innovative like sending a telegram to the network operations center, just to tell them how to do their jobs correctly.
posted by majick at 11:51 AM on January 15, 2005

If the sysadmins are in currently struggling through bringing their systems back, they'll probably want to stick with their current plan, which presumably they've considered carefully, rather than switch to a plan that may have unknown side effects.

I'd suggest waiting until the current crisis is over, then presenting with your proposed solution for the next time this happens.
posted by SPrintF at 11:52 AM on January 15, 2005

On preview, what majick said.

Don't go poking the angry monkies with sticks.
posted by falconred at 11:56 AM on January 15, 2005

Shouldn't a site with 100+ servers be able to drop the clutch and slam right into second gear?
posted by sled at 12:00 PM on January 15, 2005

Agreed with SPrintF. Erico, the problem they're having isn't that they didn't have a Plan B, it's that both their Plan A and B failed in some way due to events they couldn't have reasonably forseen -- and they're keeping up to date with the users in a way that sure makes it seem like they know what they're doing. If they seemed to be drowning, that'd be one thing, but when they have a web site up saying it's all under control but may take a little time, that's completely another. Cut 'em teh slack, as they say.
posted by eschatfische at 12:30 PM on January 15, 2005

I miss Live Journal. I miss it a lot. I'm feeling things, and there's no one to tell except real people.

I keep on checking. hoping. If an anonymous Ask Metafilter appears, querying the masses on what to do if you're bored and emotional in Portland during an Ice Storm and LiveJournal is down, please be gentle. It's just little traumatized me.

That said, I have faith in the good Live Journal people and their capabilities, but I think it's great you want to help.
posted by redsparkler at 1:31 PM on January 15, 2005

I miss it too, redsparkler- I don't keep a LJ, but I follow several fabulous LJ communities, and this whole power loss has produced a large hole in my life previously filled by college business and celebrity gossip.
posted by ThePinkSuperhero at 2:07 PM on January 15, 2005 [1 favorite has favorites]

Man, if I were an admin at a site like LiveJournal, and some random user I didn't know wrote in with a helpful idea for me on how to do my job, I don't think I'd take it very well. Best to keep your well intentioned mouth shut.

If you're into this sort of thing, though, read up on exponential backoff. It's not a new idea, by any means, but it's a very effective one.
posted by Nelson at 2:37 PM on January 15, 2005

I meant to say "if I were an admin at a site like LiveJournal with a high-stress problem like they've been having". I'm sure I'd be all nice and sweet other times.
posted by Nelson at 2:48 PM on January 15, 2005

Yes, I agree with a lot of the suggestions to leave them alone. I also think that they have a competent and innovative staff, used to handling all kinds of awesome problems with innovative solutions, so they probably have some super awesome way to deal with this, waiting in the wings, and their current difficulties have been due to extraordinary situations.

(In addition: hi redsparkler! I'm also a Portlander, housebound because of ice, finding it funny that Livejournal is down during this time. I'm glad someone else is using Metafilter to ameliorate the pain.)
posted by thethirdman at 3:46 PM on January 15, 2005

Hey, red sparkler, you can always just sit and watch the TV crews having a winter weather stormgasm...
(Yeah, I'm trapped in Lake Oswego near PCC, because it hasn't gotten above 25 all day here and there's a half inch of ice on everything...)
posted by SpecialK at 4:40 PM on January 15, 2005

LiveJournal is, IIRC, one of BigIP's big test sites. Their load balancers are running otherwise-unreleased code, so that sort of flood is presumably exactly the sort of thing that they're expecting to handle.

That sort of thrashing is self-throttling anyhow; if some of the rush can't get the resources to get fast responses, then the rush will attenuate, and so on. Even moreso since there's really nothing to read from the last few days -- people couldn't catch up on their friends' posts for a day, but their friends couldn't post anyhow.
posted by mendel at 5:32 PM on January 15, 2005

"Last few days"? Ok, maybe i'm jonesing too. Last day.
posted by mendel at 5:33 PM on January 15, 2005

The LJ guys are on top of it and (finally!) getting some much-deserved rest. If you really need to get something to them, feel free to ping me at anil at sixapart dot com and I can pass it on to them. But, to be honest, if it's just "here's an idea I had of how to bring the servers back", I'm quite sure they've got it covered.
posted by anildash at 7:03 PM on January 16, 2005