Internet pipes are down
September 12, 2005 2:07 PM   Subscribe

Is there a major Internet backbone down somewhere? And how can we check it? I notice Dreamhost is inaccessible. Doing a tracert via my ISP, and using the tools at all-nettools.com, tracert.com, etc, neither can even get onto a backbone. Is a chunk of the nation out right now?
posted by rolypolyman to Computers & Internet (29 answers total)
 
I believe Dreamhost is in Los Angeles.
posted by jjg at 2:08 PM on September 12, 2005


Holy cow, you're right... though I'm not sure why this would prevent a hop onto a backbone. A tracert I tried from a Russian site could not even get outside a *.ru domain before the packets failed.
posted by rolypolyman at 2:11 PM on September 12, 2005


DreamHost Emergency Status
posted by jackmakrl at 2:18 PM on September 12, 2005


http://www.boingboing.net/2005/09/12/la_dude_wheres_my_gr.html
posted by WCityMike at 2:19 PM on September 12, 2005


In the late nineties there used to be a great website that had the live status of all the major backbones as colored lines on a map. I can never find it any more :-( Does anyone else remember this?
posted by evariste at 2:25 PM on September 12, 2005


http://www.internettrafficreport.com/main.htm

That may help you.
posted by a3matrix at 2:41 PM on September 12, 2005


Of course it should have been a link.

Sorry
posted by a3matrix at 2:42 PM on September 12, 2005


The WebHostingTalk Outages Forum is where I usually end up getting my info.
posted by trevyn at 2:43 PM on September 12, 2005


I do, evariste, but I no longer have the link--the only thing even remotely close is the SANS Internet Storm Center, which focuses on security (e.g. "HOLY SHIAT LOTS OF SASSER TRAFFIC OMG"), not general network connectivity status.
posted by cyrusdogstar at 2:43 PM on September 12, 2005


a3matrix: That's linked from SANS, but it's not what I remember so it may not be what evariste remembers either. It was a, say, 7x7 or larger grid, with the major backbones along both axes and the grid squares being coloured according to that path's connectivity.

E.g. the box which had UUNET on the left axis and, I dunno, Sprint or whatever-the-heck on the top, would be green or yellow or red, and that indicated the UUNET -> Sprint traffic status. So if you saw a bunch of red squares for a specific backbone's row/column, that was a good indicator that it was down somewhere.
posted by cyrusdogstar at 2:50 PM on September 12, 2005


Cyrusdogstar, you're talking about the Internet Health Report, I think.
posted by donnagirl at 2:56 PM on September 12, 2005 [1 favorite]


That sure looks like it (or a somewhat visually updated version of same =). Thanks! *bookmarks*

Wonder why it's not showing the LA outage--no major backbones go through LA?? (I'm clueless as to the layout of our large-scale network infrastructure, where everything is, etc)
posted by cyrusdogstar at 3:00 PM on September 12, 2005


Well, the internet traffic report is saying that most of asia is unreachable at this point, so ... *shrug*

According to Yahoo News, electrical utility workers connected the wrong wires together, and that's why the outage happened. It was apparently preceded by a rather large surge, which might be why we're seeing backup power failing... that surge might've damaged a lot of equipment or blown a lot of fuses.
posted by SpecialK at 3:11 PM on September 12, 2005


donnagirl-that was the one! a3matrix-I like yours too. Bookmarking both.
posted by evariste at 3:31 PM on September 12, 2005


Wow. I started having some problems with my dreamhost a couple of hours ago, and, I was wondering if someone had asked this question on Ask.mefi and low and behold here it is.
posted by trbrts at 3:34 PM on September 12, 2005


thanks jackmakl! (again, DreamHost Emergency Status)
posted by Satapher at 3:38 PM on September 12, 2005


Thanks, i was wondering why my site was down...
posted by ig at 3:46 PM on September 12, 2005


I really need to switch to a better web host. Dreamhost is really making me look bad at work.
posted by letitrain at 4:45 PM on September 12, 2005


I really need to switch to a better web host. Dreamhost is really making me look bad at work.
I think you'll look far and wide and not find a better web host than Dreamhost. They can't help it if the entire city loses electricity!
posted by Independent Scholarship at 5:42 PM on September 12, 2005


Independent Scholarship, DirectNIC is located in central New Orleans and has remained up through the hurricane and all of its aftermath. They've been up for two weeks of running on a backup generator and never went down. Granted, they were lucky that the companies they peered with for Internet access didn't go down. From the Dreamhost status page, when they lost power all of their servers went down. In any case, my point is that you can keep things up even in the middle of a catastrophe.
posted by zsazsa at 5:56 PM on September 12, 2005


But is the power still out in L.A.? My Dreamhost sites are still down after 6 hours!

Power had returned to most areas by 2:30 p.m., according to department spokesman Ron Deaton. - CNN
posted by rolypolyman at 6:00 PM on September 12, 2005


If Dreamhost didn't have UPS' or other temporary backup power, which it seems they didn't, all their machines would have just shut down hard when the power failed. Sometimes that has dire effects, and even when it doesn't, the admins need to be very careful when bringing everything back up.

Either way it's not usually just as easy as "oh, power's back! Back to business!". Although, again, DirectNIC is a shining example of how to plan for power loss...makes you wonder why Dreamhost was apparently *less* prepared than normal. Most places would at LEAST have a couple hours' worth of backup power so they can cleanly shut down.
posted by cyrusdogstar at 6:35 PM on September 12, 2005


From the Dreamhost blog:
Okay, this sucked. Literally 10 seconds after Nate posted that last entry [about the UPS and generators working], we received an announcement over our loud speakers that our building was being evacuated and we had to leave immediately! (...)

Unfortunately, shortly thereafter we figured out (thanks to EVDO!) our sites were no longer accessible, as our three network providers Level 3, Global Crossing, and Mzima were all down. Our physical servers were okay, and probably really happy sitting in the data center (we were no longer able to access) with nothing to do.

But that didn’t last long. Shortly thereafter the entire building where our data center is located’s back-up generators (there are SUPPOSED to be four) stopped working, and all power was gone.
posted by statolith at 7:26 PM on September 12, 2005


I was wondering whether this was handled poorly, if it was just beyond their control. After reading that blog post, I can't say this is affecting my opinion of them at all. It's great that DirectNIC has managed to stay up, but is it really because they were better prepared, or was a lot of it just luck? It sounds to me like DreamHost was pretty well prepared for this, something just went wrong. Several things.

This comment explains why the generators failed. So long as they look into it and take steps to ensure they won't all fail again, I'll be satisfied. They were in the process of adding another generator... That's just bad luck.

If I was paying top dollar, I would be pissed. If it was a much smaller power outage, and it was still handled poorly, I would be pissed. But I'm getting a damn good deal, and for such a huge problem it looks to me like they've handled it pretty well.
posted by robotspacer at 9:37 PM on September 12, 2005


I'm not a Dreamhost customer, but I find it odd that they didn't plan for adequate power backup before this kind of thing happened. This comment from one of their customers is pertinent.

(Oops, this is Ask MeFi, so perhaps a discussion about this kind of thing is not appropriate, eh?)
posted by madman at 10:47 PM on September 12, 2005


DirectNIC stayed up because they were better prepared. That's the long and short of it.

My understanding of DreamHost's situation is that they don't have enough control over their building where they can still stay in it if the rest of the city's been evacuated. They also don't directly control any battery backup or power generating capability, even if their building has generation ability... and the building's generating ability is apparently not enough.

The hosting peers should also be culpable. Don't these people test this stuff? How did THEY go down? I'm hosted with two businesses in Portland that test their generators at least once a month ... as in, they swap over to emergency power completely, shut off their grid power mains, for a few minutes to make sure the UPSes don't blow up and the generator kicks in and steadies out like it's supposed to.

The reason it's taking some time to get things back up is that computer equipment, when run consistently for months and years, doesn't take being shut down all that well. At all. Hard drives will freeze up as their bearings cool, circuit boards that were running hot will warp as they cool, etc. So you have to be careful when you're starting a machine up to check for these things... if something's bad, it could cause you to lose the whole machine and all the data on it.
posted by SpecialK at 11:01 PM on September 12, 2005


where's FEMA when you really need em?
posted by Satapher at 12:46 AM on September 13, 2005


My sites were back up shortly after 11 PM EDT/8 PM PDT; DreamHost reported "all systems go" at about midnight PDT.
posted by mcwetboy at 6:50 AM on September 13, 2005


I guess when you can run your servers off of Sprint in Reston, VA (as DirectNic can), I suppose you can stay up when a hurricane hits New Orleans. Dreamhost is direct to backbone in Los Angeles. A bit of a difference.
posted by Independent Scholarship at 2:41 PM on September 13, 2005


« Older Is there a free property ownership search engine...   |   Where do mechanical/design engineers hang out... Newer »
This thread is closed to new comments.