If SanFran slides into the Pacific, which apps will stop functioning?

June 8, 2015 11:27 AM

With several major (many users) social media service providers being headquartered in, or near to, San Francisco in California, what would be the effect on these services of a catastrophe e.g. earthquake, tsunami, Godzilla etc laying waste to the city? Would these services still be able to function without a HQ, senior personnel and owners, and with all of the disruption caused by such a catastrophe to their HQ host city?

Services such as Facebook and Twitter and Instagram are based in the city, but some have big data centers elsewhere. Would this mean that they could still function as now, even without their HQ and key personnel? Are there links to serious analysis of this scenario and/or "Catastrophe continuation of service" plans?

n.b. the title, and post question, are in no way an ignorance, belittling or downplaying of the large-scale loss of life of such a catastrophe.

posted by Wordshore to Computers & Internet (6 answers total)

Many large organizations have indeed planned for these sorts of things. Flu pandemic planning was a recent variation - how to continue operations if, say, most or all of your staff was stuck at home. It might not be the staff loss that impacts the business as much as a catastrophic interruption of financial operations.

The IT part of business continuity and disaster recovery generally consumes a large amount of a company's budget, and orgs in disaster-prone areas (think Florida during hurricane season) already have contingency plans in place.

Presumably there would be people in a defined succession plan, some of whom might be in APAC or EMEA. If so, they'd step into things and (hopefully) keep things moving. For example, people in the board of directors or corporate officers located elsewhere.
posted by jquinby at 11:38 AM on June 8, 2015

Disaster recovery varies a lot per system, and we never really know how well we've prepared until everything goes to hell. Having said that, data centers go down for a variety of reasons that have nothing to do with power issues, a big one in Fremont recently had power issues and lost a switch for a while (caused problems with a surprising number of friends' personal sites), so although there'd be some hiccups, if you're a big app company these days you've pretty much lost a geographic class of servers and had to recover from that.

Also: XKCD wasn't kidding about how seriously sysadmins take uptime.
posted by straw at 11:38 AM on June 8, 2015

It's really hard to say without being in a key position at one of those companies; "Serious analysis of this scenario" would require very detailed knowledge of how each company's services are engineered and whether their employees outside of the Bay Area are knowledgeable enough to keep the services running.
posted by ripley_ at 1:27 PM on June 8, 2015

The servers are housed in multiple data centers all over the globe. Any self respecting company will have duplicates of their servers. The offices downtown are for development and management. Here is a link about the Facebook infrastructure.
posted by Mac-Expert at 1:36 PM on June 8, 2015 [2 favorites]

The servers are almost uniformly located off-site, and frequency of natural disasters is one of the considerations when siting data centers. However, the people monitoring them ("DevOps" is the current term) are often on-site. So the machines themselves could be humming along just fine, but the applications running on top of them could stop working. But that's for small-scale companies. Larger ones tend to have 24-hour teams, often located in geographically separated zones. The Internet itself would be slow and glitchy but would still work.

Make no mistake, anyone who hasn't done some serious professional planning (i.e., more than just one person whose job is disaster recovery) would be offline within days, probably closer to hours or minutes.
posted by wnissen at 1:52 PM on June 8, 2015 [2 favorites]

It's getting easier these days to set up geographically diverse remote data centers (amazon, azure, other cloud providers). I wouldn't be surprised if a lot of SF-based companies had no customer-facing servers in California at all.

The underlying network would route around things as best it could, but if you take out a couple of big peering locations you could really savage overall quality of connection.
posted by rmd1023 at 3:33 PM on June 9, 2015