Dealing with spikes in website popularity

April 3, 2007 11:02 AM

Web-hosting quandary: how to best prepare for the potential of a surge in high-bandwidth traffic?

I'm launching a website that may have an initial large spike of traffic. Storage space and number of visitors doesn't concern me so much as that I will have a few large, high-definition videos available for download, perhaps in the neighborhood of 100MB each or more, as well as some lower-resolution versions.

I currently use AT&T/Yahoo! Small Business as a web host, with a 200GB data transfer usage rate per month, and I pay about $12/month for this basic plan. To deal with spikes in traffic that exceed your allowance, they allow you to either automatically be charged an additional $5/5GB, or to automatically shut down your website if your allowance is exceeded. I've heard horror stories of people getting stuck with insanely high hosting bills when a site gets slashdotted or otherwise picked up by the blogospshere, and I don't want that. Nor do I want the site to become unavailable at the peak of its popularity.

Although I'm probably overrating the site's potential, I want to plan for the worst case scenario and say that the content on the website will be wildly popular and far exceed my current allowance. After the initial spike (maybe a month or two) I expect traffic to die down to almost nill.

Should I opt for the extra $5/5GB (seems kind of high considering I pay $12/200GB) or consider moving to another host altogether that perhaps caters to high bandwidth sites or offers a no-limit package? I know Media Temple is a popular host and have a $20/month plan that allows 1TB of data transfer, and I've heard Dreamhost is another popular option.

What say you?

posted by robbie01 to Computers & Internet (22 answers total)

Can you see the logs live or do you have to wait until the next day? This determines how much flexibility you have.

If you have to wait, choose the shut-off option.
posted by smackfu at 11:11 AM on April 3, 2007

If you are simply serving files dreamhost should be ok, but if there is anything dynamic about your pages (php or whatever), it will go down, as they cap the CPU you can use.
posted by cschneid at 11:15 AM on April 3, 2007

You could also fish around for some other hosts and mirror the site. That way, as each one falls, you've got another to go to.
posted by klangklangston at 11:24 AM on April 3, 2007

Response by poster: There's nothing fancy going on behind the scenes. Just a basic Flash-based website providing the interface, with the ability to download/view the large video files.

May be set up in a way similar to how Apple treats its movie trailers.

I don't think you get logs until the next day, which by then would be... too late! (and I don't want to sit and manually monitor the site all day, every day).
posted by robbie01 at 11:25 AM on April 3, 2007

Put it in a torrent.
posted by bigmusic at 11:26 AM on April 3, 2007 [1 favorite]

And have a youtube / google stream of it for casual vistors.
posted by bigmusic at 11:27 AM on April 3, 2007

Response by poster: There will definitely be a YouTube version for people who find the content using that method. The website will primarily be for people who want a higher-quality version of the video, perhaps after initially seeing it on YouTube.

Torrent is interesting, but I think a little too complex for the audience -- I'd consider only "hardcore" tech savvy people to have/use a torrent client regularly, and that crossover into my audience is probably pretty small.
posted by robbie01 at 11:35 AM on April 3, 2007

Dreamhost's business model seems to be based on the hope that people won't use everything they are promised. MediaTemple's model for their GridServer is that people want to be able to buy capacity at a reasonable price in manageable increments. In other words, they hope you'll use what you are promised, and then buy more, Based on that, I'd trust MediaTemple to do a reasonable job of actually delivering my downloads.

I don't know if I'd move the whole site, but I would definitely move the media to another provider. MediaTemple seems like a reasonable option. You could also look into Amazon S3, which prices transfer at ~$0.20/GB, which is cheaper than your current hoster, but will be more expensive than MediaTemple if you go over 100GB/month, which is only 1000 downloads of a 100MB movie.
posted by Good Brain at 11:42 AM on April 3, 2007

I doubt the bittorrent option is really worth while. I was involved in a project a couple years ago that ended up sparking over 100K downloads of large digital media files. Most of the downloaders found it via techie sites like Digg, and the torrent was higher quality. Even so, the # of direct downloads vastly outstripped the torrent. These days it's even cheaper to offer direct downloads, and if you offer people the streaming option via YouTube or the like, you'll probably have only a fraction downloading the full versions.
posted by Good Brain at 11:48 AM on April 3, 2007

Track down a good Dreamhost discount code and you can often sign up for next to nothing. For serving static files cheaply it's a pretty good option.

Or build your site to make changing the video file URLs quick and easy, and line up some volunteers with lots of spare bandwidth. If it looks likely you're going to exhaust your quota then you can set up a "poor man's Akamai" by sharing out the requests.
posted by malevolent at 11:49 AM on April 3, 2007

Host your videos on multiple ISPs. List them as mirrors. Give each ISP host the shutdown on cap option. I'd also add a torrent option. Good Brain's experience was a couple of years ago and torrent is constantly getting more popular.
posted by chairface at 11:59 AM on April 3, 2007

Speaking from experience, if your site is popular, you will go through 1TB like butter. I've had text+gfx only sites run 120+GB/week. These were sites that took maybe 10mb of hdd space total.
posted by jedrek at 12:00 PM on April 3, 2007

Response by poster: I'm not experienced with setting up sites as mirrors, so I guess I'd have to do some research to figure it out. But I'm not sure if the idea of paying a monthly hosting fees to several different providers is going to a practical solution. I also want the experience to be seamless for the users. If the the site has to switch hosts, I don't want them to be presented with an awkward "well this host died, here's a list of 5 others you can try."

As I said, the popularity of the content may only last a month or two, so I don't necessarily need an indefinite solution -- just something to get past the initial spike. Surely I can't be the only one facing this situation, but I've either never seen it handled elegantly, or it was done so well that I never noticed how they did it.
posted by robbie01 at 12:12 PM on April 3, 2007

For other hosting possibilities, you might want to look at Nearly Free Speech. They charge for storage and bandwidth based on actual usage -- there are no monthly fees other than charges for what you use. Money is deposited in advance (via Paypal, etc.) and can be refunded if you don't use it. You can also buy "bandwidth buckets" that insulate you against huge traffic surges.
posted by camcgee at 12:26 PM on April 3, 2007

Whatever else you do, have the site on one host and the videos on the other.

#1: if your media hosting provider arrangement temporarily tanks (over bandwidth cap, can't handle the load, whatever) you'll be able to update your site to say "Hey, sorry, movies are down temporarily" instead of simply disappearing from teh web entirely;

#2: if your media hosting provider permanently tanks (cancels you for TOS violations, decides it doesn't want you as a customer, turns out to be too expensive, you find a better host the day after you launch the site) you'll be able to update your links to the new host -- or both hosts! -- on a moment's notice;

Also, have highly-compressed low-bandwidth versions as well as the good stuff, and excerpts, and make sure your ISP lets you see real-time bandwidth usage. That way, if it looks like you're going to be in financial trouble, you can flip your site's links to the low-bandwidth versions (and move the high-bandwidth ones in case someone direct-linked 'em) and then to the excerpts if you get in real trouble.

Good luck; I hope you actually have the popularity you're planning for.
posted by davejay at 12:50 PM on April 3, 2007

Agree with davejay. Host your navigation on your current provider and each of your videos/downloads on one or more other accounts. Transparent (mostly) to your users and your eligible for the $12/200GB rate on each additional account if you went with AT&T. It's pretty easy to manage this to spread things around to even stuff out.
posted by Mitheral at 1:45 PM on April 3, 2007

100 meg files? The 200gig plan is a measly 2,000 downloads.

What happens when you start serving 10,000 or 50,000 files? Or more.

Considering youre not colocating and have a bargain basement host you should probably just put these things up as torrents (seed them yourself on a fast connection) or think seriously about moving them to youtube.

Even after 'the surge' if you get 50 downloads a day, thats 350 downloads a week which means on the 31st if you dont have a 350gig plan you're going to be shut out.

Hosting large media files is either expensive or really cheap (torrents). I dont think there's a middle ground for this kind thing that will only set you back 100 dollars or so.
posted by damn dirty ape at 2:37 PM on April 3, 2007

It has been mentioned already, but to reiterate: Amazon S3 is pretty much designed for problems exactly like this. I haven't used it myself, but according to all reports it is very easy to use, cheap, and works well. Here's an example and a overview I dug up with a quick google to give you an idea what its about.
posted by MetaMonkey at 3:53 PM on April 3, 2007

I use S3. As MetaMonkey says, this scenario is what Amazon S3 was designed for. It's fast, highly available and easy to use. At $20/GB, it will cost you 2 cents every time somebody downloads your video. If you end up getting thousands of downloads per day, then you have a problem most web sites would love to have.
posted by ldenneau at 4:10 PM on April 3, 2007

As I already noted, S3 will be more expensive than mediatemple if you go over 100GB a month, which is only 1,000 100MB videos.
posted by Good Brain at 10:06 AM on April 4, 2007

I just had a look at MediaTemple: 1TB for $20 is one hell of price (a tenth of S3). I'm curious if they deliver this reliably, and what would happen if you go over your 1TB (as the questioner may well do).
posted by MetaMonkey at 10:55 AM on April 4, 2007

Very strong second for a bandwidth bucket at nearlyfreespeech.net. They handle slashdottings regularly and their network is very intelligently and flexibly set up.
posted by allterrainbrain at 1:09 PM on April 4, 2007