Can't deal with blog's explosive growth

February 9, 2007 5:12 PM

My topic-oriented blog got more popular than I ever expected and I don't know how to handle the growth (this isn't spam, I'm not even linking to the site).

I started a blog at the start of November 2006, believing that I could make it somewhat popular because I'm a fairly good writer - and I can write a lot.

The growth was rapid and immediate - I saw 40% growth each week for the first three months of the blog's existence.

By early February, the blog was getting 20K hits a day, and then earlier today I had a post appear on digg and reddit.

It was already in the top 2800 on Technorati as well.

Also, I launched this blog with no promotion - it was basically intended to just be a place for some self-exploration that apparently hit a chord.

Anyway, this was apparently too much traffic for my host provider (dreamhost), so they simply turned off my site, changed the permissions on the files, and I can't access a thing, even if I wanted to move it.

I'm at a point right now where the site is really popular - but it's also really new. I simply don't have the money right now to afford dedicated hosting for the site - I might be able to if I had a sponsor or advertiser of some sort, but the ridiculously explosive growth of the site meant that it wasn't on the radar for a lot of potential advertisers yet.

My site is now inaccessible both by the web and by FTP, though I have a day-old backup of everything, so that's not a major issue (I would just lose a few days of posts). Unfortunately, it's not up anywhere at all.

What should I do? I honestly don't know where to go from here.

posted by krark to Technology (28 answers total)

my host provider (dreamhost), so they simply turned off my site, changed the permissions on the files, and I can't access a thing, even if I wanted to move it.

Wow. What assholes. You'd think that hosts would have some kind of system worked out by now for handling big spikes to little sites. Call them. I'm sure they have higher capacity services available. You probably signed up for a limited personal account at first. If nothing else, they'll probably be happy to sell you on a more robust package, although it will cost you. They really should give you read access, anyway. And I'm surprised that they have no more intelligent way of handling this than to just kill the account. That's dumb (although I'm sure you agreed to it when you signed up).
posted by scarabic at 5:27 PM on February 9, 2007

Eric Schaeffer? Is that you?

Okay, kidding... but seriously, a quick Google search shows that your 2/8 posts are still cached, so you can probably snag whatever you're missing if you don't wait.

And then, since you have backups, just relocate it all. Don't try to get back in with Dreamhost -- if they handled you this poorly the first time, what happens when your site gets really big?

Get your registrar to redirect your URL, and done. You could be set up with Verve in a matter of hours if you can resolve your DNS issues.

Verve lets you "top off" or pro-rate extra bandwidth if you go over whatever is offered in your monthly package; in fact, if you alert them that you are expecting a big burst of traffic or that there's been an "event", they'll keep an eye out and not let your site go down (I used them for some Congressional campaign sites in 2004 -- the customer service beats anything you've ever experienced online). It's a quick-start sign-up and monthly pay-as-you-go, too, with no contracts.

(No, I'm not a Verve reseller nor do I get referral credit -- I've just used them on dozens of sites over the last few years and recommend them highly.)

Then, if you can prove your traffic or get a sponsor, you can get on with BlogAds or a similar outfit pretty quickly, and start bringing in some income to cover your hosting.
posted by pineapple at 5:33 PM on February 9, 2007

Congrats on the success of your site!

Perhaps you could move your blog over to a free host like Blogger, at least until you are ready to pay for your own hosting. You'll lose the momentum of existing links, though. Maybe there is some way you can get Dreamhost to point your old URL to a new one.

Once you get back online, you can start placing ads on your site immediately through chitika.com and Amazon Associates, and can apply for a Google Adsense account for contextual ad placement. (I think blogger blogs have this option enabled immediately.) You can also submit your site to Blogads and hope they will add you to their pool of blogs. Once you start making some money, you'll have more options for what to do with your content.
posted by Scram at 5:38 PM on February 9, 2007

Can you still log into the control panel at Dreamhost? Have you submitted a support inquiry there yet? Or checked the Status > Bandwidth/Disk/Mysql Usage stats?

Likely, your site was chewing up CPU and affecting other customers on your shared server. (I'd have expected them to notify you by email at least, though.)
posted by xiojason at 5:51 PM on February 9, 2007

Are you worried only about traffic and hosting?
Or do you want broader advice on what "blogging path" you should follow?
posted by AnyGuelmann at 5:52 PM on February 9, 2007

uh, contact dreamhost?
posted by Señor Pantalones at 6:08 PM on February 9, 2007

Response by poster: I attempted to contact Dreamhost, but their only response was to chmod all my files to 000 and haven't responded at all. That's why I realized there's some fundamental problem going on.
posted by krark at 6:13 PM on February 9, 2007

That was their sole response to your inquiry? There was no response statement at all? What was the situation like before they changed the file permissions then? Can you still log into the panel? Can you still submit (another) support request?
posted by xiojason at 6:15 PM on February 9, 2007

This is why Dreamhost sucks. They promise you shitloads of bandwidth, but in the fine print they are allowed to shut you down at any point for excessive CPU use, which occurs rather frequently. You need a host that's not a shared server. Virtual hosts (VPS/VDS) are very cheap, and you get an entire virtual machine to yourself. You should be able to find something that fits your needs in this category for $10 - $25 a month easily.
posted by Rhomboid at 6:19 PM on February 9, 2007

A good place to find deals on VPS packages.
posted by Rhomboid at 6:21 PM on February 9, 2007

I have been a pair.com customer for several years now.

Wouldn't ever consider anyone else now.
posted by briank at 6:30 PM on February 9, 2007

Adsense. With that much traffic, a few well placed Adsense ads could probably more than cover the cost of your hosting/bandwidth.
posted by RoseovSharon at 6:31 PM on February 9, 2007

I highly reccomend Nearlyfreespeech!

They are used to bandwidth spikes, their business model is awesome (for the customer), and they support free speech. Look into their "bandwidth bucket" program as well.
posted by phrontist at 6:49 PM on February 9, 2007

Response by poster: They did inform me that they were disabling things, but they won't respond to any further requests, not even allowing me to retrieve my data.
posted by krark at 6:53 PM on February 9, 2007

Very strong second for nearlyfreespeech.net. I've used half a dozen hosts over the last 7+ years and they are the only one I can recommend unequivocally.
posted by allterrainbrain at 7:37 PM on February 9, 2007

What exactly did Dreamhost say when they told you they were disabling things?
posted by litlnemo at 7:47 PM on February 9, 2007

You know, a few minutes ago I tried unsuccessfully to get to your site (I have it bookmarked). Then I came here and saw your post. Freaky. I guess I am part of your problem because I'm one big fan of your blog.

Sorry I don’t have a solution for you, but I’m sending good thoughts your way in hopes that everything will be back to normal for you soon.
posted by vewystwange at 7:49 PM on February 9, 2007

Krark, do you have phone numbers for Dreamhost? I've had problems with providers in the past (never Dreamhost), and a phone call always provides a better response. Also, Dreamhost is very, very slow in responding to trouble tickets. I put in a ticket yesterday morning because my site was down (due to the PHP/WP problem), and they didn't get back to me for about twelve hours.

I see your site is still down. When did it go down? Do you know how much traffic it took to bring it down? I've never had prolonged surges at my site, but I've had ~30,000 visits in about 12 hours and Dreamhost has handled it fine. I'm guessing your traffic must have been much higher.

If you do find a more suitable replacement host, please let me know. I'm happy with Dreamhost at the moment, but I like to have options lined up.

Good luck — I like having you out there as a colleague!
posted by jdroth at 8:43 PM on February 9, 2007

Whatever you do, fix it fast. Kind of rare to have a hit site.
posted by xammerboy at 9:53 PM on February 9, 2007 [1 favorite]

Even if you don't have your hosting hoohah worked out right away, I strongly advise getting your domain repointed to a single-page, low-bandwidth (no images) page that says "Our web host dumped us and we're moving to a new one, back in a few days." Otherwise, a lot of that traffic you were getting will dry up as people thing you're just dead in the water.

FWIW, if you don't have this already: get your domain moved to a place that lets you manage your own DNS. Once you've done this, a DNS 'A' record change (and a quick flush by the provider after a phone call) could have your domain repointed in minutes (24 hour worldwide turnaround due to cacheing, however.)
posted by davejay at 10:43 PM on February 9, 2007

I've been considering signing up with dreamhost - but I've arrived upon your site more than a few times (and a few others that I've found are hosted on dreamhost) and it's been down - so that's really turned me off.

The good news is it has turned me off of dreamhost, and not your site.

I know you're a fan of dooce - she gets a boatload of traffic and I notice she's on liquid web. I've never experienced any blackouts on her site - so maybe they're worth looking into?

Good luck sorting this out, and I look forward to seeing you back online.
posted by backwards guitar at 5:29 AM on February 10, 2007

I've had good luck with dreamhost on all kinds of fronts, but if your site is getting serious traffic, I wouldn't necessarily trust them to handle it for you. (Although I've had all kinds of reddit/digg/etc... hits on sites hosted by them with no trouble, so ymmv I guess.)

Did they say anything?

Anyway, regardless of that... if this is going to continue to be a fairly serious undertaking, you have to get with a web provider who knows who you are personally.

Identify your technical needs, estimate your future traffic, and set a budget for the project. Then spend the day calling around, talking to hosting providers. Start with the people who host blogs you like -- they obviously know how to keep blogs online, like bg said.

I bet a deal can be worked out there.

I'd also strongly recommend that you follow Davejay's advice and put up some kind of message about how/why the site is down. Everyone can understand that kinda thing.

Two things I might add:
- If you have control of your own DNS, it gives you a ton of flexiblity to jump from host to host as needed. You can always use someone like zoneedit.com for your nameservers. I think they can do basic redirects if you need that in a pinch. I think it even has some kind of failover capability.
- I've never seen your blog, so I'm not sure wassup with it, but depending on the software, you might be able to ring massively for efficient performance out of it for your particular application.
posted by ph00dz at 6:34 AM on February 10, 2007

If my Google skills are what I think they are, the site is loading for me, aside from the stylesheet.
posted by loiseau at 9:30 AM on February 10, 2007

Can't you get together enough money to cover a dedicated server for a month or two? Or approach a few small hosting firms about sponsorship? That kind of growth in popularity is a great opportunity, not a burden, and if you can sustain it you'll easily surpass hosting costs with advertising.
posted by malevolent at 9:33 AM on February 10, 2007

krark, i work with the TypePad team, and we host some heavy-duty sites (Wired, Time, Washington Post) ... if you're still looking for a host, a lot of the most popular niche blogs are on our service, and at the very least I can connect you with some references to other bloggers who've gone from zero to very popular in a short amount of time. We've also seen a lot of people make more than enough money to support their site with just some AdSense ads and a few sponsorships courtesy of partners like Federated Media or FeedBurner.

My email's [e-mail] if you want to get in touch.
posted by anildash at 11:04 AM on February 12, 2007

i still find this surprising. the level of brevity and disinterest you've mentioned in dreamhost's responses seems completely out of character. i've used them for 4+ years now and have never seen anything like that.
posted by Señor Pantalones at 2:08 PM on February 16, 2007

Several people have asked the OP what exactly Dreamhost said, etc., trying to get a better idea what the problem is, and the OP hasn't been forthcoming. So yeah, there's something odd about this.
posted by litlnemo at 6:15 PM on February 16, 2007

Response by poster: In case someone stumbles across this, I wanted to mention that Dreamhost did eventually resolve the situation in a very equitable fashion, but only after two days of downtime. They claimed that they believed my site was getting hit with a DDOS attack.
posted by krark at 7:56 PM on March 9, 2007