Has MT been stress tested?

August 5, 2005 8:34 AM

Movable Type: How does it fare under high-volume conditions?

My boss has an idea he's sure is going to be a smash hit. Surprisingly, I think he's right. I can't go into details, but he wants to use MT to manage a dozen blogs and 1-2 million unique visitors per month.

Is this practical? Are there any stats available on how MT holds up under that kind of stress? (Obviously we'd be shelling out for a robust hosting package.)

posted by crickets to computers & internet (15 comments total)

The MT 2.x series builds static html files. If this hasn't changed in MT 3.x, then it doesn't get any more scalable than that.

That is, viewing the blog or archives incurs _no_ MT-related processor load -- merely the webserver load of delivering a static file (i.e. very low, and fundamental to the task.)

Unless you're doing things that cause MT to rebuild all the static files all the time (like redesigning your templates,) MT should not be a limiting factor.

Doing searches and users posting comments is another story -- that does incur MT-related load. Are they part of the picture?
posted by Zed_Lopez at 8:47 AM on August 5, 2005

It ought to scale well at that size too. About.com is powered by MT
posted by tonelesscereal at 8:48 AM on August 5, 2005

Zed nailed it. Static files means your bottleneck is your web server, not MovableType.
posted by waxpancake at 9:07 AM on August 5, 2005

> Doing searches and users posting comments is another story -- that does incur MT-related load. Are they part of the picture?

searches, yes. comments, not so much. maybe some of the blogs but not all of them.
posted by crickets at 9:21 AM on August 5, 2005

Cheap hosting companies (at least mine) are starting to push users from MT to WordPress because they say comment spam can create incredible strain on the CPU since pages are rebuilt each time a comment is added, and my understanding was that some strain was still caused even if you had MT-Blacklist installed.

I switched to WordPress last fall, so I don't know if they've addressed that problem since then. I'm sure Anil will be by shortly to let us know. :)

Also, with 3.0+, you can have pages be dynamically generated if you want. I haven't ever tried it, though.
posted by katieinshoes at 9:29 AM on August 5, 2005

Movable Type's comment (mt-comment.cgi) and trackback (mt-tb.cgi) are both vulnerable to spamming, even if you're not using those features on the site, so consider renaming, deleting, or disabling them. TypeKey is an option for authentication if you want to allow comments.

Six Apart has articles on Combatting Comment Spam and How to Speed Up Publishing in Movable Type.

And you might want to consider site-flavored Google Search instead of using MT's built-in script.
posted by kirkaracha at 10:07 AM on August 5, 2005

Though I use and love WordPress, Movable Type would definitely be the way to go. WordPress has some hacks to create static pages, but it's nothing fundamental to the program, and those hacks are, of course, rather awkward.

This is the sort of task in which Movable Type really shines.
posted by waldo at 10:23 AM on August 5, 2005

Waldo, in the current version of WordPress, no such hacks are required. Select "Write Page" under "Write" and create your page. Hardly awkward. You can also create multiple page templates and use a different template for each page you create, and pages can be organized hierarchically. It's actually pretty awesome.
posted by katieinshoes at 10:54 AM on August 5, 2005

A few observations:

I either host or participate in a few MT installations, and one or two of them are high-traffic enough that I can personally vouch for its stability under high traffic. Of course, as much depends on the hardware hosting the weblog software as does the weblog software itself -- but you said that you'd be willing to shell out for good hosting, so you should have that covered.

With comments and/or TrackBacks enabled, MT certainly is susceptible to spamming -- but I have yet to see how any other weblog software package is less susceptible. I'm not looking to start a religious war here, but my experience has been that MT is only "more susceptible" in that the installed base was larger when comment spamming started surging, so a lot of the spamming tools used fixed characteristics of MT installations to seek them out as targets. The spammers seem to be much more weblog-backend-neutral these days, though.

MT has support for multiple weblogs out of the box; many other solutions don't have that, and it appears that it's critical to your boss's solution that you be able to work with multiple weblogs.

Lastly, MT version 3.2 is about to come out in final form (there's been a beta for a few weeks, in which I've participated), and it does a TON to serve what seem to be your core requirements. It can do either static publishing (meaning that the sites that don't do commenting and TrackBacks can have literally *no* MT overhead when it comes to serving pages to readers) or dynamic publishing (meaning that on the sites that might have commenting, every comment doesn't force a behind-the-scenes rebuild, lowering the overhead). It can allow moving of posts between weblogs. It has a bunch of new options for comment management (increasing the ability to moderate, require authentication, filter, etc.). It's worth taking a look.
posted by delfuego at 11:30 AM on August 5, 2005

I can't speak for Movable Type, but one fellow survived a Slashdotting with WordPress and a Celeron 466.
posted by Alex Handcoding at 12:14 PM on August 5, 2005

Disclaimer, as always: I do work for Six Apart (which makes Movable Type and TypePad) and I'm notorious for jumping in on these threads. :)

That being said, we've got some *very* high volume sites running on Movable Type right now. From the Gawker Media family of sites (Gawker, Gizmodo, Lifehacker, Wonkette, etc.) to the Gothamist sites to the NYTimes's About.com, there are a large number of commercial sites that are scaling up to millions of page views per day while using Movable Type to manage their publishing.

As a few people have mentioned above, using static publishing by default means that you don't incur any overhead in serving up pages beyond the basic web server load of delivering a page.

For search, the built-in Movable Type search isn't appropriate for the largest-scale sites, but that's a pretty high bar. If you've got an extremely high load of search traffic, we'd recommend some of the search plugins (like X-Search/Plus) available for the platform, which use more powerful engines and smarter indexing to offer better performance.

If you're interested in best practices for a high-availability MT-powered site, we've also got our Professional Network community, with members who've done a lot of this type of work before who can share their experiences. We also have a number of hosting partners that offer Movable Type preinstalled as part of a hosting package.

Finally, we've definitely had performance issues with servers that were under a very large comment or TrackBack spam attack in the past. We've done a lot of work to optimize for those situations, and the 3.2 beta mentioned above has a junk folder to better manage these kinds of attacks in the future without undue load. If you're interested in minimizing this concern, you can turn off TrackBacks or comments at the system level, run the application under different environments (mod_perl or fastcgi instead of cgi) and/or use some Apache mod_security settings to reduce the possibility of attacks affecting your site.

And we've got some suggestions we can make as well; TypePad is our hosted service built on Movable Type's engine, hosting hundreds of thousands of blogs around the world, and we usually have half a dozen sites a day that get slashdotted (or the equivalent). So we can offer some firsthand experience as well.

Hope that helps! Feel free to get in touch (email's in the profile) if you need more help.

(Also, Alex, could you find some stats on people scaling up using Microsoft FrontPage serving pages on an Amiga? That'd be equally off-topic, but I just wanted to see if you were up for it.)
posted by anildash at 1:10 PM on August 5, 2005

I'm currently hosting my MT blog on a smallish community-style webhost service, and I happen to know the owner. At some point, I became a target for massive amounts of comment spam. I only get a handful of unique visitors a day, but apparently, there are scripts searching for the mt-comments.cgi file hundreds of times a day, causing a LOT of activity on the server and eating up tons of resources. (And yes, I had mt-blacklist installed and properly configured and updated). After the host notified me of this, I turned off comments on my blog and removed all links to that script. My site still got massive amounts of traffic to that one .cgi script. So I renamed it mt-comments-disabled.cgi. Still tons of traffic, although it's now being handled by the host, serving up 404 messages thousands of times a day, rather than that renamed script working it's scripty magic.

Bottom line: comment spam can be a serious problem with Movable type whether you have comments turned on or not, whether you have a blacklist plugin installed, whether you even have a comments script included as part of your installation. If my 6-hit a day blog can become a noticable drag on a smallish server, you might have server load issues with a site serving a million visitors a month. Of course, YMMV, and this may depend on a variety of issues.
posted by cathodeheart at 4:30 PM on August 5, 2005

Cathodeheart, that'd be true whether or not you're even running Movable Type; you're describing a situation where despite the CGI not even being there, there are spambots trying to use it. The same thing can happen to a WordPress blog, or a Blogger blog, or whatever -- the spambots won't care, since it's low-cost to them to try anyway.
posted by delfuego at 5:10 PM on August 5, 2005

Oh, and if your host has a problem with server load that's based on its (in)ability to efficiently serve 404 pages, then you might want to look into other hosts -- it doesn't get more optimizable than checking to see if a file exists and returning a static, cacheable, *short* HTTP response. Your host might be running CGIs through wildly inefficient preprocessers or filters or any number of before-the-hit-gets-to-the-server steps, but you shouldn't have a huge performance hit from returning 404s.
posted by delfuego at 5:14 PM on August 5, 2005

A few years ago, I worked at a large level 1 trauma center that had 5 or 6 hospitals in the system. For that large site, I used Movable Type to setup about 20 different blogs, without trackbacks or comments, to serve as the CMS for the site. We would get, on average, about 100k hits per month...that unique. While we weren't getting a million, we were just serving static files off of an iis-based server. It could handle what we threw at it, without too many problems. Mainly we were doing this on IIS and we had some hurtles to jump.

While I no longer work for the company, I headed off to greener pastures, the site is still up and has won some awards (although their new person has made some changes, honest it looked much better). While it's not a "blog", it's a good example of using MT in business and putting it up as target practice for visits.
posted by mkelley at 9:15 AM on August 6, 2005