Movable Type hangs on rebuild. Where to look?

February 28, 2007 8:49 AM

What's the best way to diagnose why Movable Type hangs when trying to rebuild my site?

I've been doing (daily) blog entries since July of 2002, and rolled my own setup. Recently, I've decided to make the switch over to Movable Type and just got done importing all of my entries (about 5,000 in total).

During my tests (using about 150 entries), I was able to tweak things and rebuild without much of a problem. Now that I've imported all of my entries, I'm no longer able to rebuild.

When rebuilding "Indexes Only," it's fine. But anything else, and I see the little wheel spinning and eventually I get a "Can't Open Page" and "lost network connection" error, which I'm assuming is some kind of timeout due to how much I'm asking the server to do.

In my three-column layout, I'm using modules I created for the Left and Right columns (Alpha and Gamma, respectively). I originally set up a calendar, but have since removed it because of all the reports I've seen about it adding to rebuild times.

I'm building all templates statically. And I've got Individual, Daily and Monthly archives checked.

As a sidenote, I was able to successfully build using Dynamic Publishing. But the rebuild time was significant, and every visit to both the home page and individual pages resulted in a markedly long load time. So while it worked... I'm not certain Dynamic Publishing is the best approach.

Any tips? Suggestions? I've yet to try to go back to my default archive templates, which I guess is the next step. It's just that I feel like I've been doing a lot of random trial and error tests, and would love some advice on how to better diagnose this issue.

I've been banging my head a lot with moving my blog over to MT, and now that I've finally got all my entries up and loaded... I can't rebuild. It feels like I've trekked over this huge mountain range and arrived home, only now I'm unable to get the dang door open.

Thanks in advice for any thoughts.

posted by avoision to computers & internet (18 comments total)

How to Speed Up Publishing in Movable Type
Speed Up
Smart Rebuild - Faster Category Rebuilds
posted by kirkaracha at 9:02 AM on February 28, 2007

It sounds like maybe there's a recursive MTInclude somewhere in there (template A includes B which includes A). That means an infinite loop, which will cause the request to hang.

If you're not using Category archives, it's unlikely that just having a large number of entries will cause a timeout, because the rebuild will be split up into small chunks.
posted by staggernation at 9:05 AM on February 28, 2007

Also, if you open an entry that's set to Published and re-save it, does that one entry rebuild successfully? (As opposed to rebuilding the whole site.)
posted by staggernation at 9:24 AM on February 28, 2007

must look at code. also try rebuilding in a different browser.
posted by phaedon at 9:40 AM on February 28, 2007

kirkaracha - Thanks for the links. I've seen the first two before, but I'll look these over again in case I overlooked something.

FWIW - I originally imported my entries under a category name of "imported." After running into rebuild issues, I deleted the "imported" category, hoping that would help.

staggernation - I can select a published entry, click "Rebuild" and it successfully rebuilds (after 30-45 seconds). On modifying an existing entry and clicking "Save," it successfully saves/rebuilds after something like 90+ seconds.

Something definitely seems amiss, and I assume it's faulty code on my part inside the individual archive. Recursion is a possibility, but doesn't seem to help explain why I'm able to rebuild on an individual entry basis (albeit reeeeally slowly).
posted by avoision at 9:46 AM on February 28, 2007

Template modules are apparently a not-very-efficient way to do things like this. I use them in the way you do as well though!

The best resource for something like this would be a) MT Technical support, if that is available to you or b) Six Apart ProNet.
posted by mikel at 9:48 AM on February 28, 2007

Can you post (here, or on a server somewhere, and give a link) the code within the template modules?
posted by staggernation at 9:57 AM on February 28, 2007

Could it be that your Web host is throttling Movable Type for some reason...? In the past it had a reputation for being a CPU hog. I believe that's not true so much anymore, though.
posted by kindall at 10:05 AM on February 28, 2007

It doesn't sound like a throttling issue--if an entry is taking 90 seconds to save and rebuild, there's definitely something not right with the templates.
posted by staggernation at 11:08 AM on February 28, 2007

Thanks for the advice all. I'm not about to have you go through my code to troubleshoot, so I'll take a look on my own. If I get super stuck, I may take you up on that offer staggernation.

Thanks again for the comments and links!
posted by avoision at 12:23 PM on February 28, 2007

Do contact your host. I moved to Dreamhost, upgrading to the latest MT version along the way but keeping the exact same templates, and ended with nearly the same problems you have. It turned out there was some problem with the server my mysql database was on, or something like that, which they fixed.
posted by kmel at 1:01 PM on February 28, 2007

I've since removed the contents of the template modules, and was able to get my site to successfully rebuild. The downside is that it's currently *still* rebuilding, and we're looking at about 20+ minutes so far (and still going).

This leads me to think that part of the problem is in the modules. But what could cause this long of a rebuild? The individual archives?

kmel - I'll send a note to my host, but my hunch is that it's an error I introduced. Worth checking out though - thanks!
posted by avoision at 1:11 PM on February 28, 2007

That does seem like there's probably something slowing it down. But also keep in mind that if you're using stylesheets and PHP includes, you should rarely have to rebuild the entire site once it's up and running. My guess is that some of the things you were using MTIncludes for could be replaced with PHP includes.
posted by staggernation at 2:28 PM on February 28, 2007

I was originally pulling in code via the Template Modules, thinking I was going to ease the stress on my server. After reading comments here, I decided to try to pull out some code into an extranl php and include those files instead.

Turns out, MT tags wouldn't process, and so that was out. I don't know why I didn't think of it earlier, but I pulled out most of my column code and tags, and created new Index Templates. Then, on my archive pages, I used a php include to point to these new index templates. In this way, I'm able to tweak and rebuild the new index templates at my leisure, without having to do a complete site rebuild.

Things are definitely a bit faster, but it's still taking somewhere in the neighborhood of 20-30+ minutes for an
entire site rebuild.

This has been a slow process, it feels. For every thing I do right, there seems to be two things I need to fix or find workarounds for.

Thanks again to everyone for posting advice and links and ideas. Muchly, muchly appreciated.
posted by avoision at 3:18 PM on March 1, 2007

Hi there, I'm the former Movable Type product manager and have been doing MT development since it debuted. I also wrote one of the articles linked above and a book on the subject. :-) More importantly though, I've diagnosed and fixed such problems more times than I can count and I can tell you that without a doubt, the problem is either in the templates themselves or in what you're asking MT to output.

If you're using a modern version of MT (particularly 3.3x), modules do not have the performance hit they used to and are extremely efficient. However, poorly coded modules or templates will still kill your rebuilds.

20-30 minutes per site rebuild is too long. So, to diagnose your problem, a few things would be helpful:

1) Tell us what version of MT you're using

2) Tell us what plugins you are using

3) Can you post a screenshot of your archive mappings (Your blog » Settings » (detailed settings) » Publishing (bottom of the page)

4) Can you copy the template code for your Archive templates and template modules somewhere where we can take a look at them

5) Do you REALLY need daily archives?

That information should give us everything we need to know, but if not, it's a great start.
posted by fooljay at 3:49 PM on March 2, 2007

Hi Jay -

At this point, I'm feeling weird continuing to post to this thread and would be happy to take the discussion to email. However, I also don't mean to presume that you'd have the time to address my issues privately.

In response to your questions, here's what I've got:

1) 3.33
2) Just the standard stuff that came with: Google Search, No follow, SpamLookup (Lookups, Link, Keyword Filter)
3) Posted here
4) Posted as text files: Main Index,
Left Column (Index Template),
Date Based Archive,
Individual Archive
5) I believe I do need daily archives. I rolled my own blog and have a ton of internal links that refer to specific days.

Old example: /2006/01/02.php
Changed to: /2006/01/02/

Given this change, I assumed that I needed Daily Archives to catch any links that point to a specific day (I could be wrong).

Hope this helps describe my problem more. Thanks again for taking the time.
posted by avoision at 6:34 PM on March 4, 2007

Hey there avolsion. So, given what you've posted above, the only big resource hogs that I see are the calendar (which you said you removed but seems to be in both the left column and individual archive templates) and the archive lists (which are much better, however, in a single index template.

Other than that, I can't see anything terribly out of line from what you posted. I wonder if the problem is in your right column template module which you refer to in both the individual archive and date-based archive templates, but didn't post here. That's all I can imagine at this point.

You say that the only rebuild mode you can successfully finish is Rebuild Indexes? You've tried individually daily, monthly and individual rebuilds?

And as far as the daily archives, I'm pretty sure that with a little URL rewriting, you could do without them as well. For example, you could rewrite /2006/01/02.php and /2006/01/02 to /2006/01/#date-02 and change this:

<MTDateHeader><h2 class="date-header"><$MTEntryDate format="%x"$></h2></MTDateHeader>

to this:

<MTDateHeader><h2 id="date-<$MTEntryDate format="%d"$>" class="date-header"><$MTEntryDate format="%x"$></h2></MTDateHeader>

Then, all of those daily links will point to the day's entries on the monthly archive which you're already publishing.

Or, if your previous links already had named anchors, don't bother putting the #date-02 at all on the end of the rewrite as your previous one will take the user to the right place on the page (the entry ID, it seems, from your site).

I guess the rule of thumb that I've always stuck by and has served me well is, don't overtax the system by asking it to write things out over and over again. Pick a single date-based archiving that works with your posting schedule and use that alone. Publish individual archives as THE canonical URL for an entry.

I sloughed off some of the older methods of doing things I had contrived (e.g. using Entry IDs for permalinks) because they were wrong-headed. Instead of forever living with that pain, I made a clean break and moved forward. It certainly doesn't have to be painful for you or your visitors, but what you don't want to do is enshrine the past in the present at your own expense. After all, if it's painful for you, that's going to make you blog less... :-)

By the way, all of your permalinks seem to exhibit the following error at the moment.

Page Not Found

Smarty error: [in mt:85 line 91]: syntax error: mismatched tag {/if}. expected {/MTEntries} (opened line 51). (Smarty_Compiler.class.php, line 2286)

posted by fooljay at 8:17 PM on March 4, 2007

Hi Jay -

Yeah, things are a bit different on my blog since I first posted my question. I've pulled out most of the left column content, and dumped it into an Index archive (search, recent entries, archive list).

I should mention that I'm able to rebuild everything now, ableit slowly (20-30+ minutes). When I removed the calendar, I still experienced 20-30 minute build times, so I left it in.

I think I grok what you're suggesting, as far as the links go. My sense is that by changing links from "/2006/01/02/" to "/2006/01/#date-02"... it'll load up my monthly archive, and jump to the appropriate spot on that monthly archive page. Is that correct?

I'll consider this approach. I think the load times will be massive for montlhy pages (I've been making daily entries, and I plan to maintain that schedule), and that makes me pause a bit. However, I do see how wasteful it is to ask MT to kick out daily archives as well.

Regarding the right column content, it's a straight php include with just text and images and links. Nothing fancy going on over there, but I can post that code if you want.

Regarding the errors: I've since taken to creating a fake blog on my site that comprises 50 entries or so. Since the calendar didn't seem to cause extra stress, I decided I could test out the best template structure on a smaller scale - and then deal with one big rebuild on my main blog.

To that end - I've got one remaining problem I'm trying to troubleshoot here. If I can figure this out, I'll deal with the 20-30 minute rebuilds and call it a day. I'm close to having things where I want, and quite frankly - getting tired of spending my time battling MT vs. spending my time creating entries.

Finally - is there a good place to go for information on MT and tag syntax, etc. I'll check out your book, but I was hoping there would be more resources online. The forum seems dead to me, as I've answered most of my own questions there. I'm looking through Google searches as often as I am my own template code, trying to suss things out.

Thanks again.
posted by avoision at 10:49 PM on March 4, 2007