Bizarre Apache/PHP issue.

January 29, 2011 12:08 PM

I have a really weird Apache/PHP problem. Ubuntu 8.04 LTS Server, fairly typical webhosting setup. One of our sites returns 404s randomly through the day, typically late at night, but sometimes in the morning. It'll do this for an hour or two and then start working all on it's own. The other sites are fine.

If it helps, this site is a CakePHP deal. I'm 99% sure it's not anything in cron because although it typically happens around 1am-4am, the times are very inconsistent. Once it happened for 15 minutes in the middle of the day, sometimes it happens around 7am for about an hour. In a browser, it will load part of the page, then 404.

The apache log looks like this:
[28/Jan/2011:02:38:58 -0700] "GET / HTTP/1.0" 404 9444 "-" "mon.d/http.monitor"

The few times I've caught it while it's happening, load does not seem to be very high, maybe a little higher than average, but nothing above 1.0.

I'm pretty stumped here, any help would be appreciated.

posted by signalnine to computers & internet (9 answers total)

Given the log line, it seems like mon might have something to do with it. Or are you using mon to find out when you're getting the 404s?
posted by zsazsa at 12:45 PM on January 29, 2011

Does the timing coincide with a database connectivity issue? Database downtime?

I expect you need to check for CakePHP settings in the event of failover -- what is supposed to happen if CakePHP fails?
posted by artlung at 12:48 PM on January 29, 2011

Are you using any sort of caching for the site?
posted by xtine at 1:15 PM on January 29, 2011

Apache and/or PHP should have an error log, separate from the one-line-per-request access log. Hopefully that will contain more useful information.
posted by teraflop at 1:26 PM on January 29, 2011

mon is most likely configured to monitor and ip, but you are using name based virtual hosting.
posted by devnull at 1:38 PM on January 29, 2011

Do the php files live on a non-local (nfs) mounted filesystem? If there anything in /var/log/messages that looks related?
posted by DrumsIntheDeep at 2:40 PM on January 29, 2011

mon is what I'm using to monitor the site, it lives on another server. It is doing http based monitoring.

I think at some point we were using APC for caching, but I dont think it's enabled at the moment- could that have something to do with it?

All files are local.
posted by signalnine at 3:24 PM on January 29, 2011

Could you please clarify the symptoms for me? Is the following scenario true:

• User is attempting to load the site.
• A 404 appears in the Apache access logs.
• User sees a 404 appear during their browsing experience, complains to your site admin.

Is that correct? Are you stating that the 404 is disrupting your users' experience, or are you stating that the 404 is a spurious and unexplained line in your Apache access logs that is not disrupting your users' experience? Please be as specific as possible about the symptoms.

Ultimately, the single line you've posted in the question is Apache being unable to find the root resource for a user agent identifying itself as the http.monitor module of mon. This suggests that there are no user-facing symptoms to your issue. However, I'm unable to be certain about this because you state in your post that:

It'll do this for an hour or two and then start working all on it's own.

which sounds like you're suffering from an extended outage.

In order to diagnose the issue further, I think everyone needs more insight into your configuration and log output from your system. Could you please provide an instance of the 404 in the Apache log along with the following log output, perhaps using pastebin.com:

• mon logs, default location is /var/log/mon
• Apache error logs, maybe in /var/log/{apache,apache2}/error.log
• PHP error logs, maybe in /var/log/{apache,apache2}/php.errors
• /var/log/messages output

Also your mon configuration, available in /etc/mon/mon.cf, would be useful.

(Also, I'd like to ask the obvious...is your root resource actually available? Some sites config "index.html" to be available and not the root resource. In order to confirm this you could post your Apache configuration to pastebin as well).
posted by asymptotic at 6:50 AM on January 30, 2011

previous post with similar symptoms. I wasn't aware you could increase the verbosity of the Apache logs; you should do this before posting the diags from your repro.
posted by asymptotic at 6:59 AM on January 30, 2011