Bizarre Apache/PHP issue.
January 29, 2011 12:08 PM
I have a really weird Apache/PHP problem. Ubuntu 8.04 LTS Server, fairly typical webhosting setup. One of our sites returns 404s randomly through the day, typically late at night, but sometimes in the morning. It'll do this for an hour or two and then start working all on it's own. The other sites are fine.
If it helps, this site is a CakePHP deal. I'm 99% sure it's not anything in cron because although it typically happens around 1am-4am, the times are very inconsistent. Once it happened for 15 minutes in the middle of the day, sometimes it happens around 7am for about an hour. In a browser, it will load part of the page, then 404.
The apache log looks like this:
[28/Jan/2011:02:38:58 -0700] "GET / HTTP/1.0" 404 9444 "-" "mon.d/http.monitor"
The few times I've caught it while it's happening, load does not seem to be very high, maybe a little higher than average, but nothing above 1.0.
I'm pretty stumped here, any help would be appreciated.
If it helps, this site is a CakePHP deal. I'm 99% sure it's not anything in cron because although it typically happens around 1am-4am, the times are very inconsistent. Once it happened for 15 minutes in the middle of the day, sometimes it happens around 7am for about an hour. In a browser, it will load part of the page, then 404.
The apache log looks like this:
[28/Jan/2011:02:38:58 -0700] "GET / HTTP/1.0" 404 9444 "-" "mon.d/http.monitor"
The few times I've caught it while it's happening, load does not seem to be very high, maybe a little higher than average, but nothing above 1.0.
I'm pretty stumped here, any help would be appreciated.
Does the timing coincide with a database connectivity issue? Database downtime?
I expect you need to check for CakePHP settings in the event of failover -- what is supposed to happen if CakePHP fails?
posted by artlung at 12:48 PM on January 29, 2011
I expect you need to check for CakePHP settings in the event of failover -- what is supposed to happen if CakePHP fails?
posted by artlung at 12:48 PM on January 29, 2011
Apache and/or PHP should have an error log, separate from the one-line-per-request access log. Hopefully that will contain more useful information.
posted by teraflop at 1:26 PM on January 29, 2011
posted by teraflop at 1:26 PM on January 29, 2011
mon is most likely configured to monitor and ip, but you are using name based virtual hosting.
posted by devnull at 1:38 PM on January 29, 2011
posted by devnull at 1:38 PM on January 29, 2011
Do the php files live on a non-local (nfs) mounted filesystem? If there anything in /var/log/messages that looks related?
posted by DrumsIntheDeep at 2:40 PM on January 29, 2011
posted by DrumsIntheDeep at 2:40 PM on January 29, 2011
mon is what I'm using to monitor the site, it lives on another server. It is doing http based monitoring.
I think at some point we were using APC for caching, but I dont think it's enabled at the moment- could that have something to do with it?
All files are local.
posted by signalnine at 3:24 PM on January 29, 2011
I think at some point we were using APC for caching, but I dont think it's enabled at the moment- could that have something to do with it?
All files are local.
posted by signalnine at 3:24 PM on January 29, 2011
Could you please clarify the symptoms for me? Is the following scenario true:
Ultimately, the single line you've posted in the question is Apache being unable to find the root resource for a user agent identifying itself as the http.monitor module of mon. This suggests that there are no user-facing symptoms to your issue. However, I'm unable to be certain about this because you state in your post that:
In order to diagnose the issue further, I think everyone needs more insight into your configuration and log output from your system. Could you please provide an instance of the 404 in the Apache log along with the following log output, perhaps using pastebin.com:
(Also, I'd like to ask the obvious...is your root resource actually available? Some sites config "index.html" to be available and not the root resource. In order to confirm this you could post your Apache configuration to pastebin as well).
posted by asymptotic at 6:50 AM on January 30, 2011
- User is attempting to load the site.
- A 404 appears in the Apache access logs.
- User sees a 404 appear during their browsing experience, complains to your site admin.
Ultimately, the single line you've posted in the question is Apache being unable to find the root resource for a user agent identifying itself as the http.monitor module of mon. This suggests that there are no user-facing symptoms to your issue. However, I'm unable to be certain about this because you state in your post that:
It'll do this for an hour or two and then start working all on it's own.which sounds like you're suffering from an extended outage.
In order to diagnose the issue further, I think everyone needs more insight into your configuration and log output from your system. Could you please provide an instance of the 404 in the Apache log along with the following log output, perhaps using pastebin.com:
- mon logs, default location is /var/log/mon
- Apache error logs, maybe in /var/log/{apache,apache2}/error.log
- PHP error logs, maybe in /var/log/{apache,apache2}/php.errors
- /var/log/messages output
(Also, I'd like to ask the obvious...is your root resource actually available? Some sites config "index.html" to be available and not the root resource. In order to confirm this you could post your Apache configuration to pastebin as well).
posted by asymptotic at 6:50 AM on January 30, 2011
previous post with similar symptoms. I wasn't aware you could increase the verbosity of the Apache logs; you should do this before posting the diags from your repro.
posted by asymptotic at 6:59 AM on January 30, 2011
posted by asymptotic at 6:59 AM on January 30, 2011
« Older If I could just get my brain around this I would..... | Looking for recipes, it's as simple as that Newer »
This thread is closed to new comments.
posted by zsazsa at 12:45 PM on January 29, 2011