How do I tell what requests are slamming my Apache server?

November 28, 2007 11:48 AM

Help my with my Apache Crisis. How do I figure out what requests are responsible for my load?

I am a programmer for a small web dev shop. In the past few days our main http server has been getting slammed with traffic, and the load is spiking several times a day.


I need to figure out how to tell what request in my access logs is causing the load spikes. What I mean is, how can I generate a report that tells me how much ram, cpu time, etc is consumed by a given request. If I had this report, I could sort on cpu % or memory size and then start to pare down what is causing the problem.

I'm on a FreeBSD server running Apache 1.whatever.

T.I.A.!!

posted by judge.mentok.the.mindtaker to Technology (7 answers total)

Maybe someone will chime in with a way to tie specific requests to CPU load, but until then, you can get a quick-and-dirty approximation by logging the size of request (%b) and the time taken to server the request (%D in apache 2, not sure about 1) and looking at those requests that are slower than average.
posted by These Premises Are Alarmed at 11:56 AM on November 28, 2007

What database are you using, and what language/framework is your web app written in? Just telling us it's "Apache" isn't going to get you very far.

If you have any of the heavy modules running (mod_python, mod_perl) so that there's an instance of a programming language interpreter launching with each Apache process, you might want to consider serving your static assets from a lightweight server like lighttpd or nginx. Why do you need the heavy weight of an Python interpreter and an Apache process to serve, say, your favicon, or some jpeg graphic? It's ridiculous.

If you're running MySQL, you can turn on the slow query log and use mysqlsla to analyze it and figure out which particular queries are killing you.

Are you using caching to reduce the number of expensive trips you make to the database? Memcached is very, very easy to use. If you don't need to generate it from scratch for every pageload, then don't.
posted by evariste at 12:01 PM on November 28, 2007 [1 favorite]

apachetop, strace, and aggressive caching are your friends.
posted by Freen at 1:08 PM on November 28, 2007

How are your pages being served? I mean, are they html, PHP, PHP getting content from MySQL, PERL, etc.?

Have you looked at 'top' during a spike? Is the usage mostly httpd, mysql or something else?

Some suggestions to track down the problem:

Try using the slow query log like evariste suggested. You may also try turning on the general query log for a day and seeing if there is a particular kind of query that shows up a lot.

If your pages are served with PHP, try adding some debug functions that use microtime() and memory_get_usage() to figure out how long a particular function takes to run and how much memory your script is using.
posted by justkevin at 1:51 PM on November 28, 2007

You may want to try going with the freeware version of splunk, will help you with some event and error correlation.
posted by iamabot at 2:39 PM on November 28, 2007

It's _hard_ with Apache 1. With apache2, at least you can enable Blackbox Logging.

You may want to put an Apache2 server up in front of it as a reverse proxy, just to get the "time per request" data.
posted by Laen at 2:51 PM on November 28, 2007

Response by poster: These are mod_perl apps running against MySQL, thanks so far for all the info, very helpful.
posted by judge.mentok.the.mindtaker at 8:33 AM on November 29, 2007