Apache went boom. Diagnosis?
December 14, 2010 1:13 PM Subscribe
What the heck just happened to my Apache server?
posted by bhance to Computers & Internet (7 answers total)
I'm running an Apache 2.2 server on an HP Fedora Core box - a LAMP setup. This runs a CMS (SilverStripe) that does a smallish amount of traffic. Nothing huge, maybe a hundred visitors per day.
This morning I got a lovely "Hey the website is down" email.
So I then:
- Attempt to load site, it fails.
- Log in, several hundred httpd processes are running.
- Restart httpd, everything's happy. Site loads now.
- Check netstat, and there's a block of about 10 IP's in the Phillipines that are scraping the website.
- I block the IP's, start looking at logs.
Here's what I'm curious abount - right when the server started losing its brains, the httpd access logs started listing accesses out of sync, like this (which was roughly from 8:09 to 8:18)
14/Dec/2010:08:09:53 -0800] "GET
14/Dec/2010:08:02:58 -0800] "GET
14/Dec/2010:08:09:57 -0800] "GET
14/Dec/2010:07:59:17 -0800] "GET
14/Dec/2010:06:58:41 -0800] "GET
14/Dec/2010:08:14:01 -0800] "GET
14/Dec/2010:08:14:02 -0800] "GET
14/Dec/2010:07:08:31 -0800] "GET
14/Dec/2010:08:16:02 -0800] "GET
14/Dec/2010:08:08:53 -0800] "GET
(... more of the same until I restarted httpd)
Reading up on this I see that apache only log sevents at the end of the request - so this means that many old httpd processes (07:59:17,06:58:41,07:08:31) were hanging out and finishing very late. I'm guessing there were several hundred of these.
So - what gives? Is this some horribly coded site scraper somewhere just eating up all my httpd processes - or was this an actual attack, something like Slowloris?