'nobody' spam

May 3, 2006 6:54 AM

How can I determine exactly from where or how a server's email queue is being filled with outgoing spam from user 'nobody'?

Here's the basic set up:
Redhat 9.0
Apache 1.3.34
Exim 4.52
Cpanel - latest version
(note: I'm looking at about 1,500 of these set ups, so massive changes aren't going to be possible, including the use of phpsuexec.)

So the deal is, a spam complaint comes in pegging a certain IP address as the culprit. I match it up with the actual server and find a mail queue with 60,000 outgoing messages, 59,995 of which are spam. The outgoing address is [e-mail] (of course), since Apache runs as user nobody.

Occassionally, I can match up the timestamp of an email to an Apache log entry, showing me that a certain "contact us" script, or something of the sort, is being exploited. At that point, the"fix" is easy. But more often than not, especially when the cPanel installation has about 250 accounts, trying to find THE insecure script responsible for the creation of thousands of outgoing emails is like looking for a needle in a hystack.

Does anyone have any suggestions on how I can attack this problem more efficiently and productively?

posted by Witty to computers & internet (6 comments total)

The Apache log files don't tell you anything? I mean, one of your users has 59,995 hits to contact.php or something like that, and the other ones don't (if you're correct about the source of the spam).

Some quick processing of log files: run them through cut to just get the file being hit, then sort, and then "uniq -c", and then "sort -n -r" again - this will give you a sorted count of the hits in the latest log file, from most hits to least. Your culprit script should jump out real quick.

If the abuse is ongoing, try netstat.
posted by jellicle at 7:15 AM on May 3, 2006

Use suexec or cgiwrap to let your 250 users' CGI scripts run as those users instead of "nobody", and then the mail will start coming from those users instead of "nobody". Run PHP as cgi, not as a module, so it runs as those users too.

Also, use rate limiting on your MTA to prevent users from sending that quantity of mail to anyone, intentionally or otherwise.
posted by mendel at 7:25 AM on May 3, 2006

mendel - The problem is, these are client servers. So can't make wholesale changes like that, as much as I may want to.

jellicle - The suggestions you've made DO work in some instances... and you've pretty much hit on exactly what I do most of the time. One problem is, sometimes the "hacker"/spammer abuses a CC or BCC feature in a script, which means that log will only show that script being hit once, my 2 or 3 times at most. Then the guy comes back the next day and uses the script another 2 or 3 times, making it difficult to dig up.... like I said, especially when the server can have upwards of 500 domains or more.
posted by Witty at 7:40 AM on May 3, 2006

*maybe 2 or 3 times at most.
posted by Witty at 7:41 AM on May 3, 2006

Since you're looking for very large form submissions, you can grep through the logs for only POST requests, which should narrow things down a lot.

If you have mod_logio, you can include the "bytes received" in your log (which should make it easy to spot a hacker posting thousands of addresses to a BCC field). Unfortunately it looks like it's in Apache 2.0 only.
posted by mbrubeck at 4:17 PM on May 3, 2006

How about setting up a snort box to monitor the whole segment? That ought to pinpoint the malicious GET/POST requests that are exploiting the bad contact forms.

Alternatively, once you find a certain vulnerable CGI on one server, come up with a script that searches for it (perhaps with some fuzz factor) across all servers.
posted by Rhomboid at 7:11 PM on May 3, 2006