Is our Linux server compromised?

May 15, 2007 8:51 AM

CracketyCrack? Is our Linux server compromised? (Strange high-load Perl processes spawned by apache.)

Our Linux web server is continually bogged down by multiple high-load Perl processes spawn by the apache user.

The server is a dedicated host that only myself and friends use.

For regular cgi scripts "top" and "ps" report the script's filename as the command. However, these mystery processes are identified only as "perl".

There's at least one of these processes running at all times, and sometimes up to 15+ (consuming 70% of cpu). When killed they are re-spawned.

Could this be a botnet script (perhaps a spam mailer)?

Is there anyway to determine what script Perl is interpreting? Or how/why these processes are being spawned?

Distribution: Red Hat w/ 2.6.9-041221 kernel
Apache: v2.0.51 (Fedora)
Perl: v5.8.3

posted by stungeye to computers & internet (17 comments total)

lsof -p <pid>
posted by togdon at 8:53 AM on May 15, 2007

Thanks togdon!

It appears to be a web statistics script running amok.
posted by stungeye at 9:11 AM on May 15, 2007

If it's compromised, it's fairly impossible to trust any binary on the system (including lsof, perl, ls, ps, top, bash, ssh, etc.). I'd look at a process trace, too, like from strace or gdb.

What does 'ps -ef' show you as far as commandline?
posted by kcm at 9:14 AM on May 15, 2007

With ps -ef, the apache process which is spawning perl shows:

apache 10184 1 0 02:24 ? 00:01:05 /usr/local/apache/bin/httpd -DSSL

This path does not exist (i.e. There is no /user/local/apache/)

[The full ps output]
posted by stungeye at 9:21 AM on May 15, 2007

Oh, and lsof shows access to files of this form:

perl 10184 apache 74w REG 3,7 198565 21055161 /home/httpd/vhosts/stungeye.com/statistics/logs/access_log

perl 10184 apache 37w REG 3,7 18577816 20971799 /home/httpd/vhosts/stungeye.com/statistics/logs/error_log

This is repeated for all the sites we are hosting.
posted by stungeye at 9:27 AM on May 15, 2007

apache 10184 1 0 02:24 ? 00:01:05 /usr/local/apache/bin/httpd -DSSL

This path does not exist (i.e. There is no /user/local/apache/)

You've written usr as user there. Make sure you didn't actually look for /user/local/apache.
posted by mendel at 9:37 AM on May 15, 2007

Oppps. That was a typo.

/usr/local/apache does not exist.
posted by stungeye at 9:40 AM on May 15, 2007

if you have physical access to the machine:
a. download and burn insert (or other favorite forensics livecd)
2. boot said livecd, run rkhunter (and/or chkrootkit) and see what you can see.
posted by dorian at 9:43 AM on May 15, 2007

dorian: No physical access, but perhaps we'll try a script like this.
posted by stungeye at 9:50 AM on May 15, 2007

the fact that /usr/local/apache/bin/httpd shows as running when you say there is no /usr/local/apache/ directory, is fairly disturbing.

altho from your ps output it looks like you are using plesk, which I have known to run huge perl processes at times.

anyway, if you do suspect the machine has been compromised:

the problem with running that script (or any other) directly atop the potentially-compromised os, is as kcm said, that none of the standard utilities or libraries can be trusted. and any script or program for detecting intruders is likely going to depend on one or more of those utilities or libraries, so the output of said script cannot be trusted 100%.

and even if you are able to detect a specific intrusion, it's generally not possible to eradicate it 100% (or even to have the knowledge that you have or have not done so) ... generally the most common (and often smartest) solution in these cases is:
save all your content/data and nuke it from orbit.
posted by dorian at 9:58 AM on May 15, 2007

Pull the ethernet cable out now, that fake apache line looks like a poorly conceived rootkit, as does the
apache 6198 1 0 May14 ? 00:00:01 /usr/bin/init.d/started line.
posted by cmonkey at 10:18 AM on May 15, 2007

If that web stats script is an old version of AWStats, you may be in trouble. I've had a server be comprimised via AWStats before. Luckily, nothing came of it. AWStats is not exactly the the most secure piece of software.
posted by easyasy3k at 11:06 AM on May 15, 2007

Um, maybe this is obvious, but can you look at its network traffic from somewhere further up on the network (using etheral, tcpdump, or whatever) and see if it's spewing out spam or doing other things it shouldn't be doing?

That would be my first concern; you want to find out if it's doing something bad before you get a six-digit bandwidth bill at the end of the month...
posted by Kadin2048 at 11:18 AM on May 15, 2007

While it's true that you can't necessarily trust lsof, ps, etc., and that the whole system could be compromised, I've had a couple of my servers infected over the years and always had it secured and back online within 30 minutes of me finding that it was infected.

My point is that wiping the system and starting from scratch isn't always necessary; you can often work out what shouldn't be running, kill it, delete it, figure out how it got it, fix that hole, and get back online.

Another fun tool to play with is netstat --inet, which lists open connections. Often, a compromised machine would have a zillion IRC connections.
posted by fogster at 11:24 AM on May 15, 2007

I didn't mean to sound like I was tooting my own horn; in fact, I'm not that great with server security, as evidenced by me having had several compromises over the past few years. My point was more that even I was able to get the machine back online in no time, without reinstalling.
posted by fogster at 11:28 AM on May 15, 2007

netstat does indeed reveal some suspicious irc activity:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 u15180621.onlineh:38478 194.247.177.78:ircd ESTABLISHED
tcp 0 1 u15180621.onlineh:47753 parkwebwin-v02.pro:smtp SYN_SENT
tcp 5140 0 u15180621.onlineh:35084 208.98.8.59:40001 CLOSE_WAIT
tcp 0 0 u15180621.onlineh:46758 200.141.229.10:ircd ESTABLISHED
tcp 0 0 u15180621.onlineh:43833 maico.intec.edu:40001 ESTABLISHED
tcp 0 0 u15180621.onlineho:3784 h218.75.140.67.ip:60072 ESTABLISHED
tcp 0 0 u15180621.onlineho:3784 mn-10k-dhcp1-4960:32961 ESTABLISHED
tcp 0 0 u15180621.onlineh:44762 maico.intec.edu:40001 ESTABLISHED
tcp 0 0 u15180621.onlineh:41684 servidor1:x11 ESTABLISHED
tcp 0 0 u15180621.onlineho:fido 85.8.354a.static.t:ircd ESTABLISHED

posted by stungeye at 12:02 PM on May 15, 2007

Yes your machine is cracked.

Last time I had this happen to me I sniffed the IRC session for traffic, collected a list of nicknames of the people using my machine as a BNC and entered the channel they were sitting on (some warez chanel on undernet) and killed their sessions as I called out their name in the channel.

Hey! SirHacksalot!
* Godguy points gun fingers at SirHacksalot
* SirHacksalot has quit IRC (Connection reset by peer)

etc...

I believe ours was caused by a bad apache perl script also...
posted by puddpunk at 1:05 PM on May 15, 2007 [1 favorite]