Am I hacked? What now?

January 2, 2006 2:37 PM

If you ran a Debian server and one day noticed that telnet and apt-get were segfaulting, how would you proceed?

I'm one of two admins on this server, and neither of us has made any changes in the last couple of months. This morning I noticed telnet and apt-get were wonky. Nothing unusual in the syslogs. It seems likely that someone has done something malicious.

How would you proceed from here?

jojopizza@askme:~$ telnet yahoo.com 80
Segmentation fault
jojopizza@askme:~$ sudo apt-get update
Segmentation fault
jojopizza@askme:~$

posted by jojopizza to computers & internet (20 answers total)

The other admin is currently working on running a vulnerability audit, fyi, just to see if we have any obvious holes.
posted by jojopizza at 2:38 PM on January 2, 2006

I'd assume I was hacked. Wipe and reinstall is simplest and safest. If you have access to the machine, boot from a trusted CD and scan all the packages to see what installed software is damaged/hacked, maybe you could repair it manually.
posted by Nelson at 2:39 PM on January 2, 2006

I'd also try running memtest to see if the RAM has gone bad. My gentoo box exhibited the same behaviour when one of it's sticks of RAM went south.
posted by slhack3r at 2:55 PM on January 2, 2006

There was a libc6 update recently... if the upgrade process didn't complete for some reason, that could cause that symptom. If essentially NOTHING loads, that's probably your problem... and that can be a very painful thing to recover from. I keep a set of system utility files compiled static for just such occasions.

Being hacked is also a likely possibility. With a good rootkit, you CANNOT detect ANYTHING amiss on the system while the system is running.... you have to boot from a CD or something and inspect the system that way.

If it's a crummy rootkit, sometimes lsof or netstat will show you that things aren't quite right, but you absolutely can't count on that.

Hackers suck. Good luckl.
posted by Malor at 2:56 PM on January 2, 2006

I'm also running the rootkit detector from http://sourceforge.net/projects/checkps/

Other suggestions welcome.

Yes, a wipe and reinstall is definitely on our minds.
posted by jojopizza at 2:58 PM on January 2, 2006

if you manage security well then its surely a hardware problem.
posted by libertaduno at 3:10 PM on January 2, 2006

I could write you pages upon pages on how to detect a rookit, but you're much better off finding someone you trust to check it out for you. Note this means that any passwords entered anywhere are compromised, and that you should immediately yank the power cord (yes, yank it, don't shut it down cleanly - think about it), image the drive, and then inspect the image or the drive in a sanitized environment.
posted by kcm at 3:12 PM on January 2, 2006

chkrootkit found nothing, nor did our external vulnerability scan... Hmm.
posted by jojopizza at 3:15 PM on January 2, 2006

Please try memtest this before you wipe everything. Here's a link to a gzipped ISO of a memtest bootable cd.
posted by slhack3r at 3:23 PM on January 2, 2006

I've found Memtest86+ to be even better, actually. Plus, a lot of Linux LiveCDs have memtest images to boot if you're going to go that route to check it out.
posted by kcm at 3:24 PM on January 2, 2006

Well, I have no physical access to the server, and I think I have to pay to have one of the datacenter guys do things remotely, so I can't run any bootable CDs.
posted by jojopizza at 3:30 PM on January 2, 2006

Debian has userspace versions of memtest that run: I see both a memtest86 and memtest86+ available via apt. While it's not as accurate to run a memory test in user space, it will avoid the nasty problem of not being able to access the hardware yourself. Give it a shot!
posted by Nelson at 5:01 PM on January 2, 2006

have you tried chkrootkit?
posted by cellphone at 5:52 PM on January 2, 2006

nevermind, missed your comment. good luck!
posted by cellphone at 5:53 PM on January 2, 2006

Can anyone think of anything OTHER than a rootkit that would be causing telnet, mysql, man, and apt-get to seg fault?
posted by jojopizza at 6:32 PM on January 2, 2006

Possible reasons I can think of are ram, corrupted filesystem/dying harddrive, malicious shell aliases/links, or replaced code (either a library or the program itself).

Have you run fsck? What filesystem are you using? Check for soft links, just to be sure, and use a different shell/check the env variables.

To isolate a libary problem, make a dummy C program and link it against various libraries and test it.
posted by devilsbrigade at 6:46 PM on January 2, 2006

Reinstalling is recommended. But a post-mortem to see what was exploited in order to firewall or patch that would be a good idea as well.
posted by NucleophilicAttack at 7:05 PM on January 2, 2006

Can telnet start without giving an address? Can you run apt-get without any paramers and get usage info? I'm curious if it's something busted in name resolution.
posted by xiojason at 9:24 PM on January 2, 2006

Turns out one extra dude had the root password after all, and tried to upgrade some libraries and broke stuff. We all spent a few hours panicking, but we figured it out. :)

Thanks for all the ideas, everyone!
posted by jojopizza at 12:02 AM on January 3, 2006

'turns out one extra dude had the root password' doesn't sound very safe to me, fwiw ;)
posted by devilsbrigade at 3:51 PM on January 4, 2006