Linux is biting me.

June 25, 2010 8:39 PM

What could have caused all my linux log files to vanish?

I was just checking an unrelated issue on our web server and the files that used to be in the /var/log directory (except for rpm logs) are not there. None of them. No maillog, cron, system...any of them. I am not exactly sure when this occurred but I also noticed I stopped receiving Logwatch messages when searching my inbox for the nightly emails.

Any ideas?

posted by thorny to computers & internet (7 answers total)

hackers.
posted by delmoi at 8:55 PM on June 25, 2010

could be a change in file mountpoints, perhaps if you had /var or /var/log as a separate partition, but most likely clumsy hackers.
posted by Geckwoistmeinauto at 9:10 PM on June 25, 2010

When hackers erase their tracks badly, they do it by deleting the logs (instead of surgically editing them), thus revealing that they've been there. Unless you have some kind of rootkit monitoring, perhaps it is time to wipe and reinstall this machine just to be safe.
posted by davejay at 9:36 PM on June 25, 2010

Look in /root/.bash_history for recent commands done in bash or ssh session. Also check other user's .bash_history . Check your default log configuration or any configuration changes in /etc folder, such as do a "sudo find /etc -mtime -1 -print" to see what has been modified recently. Do a "ls -lt /etc/* |more". Check modification dates and times. Have you been making any confuration changes? Installing new software? Disk space OK?

Even if the log files were deleted, they should almost immediately be recreated by programs and services. If it is straying empty even after a reboot, log files can't be written.

Could be a hacker trying to cover his tracks.
posted by nogero at 9:40 PM on June 25, 2010

I think I am going to go with the hacker possibility since the syslog configuration file was wiped from /etc. I restored it and the logs started immediately. Thanks for the info. Any other advice is appreciated.
posted by thorny at 9:42 PM on June 25, 2010

consider the system compromised and reformat immediately. Disconnect it from networks if you haven't already. Perform a full security audit of any systems that could have been accessed from the web server.
posted by An algorithmic dog at 10:05 PM on June 25, 2010

For fun and future reference, make a DD image of your system before you nuke from orbit. This has the benefit of keeping the 'evidence', so you can learn what happened and prevent it in the future. I used to work at a place where hacked systems were just restored from backups, and they would keep getting rehacked. People had no idea why it kept happening.
posted by Geckwoistmeinauto at 5:05 AM on June 26, 2010