Recover hacked VPS logs

August 2, 2012 9:30 PM

Is there any way to recover data from a VPS that suffered a rm -rf?

A kind, anonymous individual decided to solve my memory-storage problems on my virtual machine by way of rm -rf. According to the host, only the /dev and /proc folders remain. Fortunately, I have a server snapshot from the start of the month, so nothing significant will be lost.

However, I would really like to know how this kind, anonymous individual managed to execute the command. Is there any relatively easy way to recover enough data from a virtual machine being hosted by a third party, in order to reconstruct the logs at the time of the attack?

posted by kithrater to Computers & Internet (7 answers total)

Best answer: On my virtual host all of the "undelete" and computer forensics tools I tried appeared to work in the same way they would on a physical machine. Nothing seemed particularly easy, though; the free tools I was coming across all would give me large blocks of data I would have to sift through with a hex editor to try to find my deleted files. But that was several years ago and I didn't try any commercial tools.
posted by XMLicious at 9:40 PM on August 2, 2012 [1 favorite]

I'd write this off as a lost cause. Restore, patch/upgrade, and (if you're a business) consider investing in some security solution such as CloudFlare's WAF offering or whatever security-as-a-service company your hosting provider has a relationship with. It's possible they've negotiated a deal where at some added cost you can get centralized log management/analysis and/or network traffic monitoring, e.g. the way Rackspace does. You could also send all your logs to a Splunk instance on your desktop or whatever.
posted by Monsieur Caution at 10:20 PM on August 2, 2012

Start up the snapshot. Do not patch it. Configure it to send logs to another machine. Then change the MOTD to display a petulant response to the hackers and a warning that you know where they are so they'd better cut it out.

You'll have a log of the exploit in a few days.
posted by zippy at 2:25 AM on August 3, 2012 [4 favorites]

Deleted files aren't usually overwritten, often their directory entry isnt even erased, just marked as unused. So long as nothing is written to the disc subsequently you can crawl though the directory hierarchy and attempt to reconstruct the file structure. This is a lot of work however.

As to how they managed it? Either they guessed a login and password or they exploited a weakness in a web server (or whatever). What was the VPS doing? Did you stay patched?
posted by epo at 5:26 AM on August 3, 2012

To do this you would generally need a block-level image of the partition, taken from the standpoint of the host system. That is to say, as soon as you start writing things to the partition you lose forensic data, so you would need to get your host to give you a raw copy of the volume. That's not generally something that is done, but you might be able to convince them, depending on how big it is.
posted by Rhomboid at 11:39 AM on August 3, 2012

Best answer: It depends on a few things:

Was it electronically over-written/wiped after it appeared to be "rm -rf"ed?
What filesystem was used on the VPS?
How are logs managed by the VPS?

If it wasn't wiped by the overly kind anonymous guest, the filesystem is a widely supported filesystem (nothing exotic, like for example ZFS), and it hasn't been used or powered back on, you might be able to recover the log files. Some open-source and commercial forensic software products are capable of automatically rebuilding a deleted filesystem structure or if you feel like becoming a bitter, crusty, war-scarred sysadmin, you can rebuild the filesystem structure manually yourself after studying a book like File System Forensics Analysis by Brian Carrier or by doing a google search for "file system forensics".

If your log files were archived in a popular file format, then you could carve them out of the unallocated disk space using software like Scalpel, which is an open-source file utility for *NIX and Windows based OSes. For example, if your log files tarballed themselves after reaching a certain size, you might be able to carve any and all files that matched the header and footer of a normal .tar/.tgz file. You will get a lot of false hits as well.

If your VPS has a dedicated unique IP address, maybe the host can provide you logs of inbound/outbound traffic to your VPS before and after the liberation of space occurred, it may provide you some keywords to perform some internet searches to see who else may have been visited by your gracious guest and see what they were able to learn from their experience as an over-burdened host.

This would make an interesting exercise to learn more about computer forensics only if you have time dedicated to analyze the system. It will be a boring, tedious process and no where near as exciting as "The Movies" make it appear to be. Otherwise, since you have a restore available, I would make sure it's up to date on all security patches and move on.

Again, there is a lot to be done, this is just a single scenario based on the info you provided so far and I provided an ideal road to follow if it exists.
posted by nataaniinez at 3:31 PM on August 3, 2012 [2 favorites]

Response by poster: Thank you everyone for your answers. After talking further with the host and thinking what constitutes a fruitful use of my time, I've decided to apply the snapshot, and patch like a madman.
posted by kithrater at 6:05 PM on August 3, 2012 [1 favorite]