Help me figure out how my site got hacked - and what to do...

November 30, 2010 10:03 AM

My website got hacked the other night; a piece of code was inserted that generated spam. I'm technical enough to have found the offending code, and I simply removed it, but I'm baffled as to how the code got inserted in the first place…. (more inside)

I practice extremely secure password policies - I use a high-security password, never edit the site from anywhere but home, and don't have anyone else with FTP access. It doesn't seem that any additional tampering took place. (I've changed the passwords anyway.)
More detail: - I run both a personal website and a hosted Wordpress installation, which launches off a subdirectory. - I own several domain names, which redirect to my main site. - The code was inserted in one of the folders that contained redirect information for one of my domains.

Can anyone help me, either with clues as to how this happened, or questions I can direct to my ISP that might track it down? My expertise on this is fairly limited.






(I know, maybe I just need to admit that I was hacked…)
I've posted the suspect code as a PDF here:

https://docs.google.com/viewer?a=v&pid=explorer&chrome=true&srcid=0B9VLFAT0knWiZDlmYjNkZDItNzg4NC00ZTBhLWIxMGItYTRiNTAxYTM3MzY2&hl=en

(If there is a better or more community-friendly way to make this request, please advise me. And thanks in advance for the help.)

I just discovered another clue: there's a Wordpress installation - not mine, which resides in another directory - in the above-mentioned redirect directory. It contains a file, which appears to be the key to allowing access, called wp-app.php, that appears to call up a console which allows full access to the site. (I am hesitant to post the link in a public forum, though I am happy to share it with anyone privately.)
The questions now: - What do I do? Remove the installation? - How did the access occur?

posted by soulbarn to computers & internet (18 answers total)

You're looking in the wrong place. The answer lies in the web server logs. Can you post or send a link to the logs?
posted by bfranklin at 10:09 AM on November 30, 2010

They probably took advantage of a security hole in Wordpress. Is the hosted Wordpress installation up-to-date? I agree with bfranklin that the logs will show you how they did it.
posted by zsazsa at 10:10 AM on November 30, 2010

There are a variety of ways to attack a web site, many that don't really require that the machine be fully compromised, such as cross-site scripting (basically, it's usually inserting a malicious piece of javascript and/or CSS into a web page via "normal" mechanisms, i.e. a comment or something like that), SQL injection (which depending on the backend database can sometimes get the attacker the ability to run arbitrary shell commands), attacks on the web server itself (buffer overflows in URLs, header handling, etc). The last and worst is of course the machine you're hosted on being compromised.

If they weren't able to clear their tracks, there may still be some hints around on the system as to where the breakin occurred. Someone with some skill in such things can probably poke around and tell you what happened. I spent a pretty good part of my time a few years ago diagnosing such problems and (attempting to) close holes in products. It's a difficult task to say the least.
posted by RustyBrooks at 10:10 AM on November 30, 2010

What you're dealing with is known as a "web shell", which either gets fetched remotely with wget or curl, if they're available, or gets uploaded directly.

Poisoning a single php file to echo spam recursively through a site container, then running it, is enough to muck up a site. The logs will tell the story -- look for references to that wp-app.php file -- but it sounds like some kind of XSS vulnerability or a lack of proper filtering for uploads. (One of the more frequent routes recently has been phpThumb.) It could also be a DB exploit, but that's slightly less likely.

there's a Wordpress installation - not mine, which resides in another directory - in the above-mentioned redirect directory.

It's under the control of someone else? Or the host? Ask your hosting provider what kind of container segregation it uses. If the server's using Apache and mod_php, and the webserver has group write permissions under www-data or similar, then your container is only as secure as the least secure container on the box.
posted by holgate at 10:25 AM on November 30, 2010

I had the same problem, repeatedly, a while ago. I've got an old Drupal install and a Wordpress install that I keep, but it turns out that it was a set of very simple web pages that used some PHP includes for headers and footers. Someone was using URL munging to include in external scripts that created a vulnerability on my site, allowing them to insert linkspam.

I wound up redoing that set of pages as a Wordpress site. No such problems since.
posted by adamrice at 10:36 AM on November 30, 2010

I can't seem to find my log files - my provider says they're in a directory called "STATS," but there is no such directory (I'm logged in via FTP to the root directory, with invisible files set to show.)

Is there another place I should look? My provider's first-tier tech support is useless; they insist I have a "virus in internet explorer" or spyware (I use a mac.)

I guess I have to get the provider to escalate and get them to figure this out.

My Wordpress installation seems to be up-to-date - at least, when I use the Wordpress admin console, it shows 3.01; however, my provider's in-house control panel shows 2.7. I was told that if I'm updating via Wordpress, that's normal.

Any hints on securing my WP install? Disabling comments?

thanks for the help.
posted by soulbarn at 10:44 AM on November 30, 2010

I had a similar problem about a month ago. It turned out that this was the chain of events:

1. I unknowingly visited an infected website.

2. My computer (Vista fully patched, Firefox, Avast Pro theoretically vetting website interactions for me) was infected.

3. The malware discovered that I had stored FTP passwords in my FTP program. (I use WinSCP, but every FTP program is vulnerable to this.)

4. It grabbed those FTP passwords and sent them out to a collection of waiting zombie computers on the internet. Basically passing my FTP login information out to the Russian mafia or whatever.

5. One of those zombie computers used the FTP login info to log into the website and append the bit of infection code onto several (but not all) of the HTML and PHP files on that website.

What you want to do is:

1. Delete all stored FTP passwords. Never store an FTP password in your FTP software again. Sorry.

2. Change all your passwords to every site which you formerly had been storing the FTP password for. (Does that sentence parse? Basically if you stored the password for it, then you can assume it's been compromised, even if it hasn't been infected yet.)

3. Run a heavy-duty virus scan. I used AntiMalwareBytes, which found the problem. Luckily, the malware was trivial to remove.

4. If anyone else has the FTP login info for your site, instruct them to do the same.
posted by ErikaB at 10:45 AM on November 30, 2010 [4 favorites]

In addition to the above advice, for the future, take the advice from Matt Cutts here and lock down your wp-admin directory using .htaccess.
posted by Gator at 10:57 AM on November 30, 2010

I practice extremely secure password policies - I use a high-security password, never edit the site from anywhere but home, and don't have anyone else with FTP access

Of course, FTP is completely insecure. Your password is transmitted via unencrypted text. While you could theoretically snoop the traffic, it's an unlikely attack vector. Just to be safe, you should look into using SSH keys. For extra security you can encrypt the password with a password.

More likely is that your ISP has configured apache to give all customers equal rights to all files. Simple to set up, easy for users, and makes hacker's life quite simple. Want to hack a website on shared hosting? Get an account with the ISP, or hack an existing customer's ancient webapp.

Or it's just yet another wordpress vulnerability. Basically, you need to update rapidly, because automated tools will scan the internet looking for old, breakable installs.
posted by pwnguin at 11:05 AM on November 30, 2010

Oh, and don't forget the WordPress wiki on Hardening (securing) Wordpress.
posted by pwnguin at 11:06 AM on November 30, 2010 [1 favorite]

And for the hat trick of preview failures, the three places I'd look for logs are /var/log/httpd/, /var/log/apache2/, and /home/soulburn/ (or whatever your account name is). It's possible hackers deleted your logs, which makes it harder to trace at a price of clearly having been hacked.
posted by pwnguin at 11:10 AM on November 30, 2010

EricaB's tale of woe reminds me of something I saw a few months back: malware grabs FTP creds, uses them to poison a PHP header file, visits home page to spread spam further. The web shell comes later, to see if it's possible to exploit other vhosts or the entire box. And the presence of that stuff serves as a calling card for other bots and script kiddies.

If the spam include features a link, searching for it will probably point to the variant of the malware associated with it, if it follows the same pattern. It's especially sneaky, because it comes in via FTP rather than XSS: it often only gets noticed when Google flags up the site from search results.
posted by holgate at 11:47 AM on November 30, 2010

You want to ask the support monkey for the web server's access logs for your account. No need to mention pages, browsers, spyware, or anything that is typically touched with a mouse. Just web server logs. "If you don't know what web server logs are, can you connect me with someone who does?" Don't waste your time teaching.

As to how to understand how this happened, consider that the offending code that you found was located in a place that is accessible to the user that wordpress (or your webserver) is running under.
posted by rhizome at 11:52 AM on November 30, 2010

We had similar issues across a wide range of hosted client websites a few years ago. We never found a definitive source, but narrowed it down to two possible attack vectors... One was FTP (in that perhaps someone had compromised our FTP passwords through sniffing or malware/trojan), the second was a security vulnerability on our webhost.

While I can't be totally sure (we were never able to prove one way or another) I believe it was actually an issue on our host. Even after removing the offending edits and changing FTP passwords and using different systems to login the attacks/modifications kept occurring.

We eventually moved to another webhost and the problem stopped.

I think in our case someone had gained access to a more privileged account on the webhost that had access to all our chroot'ed FTP directories and deployed an automated script to insert code in specific files/places. The only alternative I could come up with was that someone upstream of the webhost (or even on their server) was sniffing FTP passwords, as the issues continued even on sites that we never accessed from our standard work network or computers.

Unfortunately in a hosted situation without access to full server logs it's virtually impossible to discover the original intrusion point. While the PHP web shell is probably how things were modified, it's hard to know how it got there initially.

I now run my own VPS server rather than purchasing provisioned hosting. At least this way I can be responsible for all aspects of the server. It's probably no more secure generally, but if something happens I am in full control of the server and everything on it.
posted by sycophant at 12:10 PM on November 30, 2010

you need to be using SFTP if at all possible, most apache-based hosts offer this and most ftp clients support it.

that stats directory is probably an awstats install, you'd want the "raw logs", they probably assumed you just want a measure of traffic. Raw log files will be rotated out over time, so you'll see the most recent with no added number, and the oldest will be access_logs.4 or access_logs.4.tar.gz.

You'd want to pay attention to funny looking GET traffic, or large amounts of POST traffic. When you find the attacked page, do another search in the logs based on that IP, and you'll get an idea of how/if they probed your site.

there is no shame in changing your passwords and moving on though, you might not ever find anything useful. you could even move on to wordpress.com and not even fool with the server stuff.
posted by yeahyeahyeahwhoo at 12:15 PM on November 30, 2010 [1 favorite]

Note that it can be very hard to detect these things from the logs, though it is probably possible. One possibility is that an old install of some other php software (for instance) was insecure and allowed someone to install a shell script (this wp-app thing) on the server, which then will allow them to write to any directory that is writable by the user running the web server. You can sometimes detect this by looking in the logs for suspicious patterns of POST that don't correlate with (hopefully the few) scripts you are running that use POST, and that involve the same IP over time.

I had this happen on a server (that I don't administer) because of an old, insecure mediawiki installation that had actually been deleted but was around long enough for someone to detect the vulnerability. One thing to be careful of is that you remove all hacked shell scripts, because if they can get one on the machine, they can get several. In my case, there was a main one that someone was using to install spam scripts, and there was a backup that they used to set up the main one but otherwise weren't using, so it was a lot harder to find in the logs. It is possible that the machine was compromised a while ago, before you did the most recent wp update. If you don't have full cooperation of the sysadmin, you can't really solve this, most likely...
posted by advil at 12:17 PM on November 30, 2010

Just to be more concrete, the shell installed on the server in question was called c99madshell, and every command in it results in a POST. So I was able to (eventually) find lines in the server logs like:

access_log.1273104000:ip deleted - - [11/May/2010:03:09:06 -0400] "POST /wikiBackup/LocalSettings.orig.php?stat=123 HTTP/1.1"

that didn't have an obvious explanation, and looking at this file, I discovered that it was loading c99madshell in binary from another server and running it (off of the root directory of msn.com of all places, I think it was). Aside from the very first line, it looked like an innocuous part of the php code for mediawiki. I found this originally by tracing back to when the server started serving spam and looking right before then; luckily we keep enough logs that I could go this far back. (Otherwise I might not have found the backup install of c99madshell, which had been installed something like a year before this particular access, possibly even by a different person or bot.) I don't know whether this will work for every shell script, but it is one way to start. I also did a lot of cross-checking IPs as people suggest above.
posted by advil at 12:28 PM on November 30, 2010

Hi - this thread has fallen off the recent pages, but if anyone is still reading, I've finally obtained the log files, and I'd be grateful if those with expertise in such things could take a look…

thanks,

Dan
posted by soulbarn at 10:11 AM on December 2, 2010