Can you help the hacked and hopeless? (Hacked site? Email?)

May 13, 2015 6:54 PM

Hostgator says my site has been hacked but can't find where or how, and after two days and numerous calls, we're lost. Can anyone explain in teeny, tiny words what I should do next?

I'm not sure what's important for anyone to know, so I hope this isn't excessive.

Background: I have a small website for my business. My website's domain is (unsurprisingly) the domain name portion of my email address. I've had them both almost 15 years, though we upgraded the site to Wordpress about two years ago, and moved to a new host (Hostgator) last summer, when my formerly reliable (small host) shut down.

Problem: As of a little over 48 hours ago, I became absolutely FLOODED with "Mail delivery failed: returning message to sender" messages for email sent to strangers, allegedly from my email address. (All emails have brief gibberish content, at least one profanity, misspellings and a link. All of the IP addresses noted in the bounce emails are from outside of the U.S.) In addition, I'm getting messages from Hostgator letting me know that many other messages didn't go through because I've exceeded my 500 messages/hour limit. How flooded? 27,000 of these between lunchtime yesterday, for example, and lunchtime today.

My immediate thought? Hacking. Logged into WordPress, but there was no sign anyone had successfully EVER logged into my site except my web guy (a close friend) and myself, as I recognized the IP addresses immediately. Ran Securi -- no malware found. Nothing's seems wrong on my site. Logged into my Hostgator C-Panel. No sign that any mail was being sent from my account's web mail (a thing I never use, as everything shoots straight to my Outlook on my Mac).

First contact with Hostgator, they had me change my email password, which I did within 2 minutes of starting the call and within 5 of discovering the problem. The guy was certain this would solve the problem, even though I kept him on the phone and noted I was still getting bounces, though more slowly. He did not put in a ticket.

I waited a few hours, called Hostgator back and was on the phone for hours and hours, until close to 4 a.m. my time. The guy obviously had very little experience, ran one automatic scan which he couldn't really parse, and said I'd have to put in a ticket and I'd have to wait a while -- possibly weeks -- for Security to get around to reviewing my site. He spent a lot of time putting me on hold, coming back, and offering no suggestions. He said he was unable to pass me to someone higher up.

Appalled, I sent a message to my web guy. He spent yesterday (while I was on the road) checking everything, and he found no malware, and no sign of anything. This morning, he tried changing all of my email passwords (so, for the account with the problem, this is the second time it was changed in 36 hours). No change.

Got back into town today (was gone 24 hours), contacted Hostgator again (because I've heard nothing from my ticket). This third guy actually did a variety of tests, found no malicious code, debunked his theory that it was due to Contact Form 7, posited and then debunked that it was actually lack of security on my personal computer or my web guy's (by changing my password yet again, from his end, without me getting to see it), and had me delete my entire email account (which I did, once I sorted six ways from Sunday to make sure I'd read every real message and determined they could all go, all 30 out of 27,000+).

And then he had me recreate the account in the C-Panel. No benefit. I'm still getting thousands of these bounces every hour. Unless I check mail every ten minutes and sort by sender (looking at the first page, and then last page of the ugly, awful webmail platform) to find my *real* messages, mail is largely unusable.

Emails are still being sent from (or at least under) my account. He said he could see many failed attempts at sending (not sure where he was seeing this -- some back end thing?) and many more emails sending, but could not figure out how or from where. He said I had three options:

--Get Google Apps so that my email will be filtered through Google, I will not have to worry about the 500/hour limit for sending (so I can actually send clients emails!) and the spam will be swallowed up, but it's not clear that spam won't still be going out under my name at all, and it's definitely not clear that this "solution" impacts my being hacked at all.

--Wait for Hostgator Security to get around to following up on my ticket. This has been going on since Monday, I've heard nothing, and a previous ticket for a merely annoying inbound spam issue took 3 weeks to yield a dismissive, boilerplate reply. And they can't tell me IF Security will be able to help, or how much they might charge. Seriously?

--Use Site Lock, which they promise will start working to fix the problem right away. However, before spending a penny, I started researching, and found this review and the attached comments, and pretty much everything I'm finding now debunks them as more of a problem than a solution.

TLDR: My site "appears" fine, but spam email is flooding out through my account and yielding tens of thousands of bounces back to me, obliterating my email account and making it near-impossible for me to use my professional account to send or receive email. My business is small. One-person small. I can't afford to make a mistake and spend money on a solution that doesn't stop this ASAP.

Thanks for any guidance.

posted by The Wrong Kind of Cheese to Computers & Internet (11 answers total)

Can you post the complete header of one of the bounced emails so we can see the path?
posted by Mac-Expert at 7:11 PM on May 13, 2015 [1 favorite]

Emails are still being sent from (or at least under) my account. He said he could see many failed attempts at sending (not sure where he was seeing this -- some back end thing?) and many more emails sending, but could not figure out how or from where

Logs. Either there's a web access log that shows a metric shit tonne of POSTs to your site that are generating outbound mail, or there's a mail log that shows a metric shit tonne of SMTP traffic because you've got an account under your domain that has had the credentials guessed or hacked. Hostgator should be able to tell you which. This is basic stuff. If they can't tell you which, then get thee to another host.

My best guess would be that it's on the web side, because a handful of WP plugins recently had arbitrary upload exploits which allowed arseholes to upload PHP files that then mail out spam when they're hit with a POST. (Contact Form 7 was one of them, but there were others.) Sucuri should identify them, but it's not 100% foolproof. OTOH, your WP core files might have been injected, in which case you're mostly fucked.

(This is all assuming that Hostgator has proper privilege separation on its servers. If it doesn't, there's the chance that you're fucked via another user on that box, but that's rare even among commodity hosts these days.)

If it's on the web side, your options are more or less "restore from a trusted backup", with IP-limited access until you get that shit patched, or "nuke and pave" with a fresh WP install and selective, audited restoration.
posted by holgate at 7:12 PM on May 13, 2015

Response by poster: Remember what I said about teeny tiny words? I'm not sure I know what you mean by "complete header," Mac Expert, but guessing, here's the source code for one of the many messages with my email address/domain hidden:

=================

Return-path: <>
Envelope-to: [My email address deleted]
Delivery-date: Wed, 13 May 2015 20:54:58 -0500
Received: from mailnull by gator3031.hostgator.com with local (Exim 4.82)
id 1YsiMU-0004WZ-QI
for [My email address deleted]; Wed, 13 May 2015 20:54:58 -0500
X-Failed-Recipients:
Auto-Submitted: auto-replied
From: Mail Delivery System
To: [My email address deleted]
Subject: Mail delivery failed: returning message to sender
Message-Id:
Date: Wed, 13 May 2015 20:54:58 -0500

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

[e-mail]
Domain [My domain deleted] has exceeded the max emails per hour (500/500 (100%)) allowed. Message discarded.

------ This is a copy of the message, including all the headers. ------

Return-path: [My email address deleted]
Received: from [180.254.240.138] (port=50392 helo=[10.100.100.99])
by gator3031.hostgator.com with esmtpa (Exim 4.82)
(envelope-from [My email address deleted])
id 1YsiMU-0004U8-Bf
for ; Wed, 13 May 2015 20:54:58 -0500
From: "[My name]" [My email address deleted]
To:
Date: Thu, 14 May 2015 04:54:59 +0200
MIME-Version: 1.0
Subject: cciiaalliisss BETWEEN against
Message-ID: <6>
Priority: normal
X-mailer: Pegasus Mail for Windows (4.70)
Content-type: text/plain; charset=windows-1251
Content-transfer-encoding: 7BIT
Content-description: Mail message body

BEST DRUGS matter iphone friend maiden viagiris cock cciiaalliisss SUMMER thrust https://www.google.us/?qgn_sh=jg&uz=xlRIs4kknsx4tRaqW#&q=d664y34bzdz0t4thciauwhk&btnI=qupx&Sncd

=================

holgate, I appreciate what you're trying to say, but it's like you're talking cardiothoracic terminology to a seven year old. The third guy was able to see, somewhere, in real-time, as there were messages being sent from my address, but they weren't being sent by me, and he could not see where they were coming from beyond, "Oh, Iran! Turkey!" Again, these "front line" guys aren't giving me anything useful to report to my web guy, they will not pass me upward to a supervisor, and they keep saying I have to wait for Security to look for my ticket.

If you know how to walk me through a C-Panel to try to see these logs myself, I'm game. But I have no idea what I have access to.

If my "WP core files" were injected, how would I know, if my web guy doesn't see anything?

I do know that Contact Form 7 was disabled by my web guy 12 hours before this last call, and the messages keep going out.

And I'd be happy to move to a new host, but I feel like finding the problem so that I'm not carrying a borked site elsewhere is a pretty important step, if possible.
posted by The Wrong Kind of Cheese at 7:35 PM on May 13, 2015

Okay, from those headers, that looks like an email hack going through Hostgator SMTP. (From Indonesia, fwiw.) There's nothing obvious to indicate that it's coming from the website.

Do you have any additional email accounts enabled for your domain that aren't your primary email? Are you quite sure that you don't have any malware on your system that might be snooping your new credentials?

I assume that Hostgator requires authenticated SMTP (if they don't, and they're somehow allowing unauthenticated inbound, then whoa) so there's something that has credentials.
posted by holgate at 7:44 PM on May 13, 2015

Response by poster: holgate, I sent you MeMail to get clarification on what you mean, but unless I hear that I've totally misread you, I do have other (but entirely unused) email accounts on set up on that domain, and since the credentials were changed once by my webmaster (without sharing the details with me) and once by Hostgator (ditto), both in completely different parts of the country, I couldn't even be snooped by a bad guy with powerful ESP.
posted by The Wrong Kind of Cheese at 8:21 PM on May 13, 2015

From the headers it seems something is actually using your email account credentials to send emails out on your behalf through your hosts mail server. It does not appear to be coming from your website though . It's as if someone or something stole your credentials and is using them to authenticate to your mail server. Which absolutely shouldn't be possible if you changed your password unless whatever machine youve been on while you've logged into mail is compromised with a key logger or other malware.
posted by TestamentToGrace at 8:25 PM on May 13, 2015

Response by poster: TestamentToGrace, we know it's not that because the credentials have been changed three times, from three separate computers:

--once by me in TN
--once by my web guy in GA
--once by Hostgator wherever in Texas the rep was

So, we know it's not a local issue. How do we get Hostgator to see/look into what you're saying in your first sentence, which is what I'm inclined to believe is the case? Magic phrases? Thanks!
posted by The Wrong Kind of Cheese at 8:47 PM on May 13, 2015

Best answer: I've replied on MeMail, but the gist, for public consumption: when you send email through a SMTP server that requires authentication, the credentials that permit you to send email are separate to some degree from the address that shows up in the From: header. Authentication is the key to open the door, but once the door's open, others can step inside.

It's possible that another email account for your domain has had its credentials hacked. It's not impossible (but unlikely) that Hostgator has a shonky mail server where [e-mail] has had his credentials hacked but allows mail to be sent out from around the world via its server as [e-mail] -- basically what's known as a joe job, but with the added ugh of authentication.

Hostgator's mail logs should show which user is authenticating to send out these emails. That's what you need to ask for.
posted by holgate at 8:58 PM on May 13, 2015 [3 favorites]

I suggest to take your business away from Hostgator.
At least host your email elsewhere. For my sake a $50 Google Doc's account. Change the DNS records enable 2 step verification on your Google account and call it a day.
posted by Mac-Expert at 10:50 PM on May 13, 2015

Response by poster: To update you lovely people, the problem was resolved around 3 a.m. I created a slight PR issue for them by replying to and tagging a comment Peter Shankman made on Twitter regarding customer service. Peter's got 166K followers (to my lowly 5000+), and this time, Hostgator's support team got back to me right away.

Within an hour, their customer support people were actually reading the 9 hour's worth of phone notes the reps had included in the ticket, and offering temporary solutions (at no cost, vs. the ones the front-line reps had offered, which all involved paying), but saying (as if I should be excited) that I should hear from someone "in a day or so." Another complaint to the Hostgator Twitter account, describing the full experience, and within perhaps 20 minutes, oh, lookie, the Security team was on the case and had solved the problem.

Interestingly, by which I mean annoyingly, the response was that obviously I caused the problem:

==
We have completed our automated scans of this account and found our scans were not able to detect any malicious content.

Upon further investigation it was determined that an email account under your control has been exploited. It appears that your email password has been compromised which allowed these messages to be sent. We have updated your password(s) in order to prevent this from recurring.
==

In response to their assertions that it must have been caused by poor security on my computer or weak passwords, I reiterated the facts shared with all of you, including the fact that the passwords were changed by four parties (including their own rep), on three separate computers, while the other two parties were never granted access, and in fact, when they changed the passwords at Hostgator's end the first time, I not only didn't have access to that change, I didn't have access to my C-Panel.

As for their assertions about weak passwords, I could only note that in every case, they were generated by password generators, and two of the cases, by Hostgator's OWN password generator, and that their system had ranked my own selections at the highest possible level of strength.

So, yes, the search is on for a new host. Thank you all!
posted by The Wrong Kind of Cheese at 8:45 AM on May 14, 2015

Well the few times I had to deal with Hostgator it was shaky at best.
Very concerning that they only kick into action after calling them out in public.
If I were you I would take my email and possible domain name hosting away from them.
I rely on Hover.com for domain name and IMAP email hosting. If you want more go with Office 365 or Google doc's. However with Google you have to keep in mind that this servers only works well in a a web browser...
posted by Mac-Expert at 5:00 PM on May 14, 2015