Undelivered Mail Returned to Sender

May 29, 2014 1:25 PM

I'm getting a lot (like thousands) of Undelivered Mail Returned to Sender mails, how can I a) determine whether the spam mails actually came from my server and b) deal with all these return mails without having to sit here deleting them every few minutes.

Firstly, I'd like to be able to stop all this bounce mails coming in - there's been around 20k in the last couple of hours. Once that is under control, I'd like to determine if the mails actually came from my server or whether the email/return address was just spoofed.

What can I do? Its a debian server running postfix

posted by missmagenta to Computers & Internet (14 answers total)

What e-mail service do you use?
posted by alms at 1:38 PM on May 29, 2014

Response by poster: Postfix
posted by missmagenta at 1:43 PM on May 29, 2014

Firstly, look at some of the bounce messages - findone that includes the headers of when your system allegedly made contact with their system. Check your logs to see if you did at that time (pay attention to timezones). Your logs will be /var/log/mail.log* (if you've received 20k in the last few hours, you'll likely only care about today's log).

I guess some needed info is what level of familiarity do you have; do you know where your mail logs are? Do you know at a cursory level how to read these logs?
posted by nobeagle at 1:50 PM on May 29, 2014

To stop the mail coming in, Do the messages have a common subject? If you're using header_checks , you could put in a line like:

/^Subject: Undelivered Mail Returned to Sender/ HOLD 20140529 stop the deluge

This will route all messages with that similar subject to the hold queue instead of delivering. You can go through them later to possibly find any which might be needed, or if they're all not needed "cd /var/spool/postfix/hold && ls | postsuper -d -"

Note, check the headers, are the messages being generated by your own server, or are they coming from third parties?
posted by nobeagle at 1:58 PM on May 29, 2014

Response by poster: This is the message headers from one of the returned messages:

Received: from BN1PR05CA004.namprd05.prod.outlook.com (10.255.197.24) byBY2PR05MB966.namprd05.prod.outlook.com (10.141.221.154) with Microsoft SMTPServer (TLS) id 15.0.949.11; Thu, 29 May 2014 20:45:00 +0000
Received: from BN1AFFO11FD030.protection.gbl (2a01:111:f400:7c10::154) byBN1PR05CA004.outlook.office365.com (2a01:111:e400:400::24) with MicrosoftSMTP Server (TLS) id 15.0.949.11 via Frontend Transport; Thu, 29 May 201420:44:52 +0000
Received: from xxx.members.linode.com (*my server's IP*) byBN1AFFO11FD030.mail.protection.outlook.com (10.58.52.168) with Microsoft SMTPServer id 15.0.949.9 via Frontend Transport; Thu, 29 May 2014 20:44:52 +0000
Received: from lexwpfosv (unknown [41.58.85.160])by xxx.members.linode.com (Postfix) with ESMTPA id D22C2A4271;Thu, 29 May 2014 21:54:06 +0100 (BST)
Subject: mankind heighten
From: xxxxxx
Content-Type: text/plain; charset="us-ascii"
X-Mailer: iPhone Mail (9A405)
Message-ID:
Date: Thu, 29 May 2014 23:43:44 -0700
To: "[e-mail]"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0 (1.0)
Return-Path: [e-mail]
X-EOPAttributedMessage: 0
X-Forefront-Antispam-Report: CIP:109.74.193.169;CTRY:GB;IPV:NLI;EFV:NLI;
X-Microsoft-Antispam: BL:0;ACTION:Default;RISK:Low;SCL:0;SPMLVL:NotSpam;PCL:0;RULEID:


So [e-mail] is the email address the messages claim to be from and xxx.members.linode.com is my server.

Does this mean the message definitely originated from my server? I'm searching today's mail log but its 60mb
posted by missmagenta at 1:59 PM on May 29, 2014

Best answer: It's looking strongly like it came from your system.

Post the output of:

grep D22C2A4271 /var/log/mail.log

If there's a line like:

May 29 17:02:20 mail postfix/qmgr[2023]: 4009E28A944D: (stuff ) 250 2.0.0 Ok: queued as D22C2A4271)

Then post the output of:

grep "D22C2A4271\|4009E28A944D" /var/log/mail.log

... hopefully this will show the sasl authentication - but that should be in the headers.

If you memail me the IP address, I'll give it a quick look to see if it's an open relay.
posted by nobeagle at 2:05 PM on May 29, 2014

Response by poster: I tried those and got a screenfull of stuff but before that I did notice this in the logs:

May 29 21:54:09 li140-169 postfix/smtpd[13419]: A8D6FA4256: client=unknown[196.38.112.2], sasl_method=LOGIN, sasl_username=xxxx

So someone is logging in as the user to send the mails?
posted by missmagenta at 2:14 PM on May 29, 2014

Response by poster: I've changed the user in question's password but now the logs are filled with this and I'm not sure what it means

May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: imap-login: Login: user=, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
May 29 22:25:54 li140-169 dovecot: IMAP(xxxx): Connection closed
posted by missmagenta at 2:25 PM on May 29, 2014

That screenfull of stuff indicates that the message ID actually matches the message id's in your log files.

the sasl_user stuff indicates that unless you were in ZA earlier today, yes, your password for account XXX is compromised.

What you want to do is change your password. then you want to clean up your mail queue's (there might be a lot of messages in your active , incoming and deferred queue's ).

This might catch them: (assuming you're running bash as your shell)

cd /var/spool/postfix
for i in active incoming do
cd $i || break 5
grep -m1 "Authenticated sender: xxx" * 2> /dev/null | cut -f 3 -d ' ' | postsuper -h -
cd .. || break 5 
done
cd deferred || break 5
grep -m1 "Authenticated sender: xxx" ?/* 2> /dev/null | cut -f 3 -d ' ' | cut -f 2 -d/ | postsuper -h -
postfix reload

On my systems* (freebsd and debian) that should work. Assuming that you replace xxx with the user. This won't delete anything, but it will put everything into /var/spool/postfix/hold for you to check out later.

Then rerun the commands (except for the postfix reload) again; because there might have been some processes still allowing the relaying of messages after you changed the password, but before you ran the reload.
posted by nobeagle at 2:27 PM on May 29, 2014

Response by poster: I already deleted all the mails in the queue and changed the user's password.... seems to be working so far.

Anything else I can do to prevent this in the future?
posted by missmagenta at 2:34 PM on May 29, 2014

Best answer: Install fail2ban with a recipe to watch your SMTP, IMAP and POP logs for brute-forcing of the user passwords.

If you have a system that stores user passwords in plaintext, run a password strength tester on them and change the weak ones.
posted by tomierna at 4:15 PM on May 29, 2014

In the near future, be prepared for Google, Barracuda, and other blacklists to start bouncing legitimate mail from your server -- had this happen last year, compromised password sent a bunch of spam from that account, and even six months later we still had one business who bounced everything sent to them. Mostly, we were able to resolve it, but it required going through whatever reporting process each blacklist needed to get removed from their list.
posted by AzraelBrown at 7:18 PM on May 29, 2014

Response by poster: Thanks guys, I actually already have fail2ban but it wasn't enabled for postfix, I've been through the config and made it check/block a lot more stuff!
posted by missmagenta at 10:42 PM on May 29, 2014

Going forward, some of the policy add ons may let you put harder limits on accounts (such as number of IP's accessed within a time frame). As I have about 0.5% of my accounts compromised yearly I have a 10 minute cron examining the last ~1 hour worth of log (specifically looking at webmail access and sasl based authentication from IP's we don't allow relaying from). I have limits if there's 5 or more different IP's I get notified. If there's 15 or more IP's the account is auto suspended (in the database; active='N'). If there's 200 messages, I get notified; if there's more than 800 it's auto suspended.

For cases where I'm notified, I'll take a look at the logs. I'll grep the mailid's with the sasl_user=xxx into a file and then grep -f $file . As the last two lines of our header checks file are:

/^subject:/ WARN
/^from:/ WARN

I can see obviously spammy subjects and randomly generated forged from headers with the mailid grep results. Additionally, I'll see how many people are addressed with each message. It's very quickly obvious if it's a user being a bit more active than usual, or a compromised account.

If you only have 10 users, it might be enough to see when you're getting a ton of bounce messages, and keep this fresh in your memory. If you have 30k, then you go the periodic scripts route.

Note, fail2ban is useless for me for pop3/imap account scanning. I can watch my logs for failed password attempts, and it's obvious that a distributed botnet is making a run, as the same user will have a failed attempt from one IP. 10 seconds later, from a new IP, 10 seconds later, a new IP etc. Never multiple failed attempts from the same IP, unless its one of our IP's (I.E. we don't want to ban it). As I don't have a particularly sophisticated user base, it's not uncommon to regularly see 10-20 failed attempts before a successful access from a user's IP.
posted by nobeagle at 7:33 AM on May 30, 2014