Are your tubes clogged, too?

October 21, 2006 10:16 AM

Mail Administrators: are you seeing a huge increase in the amount of e-mail?

I track lots of things on my small mail server, and one of them is the
number of times my mail server accepts a connection on TCP/25 from a
remote server. This number has steadily risen from around 9,000 times
per week (several years ago) to about 14,000 times per week (this summer).

Recently, however, I've seen an explosion in the number of machines that
are connecting to me. Like double. Also, I am getting an astounding
amount of spam. I've done some analysis of my logs using perl, but
can't find anything obvious. Buddies that also run their own mail servers
are seeing similar jumps in volume. What's going on?

posted by popechunk to Computers & Internet (18 answers total)

Response by poster: Yes, my logs are rotating
posted by popechunk at 10:17 AM on October 21, 2006

Spam volume is your problem. I maintain an e-mail infrastructure for a mid-size company. I've seen about a 250% increase in inbound connections over the last 18 months. 90+% of the total inbound messages hitting my gateways either get blocked due to the originating IP being on an RBL, or due to content-based spam filtering & virus checking. 2 years ago, that number was about 60%.
posted by deadmessenger at 10:37 AM on October 21, 2006

The large increase in the number of individual connecting hosts is due to the widespread adoption by spammers of botnets to deliver spam. In the last year or so the problem has really gotten out of hand. Tools for creating and controlling botnets are apparently readily available, and for those not technically inclined a market has developed in which botnets are "herded" by hackers and "leased" to spammers.

Take a look at this article in eWeek: Is the Botnet Battle Already Lost?
posted by RichardP at 10:56 AM on October 21, 2006

I'm not an admin, but I know some in the large organization where I work. They tell me the same. Spam traffic is several times more than legitimate traffic and it's recently getting even worse. Regular users have no idea about how bad it is because the vast majority of the crap never gets near their inbox.
posted by normy at 11:03 AM on October 21, 2006

Take a look at this article in eWeek: Is the Botnet Battle Already Lost?

Wow. I knew it was bad, but not that bad!
posted by Roach at 11:18 AM on October 21, 2006

Response by poster: This increase is in the last two weeks....that's mostly what I'm trying to understand. I've administered large MTAs on the internet for many years, so I understand that there's a lot of spam out there.
posted by popechunk at 11:28 AM on October 21, 2006

Best answer: I use Tuffmail as MX for my domains, and they provide a log of all SMTP connections that attempt to send mail to one of my domains. This includes everything that gets rejected for one reason or another, which steadily hovers at about 1000-1200 rejects/day. I have not noticed a substantial increase in either accepted or rejected envelopes in the past couple weeks.

Since the OP's mail volumes are still relatively low, my first thought is that his address(es) and/or domain(s) recently got onto a list.

Assuming many people are seeing this increase, and the connections are actually attempting to send a message, I have no good explanation.
posted by trevyn at 2:54 PM on October 21, 2006

Best answer: I haven't noticed any such increase. I use the size of my log files as proxy for this; they've been holding steady at around 30-40K most days, with occasional spikes (68K one day last week) but these aren't sustained.

Assuming you have control over your domain's MX records in DNS, I suggest entering an entirely bogus MX record as your secondary. A lot of spammers go straight to the backup MX record if there is one, since they are 1) less likely to have as stringent filtering and 2) more likely to be trusted by the main host. Putting a bogus host as secondary (I use an address in the unallocated 5.x.x.x subnet) cut the number of connections I see each day by about two thirds.
posted by kindall at 2:54 PM on October 21, 2006

Best answer: That's handy, I keep two weeks' worth of archived logs! My company's mailserver for corporate mail, which gets an average quantity of spam, saw 1,782,439 connects from the Internet this week, vs. 1,662,094 last week and 1,562,098 the week before that. That's only 1% per week or so.

I agree with others' assessments that you've just been lucky until now.
posted by mendel at 5:57 PM on October 21, 2006

Best answer: Yes. My web hosting company (that I own) and my full time day job's mail servers have both noticed it. Over the past few weeks there has been a HUGE uptick in spam. Seen chatter about it on the spamassassin-users list and isp-tech lists.

You're not the only one noticing it.
posted by drstein at 9:06 PM on October 21, 2006

I work in the office of information technology at a large public University in the US. There has been a very large increase in the volume of spam since late August/early September, and many people have been working very hard to reduce the effect on end-users. Can't say what caused the sudden increase in magnitude though, I think there's only speculation at this point.
posted by Frankieist at 12:18 AM on October 22, 2006

Response by poster: my first thought is that his address(es) and/or domain(s) recently got onto a list.

I'm pretty sure that this is not the case. I can tell who they're sending the spam to, and none of my users or domains ( there are like 50 small domains) have had a spike.

and the connections are actually attempting to send a message

Jeez, I'm not totally sure of an easy way to mine this out of my postfix logs, but that's a good question. I've been DoS-attacked lots of times (on my "work" MTAs), and this doesn't really feel like one. And I know for a fact that no user or domain has recently spiked. I've been trending a great deal of data I've mined out of my mail logs for years both on this small MTA and my MTAs at work (that handled millions of emails per day). Happily, I no longer administer email at work, but it has taken away another set of data for me to do comparisons with.

A lot of spammers go straight to the backup MX record if there is one

I know this to be 100% true, and have leveraged this approach before. But I doubt they'd stop hitting MXes if the highest cost one was dead. I will talk this over with the guys I trade secondaries with.

I agree with others' assessments that you've just been lucky until now.

This is a possible root cause, but I doubt it. Most of the domains I'm hosting are quite old: they've been out there since the 90s.

So, of the people who look like mail admins and answered my question, the results appear to be:

Yes: hades, drstein, Frankieist
No: deadmessenger, trevyn, kindall, mendel

Very curious. I appreciate y'alls help, and I'll update the thread if I learn anything.
posted by popechunk at 9:08 AM on October 22, 2006

I know this to be 100% true, and have leveraged this approach before. But I doubt they'd stop hitting MXes if the highest cost one was dead.

I have found that this is more true than you might expect.
posted by kindall at 9:33 AM on October 22, 2006

Response by poster: I have found that this is more true than you might expect

I think that the age of the moron spammer has passed us, and we are moving into the time of the savvy spammer, as RichardP's link suggests. I have no (non-anecdotal) evidence for this, but I'm starting to think that the oppo is much better these days.

Your idea is easy to test, though, so once my trending stabilizes, I will see what effect it has.
posted by popechunk at 10:55 AM on October 22, 2006

Worse comes to worst and they DO try all your MXs, just throw a hundred of 'em in and waste a bunch of the spammers' resources.
posted by kindall at 11:53 AM on October 22, 2006

Do you teergrube? This looks, at present, like a valuable strategy for smaller MTA's. Eventually, spammers with big bot networks may decide to target teergrube sites particularly, but that does not seem to be happening now, and it could be that, by the very nature of teergrube, such attacks will never be "worth" their cost to spammers, even those operating very, very low cost, hijacked bot networks.
posted by paulsc at 12:55 PM on October 22, 2006

Response by poster: This trend is continuing unabated, as my most recent graph indicates. (I'd post the image, but IMG is turned off currently.)

Here's a Register article (and the accompanying Slashdot article) that seems to agree with my experience, as well as a handy chart TQM has published.
posted by popechunk at 9:37 AM on November 1, 2006

Response by poster: Another graph for the morbidly curious. Seems to have flattened out, sorta.
posted by popechunk at 7:53 AM on November 25, 2006