How can I aggregate (scrape) email from many different POP/IMAP/web-based accounts to a single server?

July 9, 2007 4:32 PM

How can I aggregate (scrape) email from many different POP/IMAP/web-based accounts to a single server?

I'm attempting to simplify how I use email. Right now, I have many different accounts, each with a very specific purpose. I have most of them feed into Outlook, and they're treated as separate accounts, each of them fetched individually by Outlook at regular intervals. Because I don't always have my computer with Outlook handy, I check my email via web interfaces - one for each account. I recently set up a new server box at home, and want to turn it into an email aggregator, sort of a home-rolled Gmail. I'd like it to pull email from all the different accounts I have, store it locally, and serve it up via IMAP or (by extension) a web interface that I'd install on it.

The question is, what kind of software am I looking for, and does it exist? I've tried searching Google with "email aggregator", "imap aggregator", "pop aggregator", and any combination of those with other keywords such as "server" and "open source", but nothing pops up. Can the wise MeFites help me out on this one?

* FYI: I do not want to use Gmail to do this, for three reasons:
1. Google's poor privacy policies.
2. Remote-storage only, hard to back up.
3. No IMAP access to Gmail.

posted by merkuron to computers & internet (11 comments total)

The program you're looking for is fetchmail. It will download email from IMAP/POP/exchange etc, and can then be set up to redirect that mail to another account.
posted by chrisamiller at 4:58 PM on July 9, 2007

You could run Gmail as a front end aggregator, and have it forward everything to your box, after running through Gmail's excellent spam filter, or behind fetchmail, you could run Exim on Debian, if you have GPL sensitivity. Postfix if you don't. Both are highly configurable MTA systems, that support maildir. You'll need something like Squirrelmail to provide a Web interface on your own box.
posted by paulsc at 5:02 PM on July 9, 2007

And if you combine fetchmail with a properly configured mail delivery agent (like procmail), you can have all the different-purposed accounts filtered into separate folders automatically.
posted by rkent at 5:03 PM on July 9, 2007

You do understand the implications of aggregation for reply, right? Because if you try replying to aggregated messages directly from your own box, that's not gonna work so great on many ISPs these days. So, you may want to configure your MTA to use Gmail as a mailer, to keep from being trapped by various spamkillers.
posted by paulsc at 5:07 PM on July 9, 2007

Thanks for the answers so far, everyone! I've got a lot of reading to do. If you're just reading this and haven't posted an answer yet, please do! The more info, the better. paulsc, I already run SMTP from various addresses all through the same (non-Gmail) server (each with its own from: field obviously), and nobody's complained yet... so I'm fairly certain that aggregated reply will work just fine.
posted by merkuron at 5:29 PM on July 9, 2007

Slight derail: Since I'm currently using Gmail to do exactly what you describe, I'd be interested to know which provisions in Google's privacy policy you object to. There was nothing there that struck me as unreasonable, and I'd like to know if I missed something important.

As far as the "hard to back up" objection goes: I use a POP3 client with Gmail for exactly this purpose, and it wasn't the slightest bit hard.

And their web interface is plenty good enough for my purposes, so I don't miss IMAP at all.

So if you could help me see the error of my ways re. their privacy policy, you could perhaps save me some grief!
posted by flabdablet at 5:40 PM on July 9, 2007

To avoid repeating what other people have already written, I'll post some links about Google Mail's insufficient privacy policy:

http://www.privacyrights.org/ar/Gmail.htm
http://www.google-watch.org/gmail.html
http://www.templetons.com/brad/gmail.html

In short, since business email travels through these accounts, I owe it to clients (as well as myself) to not add their info to the Google/Doubleclick database mess.
posted by merkuron at 5:53 PM on July 9, 2007

If you decide that you don't really need to roll your own mail server, but just want an email aggregator that's independent of a particular PC, I'd recommend Portable Thunderbird.

It's an email client that can gather POP, IMAP, and (with an extension, most flavors of) webmail into one set or many sets of mailboxes, as you like. Free, open source, and you can put it on any USB stick/drive and take it with you from PC to PC. It does not install/leave data on any machine, so no permissions, cleanup needed - just a Win98 or newer OS. Mail is searchable, taggable, secure send/recieve etc.

You'll end up with some problems replying through various addresses (e.g. I want to send the reply to this email through my X email account via SMTP server Z) with either Portable Thunderbird or your home-rolled server...but the Portable App method willl only take about 5 minutes per email account to set up, and it's free. {if you have more than ~3.5GB of mail you want to store, you'll have to use a small USB hard drive, instead of a flash drive you can keep on your keychain.}

With the Lightning calendar extension added on, you can also aggregate calendars published using CalDav, iCal, XML, or (yeah, i know) Google calendars. Or, you could keep paying for Outlook. Jus' sayin'....
posted by bartleby at 7:34 PM on July 9, 2007

Grab the Communigate Pro server. It's free for five or fewer accounts and has a remote POP feature that can retrieve your mail from other POP mailboxes. I'm pretty sure you can use a filter to put the mail from each account into its own IMAP mailbox, too.
posted by kindall at 9:32 PM on July 9, 2007

Oh yeah, I forgot: Communigate Pro link.
posted by kindall at 9:34 PM on July 9, 2007

Mail2Web has been perfect for my needs. My other email forwards to Mail2Web (so M2W doesn't need to pull it in) and then I sync to it using ActiveSync on my phone. I pay the $2 a month to allow me to customise my from header.
posted by dance at 7:54 AM on July 25, 2007