Why hasn't a new version of POP3 been developed?

November 3, 2009 11:13 PM

Why is the POP3 mail protocol so slow and why hasn't a new version of POP been developed that fixes the speed problem? I'm pretty sure that it would be far faster to just download a compressed file containing all the messages and then unzip it and parse it locally. POP3 is fine when you keep up with your mail, but leave for a week and come back and I end up waiting quite a while to get all the mail downloaded.
posted by HappyEngineer to technology (13 comments total)

I don't think your problem is really with the protocol; POP doesn't have that much overhead. Your problem is more likely with the way that protocol is being implemented on your ISP's or mailhost's server.

The amount of time you spend waiting around due to protocol overhead is almost certainly dwarfed by the amount of time taken by the server to retrieve messages from some sort of mailspool (which might be in big flat files, lots of small files, or in a database) and then start sending them.

About the only bad things you can say about POP are that it doesn't have compression or security in it natively, but those are both remedied if you use POP inside TLS (port 995, typically), which is common to the point of near-universality (or at least to the point where if your mailhost doesn't offer it, you should get a new one). There's also the issue of email in general, and how binary data gets inefficiently encoded via hacks like MIME for historical reasons (non-8-bit-clean channels), but you get that with SMTP too.

I don't use POP — I use IMAP because I want synchronization, not just retrieval, and there are a lot of criticisms of IMAP that are quite legitimate — but there's nothing fundamentally wrong with it as long as what you're trying to accomplish is within it's relatively narrow domain.
posted by Kadin2048 at 11:22 PM on November 3, 2009 [2 favorites]

Short answer - it's Good Enough^TM. Also, there's not a whole lot of money in developing new transport protocols.
posted by squorch at 11:26 PM on November 3, 2009

But ... none of that really answered your question (why hasn't a new version been developed), which I think boils down to three issues:

1.) It's "good enough," albeit imperfect, for most users.
2.) It has been superseded in many people's minds by IMAP and proprietary synchronization protocols used by PIM suites (Outlook/Exchange and, to a much lesser extent Lotus Notes).
3.) There actually was a POP4 proposed, but it wasn't adopted, mostly because it was a solution to a problem that too few people had.

That's the closest I can come to a direct answer; probably not the most satisfying thing in the world.
posted by Kadin2048 at 11:28 PM on November 3, 2009

There's also the fact that POP mail accounts have been replaced, for a lot of users, with webmail accounts. Many people simply have hotmail, gmail, or yahoo accounts because it's more convenient to be able to get your mail from anywhere, not worry about losing it if your hard drive crashes, and rely on the far superior spam filtering that the big webmail hosts provide. POP isn't just not being replaced; it's likely to become a niche email solution in the next decade.
posted by fatbird at 12:01 AM on November 4, 2009

Long Answer to the why part of your question: it's by deliberate design:

Future extensions to POP3 are in general discouraged, as POP3's usefulness lies in its simplicity. POP3 is intended as a download-and-delete protocol; mail access capabilities are available in IMAP. Extensions which provide support for additional mailboxes, allow uploading of messages to the server, or which deviate from POP's download-and-delete model are strongly discouraged and unlikely to be permitted on the IETF standards track.

This is consistent with what's known in hackerdom as the Unix Philosophy, parts of which could be interpreted to mean that you should structure the data as openly and as simply as possible so that humans can work with it easily, and rely on hardware upgrades to speed things up. Since the foundations of the Internet were laid in Unix, the Unix Philosophy heavily influences most aspects of it, i.e. the plain-text rather than binary nature of HTML, XML, HTTP, POP etc.
posted by randomstriker at 12:04 AM on November 4, 2009 [1 favorite]

One other point is that some mail clients implement POP3 very poorly - was it certain versions of OE that did stupid things, like issue a STAT, then a LIST (or TOP?), between RETRieving each message? Asking the mailserver to go away and recalculate how many undownloaded messages there are each time you download one, then asking for the header (or header + first 10 lines), then asking for the whole message including headers, is just a dumb recipe for slow speed.

I suspect some versions of Apple's Mail.app do a similar thing, but can't be arsed to look.
posted by Pinback at 12:13 AM on November 4, 2009

I don't think your problem is really with the protocol; POP doesn't have that much overhead.

It's not the overhead, it's the latency. POP has a pipeline of length one; you have to send a RETR and wait for the message to be fully received before sending the next RETR. All of those round-trips add up to limit throughput. It's analogous to the bad old days of HTTP 1.0 before persistent connections and pipelining.

The real answer is use IMAP.
posted by Rhomboid at 12:15 AM on November 4, 2009

Use IMAP, or Webmail on the server (a la Squirrelmail).
posted by effugas at 2:14 AM on November 4, 2009

First of all your idea wouldn't work like that, you'd compress on the wire instead of downloading a gzipped file. You can do this already if you want: tunnel over a compressed ssh tunnel.

Secondly, what are you doing? Are you leaving messages on the server, or downloading them locally and removing them from the server? The second is often quicker because the server doesn't have to seek through a potentially big file.

Thirdly, everyone technically minded already switched to IMAP or webmail+IMAP or offlineimap..
posted by devnull at 2:48 AM on November 4, 2009

POP4 is IMAP. Aside from complexity, there isn't a problem with POP3 that IMAP doesn't solve. (Complexity is a problem, though.)
posted by cmiller at 7:09 AM on November 4, 2009

IMAP is better than POP in almost every respect.

Is your POP server doing spam filtering server-side, or do you have to download all the spam to your mail client and then filter? That right there will make POP intolerable.
posted by Nelson at 7:36 AM on November 4, 2009

As others have said, you should just use IMAP. Copying all your mail off the server into a local folder is basically the same as downloading and deleting via POP3.
posted by chunking express at 9:03 AM on November 4, 2009

Everything you want in some magical future release POP you can get right now with IMAP.

I think its important to understand how unixy protocols, software, and standards works. One guy or a comittee punch out some standard or implementation. It stagnates for a while, perhaps it gets changed, who knows, depends on the project. Then another guy who gets pissed at this starts a similar but more advanced project. POP to IMAP, telnet to ssh, ftp to ftps, Mozilla to Firefox, etc.

Dont hold your breathe expecting POP to do anymore than it already does. Its deemed a lightweight protocol and will probably never change. People expect it to behave a certain way, like SMTP.

Start using IMAP with SSL. Not only will the encryption protect your from prying eyes, it also tends to compress data so there's less to send.
posted by damn dirty ape at 10:19 AM on November 4, 2009