Can you ever move more than 1 server-based IMAP folder at a time?

January 7, 2022 9:28 AM

I know you can work around stuff with scripting, but assuming you use a typical GUI desktop email client connected to a typical IMAP server, or maybe webmail. Is this doable?

Or even with an IMAP-based webmail interface, like roundcube. I feel like the answer is 'no', but maybe not.

Example: in your standard GUI user interface, you either
a) want to highlight two IMAP folders and move them simultaneously into another folder on your mail server, or maybe
b) you highlight several folders on your IMAP mail server, and want to move or copy them over to another IMAP mail server on a different IMAP server host.

thx mefite email wizards

posted by bitterkitten to Computers & Internet (8 answers total)

Best answer: Anecdotally, I wouldn't expect this to work, and based on experience I wouldn't try.

More technically: as of the last time I needed to know, most non-Gmail IMAP servers stored messages in a format called Maildir, where each message is stored as a text file on a logical storage volume. The servers I have experience with will allow an individual user to have multiple server connections going at once (which supports things like checking multiple folders for new mail at once), but that is a great scenario for a race condition (where two processes both try to act on the same resource, with unpredictable results based on which process finishes first) or a deadlock (where two processes are stuck waiting on each other and neither can finish).

In theory you could design an IMAP server and client pair that performed big operations like this in what's called a thread-safe way (i.e. one that prevents deadlocks and has a way to prioritize operations so race conditions don't get too messy) but there are far too many complications for that to be considered a solved problem. In practice what happens when you try to move too many things at once (where "too many" varies based on access speed of your storage media, available RAM, per-user process limits, process execution time limits, and any number of other factors) is that a bunch of things will get moved, but not all of them, and then the server and client will spawn a bunch of error messages that quite possibly run out of control. Your IMAP client will then have to spend a bunch of time rebuilding its message index so it knows where things actually are, before you can select the next batch to (try to) move.

In my experience the most effective way to do large scale moves like this is to take your IMAP client offline and perform direct operations on the filesystem where the Maildir lives. Once the filesystem operations are done, bring your IMAP client back online and force a full index rebuild. Yes, you're still rebuilding the message index, but you're only doing it once, and you reduce the risk of weird filesystem artifacts from aborted operations and IMAP folders full of messages flagged as deleted but not actually, y'know, deleted.

The second most effective way to do this is to figure out the largest "safe" number of messages you can manipulate through a single bulk operation (quite possibly by iterating through batches that are first too large or too small) and then use your IMAP client to move only that many messages at once. Allow each operation to complete before starting the next one. This has the advantage of being a technique you can replicate across multiple IMAP clients and servers (especially the ones where you don't have the necessary local filesystem access), adjusting the "safe" number to match what the current environment can actually do. It's a pain, but when your batches are nice and Goldilocks-sized, you probably won't have too many errors, your IMAP client won't spin endlessly as it rebuilds the index, and you won't bring the server to a screeching halt.

I can't tell you the Goldilocks number for your setup, but 500 may be a good number to start with.
posted by fedward at 10:54 AM on January 7, 2022 [2 favorites]

Best answer: Look at the RFC for IMAP. RFC2060 from 1996 was superseded by RFC3501 in 2003. I think server-side moves are achieved by RENAME which doesn't seem blocked by concurrent rules in section 5.4.

Regarding (b) moving between hosts will have to download-then-upload to migrate the data between hosts.
posted by k3ninho at 11:06 AM on January 7, 2022 [1 favorite]

Best answer: Oh, as an extra bonus answer: moving from one IMAP server to another by doing things in an IMAP client will be much slower than moving things around within a single account. If you want to migrate an archive of mail, check to see if the provider of services at the destination has a migration tool (I know Gmail does). In that case you enter the address of the source IMAP server and your login credentials, and the migration tool will actually just log in as an IMAP client and extract that messages that way, without also trying to keep a local index updated at the same time. This generates fewer errors that require handling so the process will be much smoother.
posted by fedward at 11:09 AM on January 7, 2022 [1 favorite]

Best answer: RFC3501 section 5.5 says clients can send multiple instructions except:

The exception is if an ambiguity would result because of a command
that would affect the results of other commands. Clients MUST NOT
send multiple commands without waiting if an ambiguity would result.
If the server detects a possible ambiguity, it MUST execute commands
to completion in the order given by the client.

Basically the IMAP client is supposed to wait for those commands to complete before issuing new commands, but IME not every client implements the rules correctly. IIRC (from a recent inbox cleanup) Roundcube plays by the rules and blocks further input on big operations like that. By definition that block would prevent you from dragging two different folders into one because any operation on the target folder would be ambiguous (for example, if you're trying to drag both A and B into C, once you start the A-to-C operation any further command on C while that first command is running would result in "an ambiguity" on C; dragging both A and B at once is itself an ambiguous command). In theory the server should support dragging A into C and B into D concurrently, but if the client prudently blocks further inputs on such a complex operation there's no way to find out.

Regardless of RFC, trying to perform an operation on too many items at once is almost always problematic; the trick is to figure out what "too many" means and stay below that number.
posted by fedward at 11:29 AM on January 7, 2022 [1 favorite]

Response by poster: Wow. Thanks! Those were pretty technically super answers. : )
posted by bitterkitten at 3:24 PM on January 7, 2022

a) you would in effect select the first two folders and merge them into a temporary folder locking both in place while that happens, then you would merge that temporary folder into the final destination locking both during the operation. During the first part, new mail assigned to the moving folders would be queued until the lock is released. Same for the second but only the final destination would be locked. But that leaves you with what is dropping mail into folders and as soon as the queue starts up possibly during the second move those new mails would re-create the folders that you just moved by means of new mail that was blocked.

You first have to change the rules that direct mail into folders to redirect mail that would go into the first two folders to instead go into the final destination so that the first two would have no new incoming messages. Then you're all good.

How does your mail get into folders in the first place? Is it automatic by filtering rules, or is it just something you do manually?

The answer will also depend on that.

But likely it's a 'no' because of what fedward said in the first comment. It's not trivial and depends on the server you're using and how you're sorting mail.

b) Pretty much the same. Stop the creation of new messages into the moving folders first either directing them to the final destination or to somewhere else. Then download/upload and maybe take care of the rest of the mail that arrived. But you should have been sending that new mail to the new server before you started moving the old mail.
posted by zengargoyle at 3:41 PM on January 7, 2022

So while the RFCs and such are interesting, the real answer is that it's up to the MUA (mail client). There are lots of ways that a client could queue requests up such that it complies with the RFCs while allowing multiple-folder operations.

I just did a quick test using Apple Mail v14 (the version included with Big Sur) and it definitely does allow copying multiple IMAP folders from one server to another. I was able to copy two Gmail IMAP folders onto my personal IMAP folder in a single operation (as far as I, the user, was concerned).
posted by Kadin2048 at 2:45 PM on January 9, 2022

What happened to any incoming mail while you were copying? The server is/could receive mail destined for those folders your copying. Where does that mail end up? It's a server thing, not a client thing. Server side automatic filtering of incoming messages into folders in the first place throws a monkey wrench into things. When you move a folder do the filters also move?

(heh, I've argued RFCs with M$ and won, and lost. Don't let me get started on their old TCP/IP stack or DHCP handling. grrrrrrrr)
posted by zengargoyle at 3:46 PM on January 9, 2022