Recover lost Apple Mail after creating IMAP account with same address?
October 28, 2020 11:25 PM Subscribe
A friend of mine was trying to switch his Apple Mail IMAP account from using 1 provider he no longer has access to to another provider, but using the same emall address. Now all his old email has disappeared. Is it still there on his Mac but hidden and recoverable?
In trying to switch, after he created the second account on his Mac (same email address, but different remote hostname, username, and password), all his email from the old account seems to have disappeared. Can he recover the email by changing some settings, maybe removing the new account? The old account no longer exists remotely; he can't log in to the old provider to redownload email.
What he hoped would happen was that by setting up the new account with the old email address, his email would either be transferred to the new provider; or would just be available on his Mac anyway, and he'd be able to use the email address with the new provider.
When the mail seemed to disappear, I told him to duplicate his /Library/Mail/ folder in the Finder and quit Mail until we could figure something out. Is there anything to be done? Short of sending his hard drive somewhere for data recovery or running some iffy data recovery program.
In trying to switch, after he created the second account on his Mac (same email address, but different remote hostname, username, and password), all his email from the old account seems to have disappeared. Can he recover the email by changing some settings, maybe removing the new account? The old account no longer exists remotely; he can't log in to the old provider to redownload email.
What he hoped would happen was that by setting up the new account with the old email address, his email would either be transferred to the new provider; or would just be available on his Mac anyway, and he'd be able to use the email address with the new provider.
When the mail seemed to disappear, I told him to duplicate his /Library/Mail/ folder in the Finder and quit Mail until we could figure something out. Is there anything to be done? Short of sending his hard drive somewhere for data recovery or running some iffy data recovery program.
Most email these days is IMAP which lives on a server unless you explicitly move or copy it elsewhere. Your friend might be screwed.
posted by infinitewindow at 5:45 AM on October 29, 2020
posted by infinitewindow at 5:45 AM on October 29, 2020
Macworld has a suggestion if they have a Time Machine set up. Otherwise it might be recoverable from the ~/Library/Mail backup as well.
On the other hand, none of the file sizes in there are anything like the right total size for my gmail imap mailbox, so you might just end up recovering headers. On the other other hand, if the source IMAP server was already gone, that might be all that's left anyway.
posted by Kyol at 6:45 AM on October 29, 2020
On the other hand, none of the file sizes in there are anything like the right total size for my gmail imap mailbox, so you might just end up recovering headers. On the other other hand, if the source IMAP server was already gone, that might be all that's left anyway.
posted by Kyol at 6:45 AM on October 29, 2020
Did your friend create a new account with the same name, then disable the old account? If so, he might be able to recover the email by going to Mail > Preferences > Accounts, clicking on the old account (which will say "Inactive" underneath the account name), and clicking the checkbox "Enable this account".
The Apple Mail app hides the mailboxes of inactive accounts but doesn't delete them.
If your friend selected the old account and hit the "-" button at the bottom of the account list to remove it, though, he may be out of luck.
posted by brianogilvie at 7:56 AM on October 29, 2020
The Apple Mail app hides the mailboxes of inactive accounts but doesn't delete them.
If your friend selected the old account and hit the "-" button at the bottom of the account list to remove it, though, he may be out of luck.
posted by brianogilvie at 7:56 AM on October 29, 2020
Having given an answer to the question, I would now like to know what magical thinking was behind this idea:
posted by brianogilvie at 7:58 AM on October 29, 2020
What he hoped would happen was that by setting up the new account with the old email address, his email would either be transferred to the new provider; or would just be available on his Mac anyway, and he'd be able to use the email address with the new provider.That's not how email works.
posted by brianogilvie at 7:58 AM on October 29, 2020
What he hoped would happen was that by setting up the new account with the old email address, his email would either be transferred to the new provider; or would just be available on his Mac anyway, and he'd be able to use the email address with the new provider.
This kind of thing infuriates me. I've seen it happen so freaking often. And the reason it infuriates me so much is because it's such an absolutely perfect illustration of why people really should be told that it matters where their data resides. But instead we've seen a smarmy fucking marketroid decide that instead of helping people understand the tools they're using, he's going to profit from their ignorance by patting them condescendingly on the head and saying Oh Dear Is That Too Hard For You, Never Mind, I Am A Genius, Pay Me To Worry About It Instead. And then doing a shit job of it. Which shittiness should be unsurprising, because it's impossible. Because there is no substitute for understanding where your stuff is kept.
IMAP is, as infinitewindow says, designed to pull individual mails from a server on demand; and it also defines a standard way for that server to organize mails into folders. That makes it completely feasible to have an IMAP-based mail client that always pulls mails down from the server when asked to display them and doesn't bother storing a local copy.
In fact most IMAP clients don't actually do that, because in many cases it's too slow. What most of them do instead is store local copies as well, and then use IMAP to keep those local folders and the server folders in sync. So even if you temporarily lose access to the server (which, seeing as how it's usually on the far end of an unreliable network, you are almost certain to do from time to time) you can still read the local copies of mails you already have. And when you want to re-read a message you don't suffer the network lag you'd see if it had to get fetched from the server a second time.
But the fact that you don't see lag, as you would if you were using for example webmail instead of IMAP, gives rise to confusion about where the stuff is actually stored. The natural assumption is that anything accessible as quickly as that has to be local. And of course it is. But there's a perennial problem inherent in the whole notion of keeping a local folder in sync with a remote one: what exactly should be done about items that are found to exist on one side but not the other once there is a working connection to the server?
IMAP mail clients will almost always resolve this by (a) downloading messages that exist on the server but not on the client, on the basis that the most likely cause for this condition is the arrival of new messages that haven't yet been seen by this client; (b) deleting messages that exist on the client but not on the server, on the basis that the most likely cause for this condition is that the user also owns some other client syncing with the same mail account, has used that client to delete the message from itself and from the server, so clearly wants it gone from this client as well.
However, as your friend has now discovered to his considerable cost, both these cases have to be handled according to a best guess. IMAP simply has no way to represent a message that used to exist but doesn't any more. Messages either exist or they don't. And in particular there's no way for an IMAP client to distinguish a server from which all the mails really have been deleted from a new, blank server that happens to have the same email address. Your friend apparently made an implicit assumption that his IMAP client would resolve any such mismatch by uploading all the messages found to be missing from the server. I have yet to encounter an IMAP client that would actually do that.
The only way to guarantee that an IMAP-based mail client will never accidentally delete a locally-stored message that you really didn't want it to is to get into the habit of making an explicitly local copy yourself into a folder that is not connected by IMAP to any mail server. Thunderbird calls these "local folders" and copying stuff into them is easily done by dragging and dropping. I'm sure Apple Mail has something similar. But most people never do this, because most people don't understand why on earth they would ever want to, because every computer marketer that's ever been has kept on repeating Computers Are Hard, Let's Go Shopping until they bloody well got believed. Grrrr.
Your friend could have got the behavior he wanted by disabling his Internet connection to block access to any mail servers, then starting Apple Mail, then dragging and dropping all the existing mails into local folders that don't belong to any particular mail account, then turning the Internet back on to let Apple Mail connect to the new server, then dragging and dropping from the local folders back into the IMAP-synced ones. At which point Apple Mail would start a probably-several-hours-long process of uploading those, which it might or might not be safe to interrupt.
And if he's really really lucky he'll still have that option available after restoring his ~/Library/Mail folder from a backup, perhaps from Time Machine.
And if he's not really really lucky, then what Kyol suggests might have happened has happened, and Apple Mail was only ever set up to store headers locally for IMAP connections instead of whole messages, and really was pulling mail bodies across the network on demand every time. In which case the only remaining option is heavy drinking.
posted by flabdablet at 8:01 AM on October 29, 2020 [4 favorites]
This kind of thing infuriates me. I've seen it happen so freaking often. And the reason it infuriates me so much is because it's such an absolutely perfect illustration of why people really should be told that it matters where their data resides. But instead we've seen a smarmy fucking marketroid decide that instead of helping people understand the tools they're using, he's going to profit from their ignorance by patting them condescendingly on the head and saying Oh Dear Is That Too Hard For You, Never Mind, I Am A Genius, Pay Me To Worry About It Instead. And then doing a shit job of it. Which shittiness should be unsurprising, because it's impossible. Because there is no substitute for understanding where your stuff is kept.
IMAP is, as infinitewindow says, designed to pull individual mails from a server on demand; and it also defines a standard way for that server to organize mails into folders. That makes it completely feasible to have an IMAP-based mail client that always pulls mails down from the server when asked to display them and doesn't bother storing a local copy.
In fact most IMAP clients don't actually do that, because in many cases it's too slow. What most of them do instead is store local copies as well, and then use IMAP to keep those local folders and the server folders in sync. So even if you temporarily lose access to the server (which, seeing as how it's usually on the far end of an unreliable network, you are almost certain to do from time to time) you can still read the local copies of mails you already have. And when you want to re-read a message you don't suffer the network lag you'd see if it had to get fetched from the server a second time.
But the fact that you don't see lag, as you would if you were using for example webmail instead of IMAP, gives rise to confusion about where the stuff is actually stored. The natural assumption is that anything accessible as quickly as that has to be local. And of course it is. But there's a perennial problem inherent in the whole notion of keeping a local folder in sync with a remote one: what exactly should be done about items that are found to exist on one side but not the other once there is a working connection to the server?
IMAP mail clients will almost always resolve this by (a) downloading messages that exist on the server but not on the client, on the basis that the most likely cause for this condition is the arrival of new messages that haven't yet been seen by this client; (b) deleting messages that exist on the client but not on the server, on the basis that the most likely cause for this condition is that the user also owns some other client syncing with the same mail account, has used that client to delete the message from itself and from the server, so clearly wants it gone from this client as well.
However, as your friend has now discovered to his considerable cost, both these cases have to be handled according to a best guess. IMAP simply has no way to represent a message that used to exist but doesn't any more. Messages either exist or they don't. And in particular there's no way for an IMAP client to distinguish a server from which all the mails really have been deleted from a new, blank server that happens to have the same email address. Your friend apparently made an implicit assumption that his IMAP client would resolve any such mismatch by uploading all the messages found to be missing from the server. I have yet to encounter an IMAP client that would actually do that.
The only way to guarantee that an IMAP-based mail client will never accidentally delete a locally-stored message that you really didn't want it to is to get into the habit of making an explicitly local copy yourself into a folder that is not connected by IMAP to any mail server. Thunderbird calls these "local folders" and copying stuff into them is easily done by dragging and dropping. I'm sure Apple Mail has something similar. But most people never do this, because most people don't understand why on earth they would ever want to, because every computer marketer that's ever been has kept on repeating Computers Are Hard, Let's Go Shopping until they bloody well got believed. Grrrr.
Your friend could have got the behavior he wanted by disabling his Internet connection to block access to any mail servers, then starting Apple Mail, then dragging and dropping all the existing mails into local folders that don't belong to any particular mail account, then turning the Internet back on to let Apple Mail connect to the new server, then dragging and dropping from the local folders back into the IMAP-synced ones. At which point Apple Mail would start a probably-several-hours-long process of uploading those, which it might or might not be safe to interrupt.
And if he's really really lucky he'll still have that option available after restoring his ~/Library/Mail folder from a backup, perhaps from Time Machine.
And if he's not really really lucky, then what Kyol suggests might have happened has happened, and Apple Mail was only ever set up to store headers locally for IMAP connections instead of whole messages, and really was pulling mail bodies across the network on demand every time. In which case the only remaining option is heavy drinking.
posted by flabdablet at 8:01 AM on October 29, 2020 [4 favorites]
I think there was some initial confusion on this as well. Unless you've got something more complicated than the average situation, the email address is inherently tied to the provider. If you no longer have access to the account via the provider, you are not going to be able to keep the same email address by switching to a different provider.
(On preview, it appears that others have the same thought.)
Was this an email address on a domain they own and manage, a free address (gmail, hotmail, etc.), one provided by an ISP (AT&T, Comcast, etc.), a school/work address, or something else altogether? Or was it an alias that was set up to forward to somewhere else?
Depending on the reason he no longer has access, and the time since he lost access, it may be possible to pay the provider to restore the original account so that you can access it to properly back up the email. Many ISPs will keep the contents of a disabled account for a short time after the customer cancels service, so that if they re-activate, the data is still available. Some will even allow you to pay for an email-only account, so that you can keep your old email address without getting service from them. Depending on your situation, you may be able to come at this from that direction if the local backups don't pan out.
posted by yuwtze at 8:08 AM on October 29, 2020 [1 favorite]
(On preview, it appears that others have the same thought.)
Was this an email address on a domain they own and manage, a free address (gmail, hotmail, etc.), one provided by an ISP (AT&T, Comcast, etc.), a school/work address, or something else altogether? Or was it an alias that was set up to forward to somewhere else?
Depending on the reason he no longer has access, and the time since he lost access, it may be possible to pay the provider to restore the original account so that you can access it to properly back up the email. Many ISPs will keep the contents of a disabled account for a short time after the customer cancels service, so that if they re-activate, the data is still available. Some will even allow you to pay for an email-only account, so that you can keep your old email address without getting service from them. Depending on your situation, you may be able to come at this from that direction if the local backups don't pan out.
posted by yuwtze at 8:08 AM on October 29, 2020 [1 favorite]
It turns out Time Machine just wasn't showing me the sizes properly - in ~/Library/Mail, my gmail mailbox is ~10gigs, which is within spitting distance of what google claims is in my account in the webapp. So if they have time machine backups or they got to ~/Library/Mail before it was actually removed from disk, they might be in luck if Importing the old mailbox actually works.
posted by Kyol at 9:25 AM on October 29, 2020
posted by Kyol at 9:25 AM on October 29, 2020
It turns out Time Machine just wasn't showing me the sizes properly
Time Machine does some pretty heavy-duty data de-duplication. I would expect that the size it shows you for any particular folder at any particular point in time has more to do with the amount of disk space you could free up on the Time Machine drive by removing just that snapshot, rather than the amount of space the data restored from it will eventually occupy. And if the de-duplication works the way I think it does and about as well as I'd expect it to, that will often be two or three orders of magnitude smaller than the size of the original data except for the earliest snapshot still available on the Time Machine drive (or possibly the latest, not sure which way Time Machine does its deltas).
posted by flabdablet at 9:55 AM on October 29, 2020
Time Machine does some pretty heavy-duty data de-duplication. I would expect that the size it shows you for any particular folder at any particular point in time has more to do with the amount of disk space you could free up on the Time Machine drive by removing just that snapshot, rather than the amount of space the data restored from it will eventually occupy. And if the de-duplication works the way I think it does and about as well as I'd expect it to, that will often be two or three orders of magnitude smaller than the size of the original data except for the earliest snapshot still available on the Time Machine drive (or possibly the latest, not sure which way Time Machine does its deltas).
posted by flabdablet at 9:55 AM on October 29, 2020
Response by poster: He doesn't have Time Machine, and no backups. The email address and old service were discontinued at least a year ago, and he actually hasn't bothered reading any of that email, even as it sat on his hard drive, for quite a while. So while it's unfortunate to lose it, it's not a tragedy.
He's a smart guy, but doesn't know how IMAP works, and was very tired when all this happened.
I actually run IMAP servers, but don't read email on a Mac, so was hoping that there might be a peculiarity to how Apple's Mail handles IMAP or accounts that we might be able to exploit easily. If I had access to his computer this wouldn't have happened, and I would now try to poke around on the command line to see if Apple was storing things based on something besides email address and so might have just stopped pointing at things. But he's not going to try if there's not a short, non-technical procedure (or, apparently, any procedure) for getting things back.
posted by mistersix at 12:37 PM on October 29, 2020
He's a smart guy, but doesn't know how IMAP works, and was very tired when all this happened.
I actually run IMAP servers, but don't read email on a Mac, so was hoping that there might be a peculiarity to how Apple's Mail handles IMAP or accounts that we might be able to exploit easily. If I had access to his computer this wouldn't have happened, and I would now try to poke around on the command line to see if Apple was storing things based on something besides email address and so might have just stopped pointing at things. But he's not going to try if there's not a short, non-technical procedure (or, apparently, any procedure) for getting things back.
posted by mistersix at 12:37 PM on October 29, 2020
I think it would be a big help if you could find out more detail about exactly what he did:
1. He opened Accounts; pressed + to create a new account; gave it the same name as his old account; entered server, userid, and password; changed any other settings that he might have wanted to change. Then he selected the other (old) account with the same name, clicked "inactive," and lo and behold, it disappeared!
2. He opened Accounts, selected his current account, and changed the server, userid, and password.
If he did #1, he should be able to do what I suggested above. Mail.app allows you to have two (or more) accounts with the same name; it actually stores data in folders with lengthy, unique hexadecimal names.
If he did #2, his old email is probably toast, or at least overwritten and difficult to recover: Mail.app will have queried the new server, discovered a completely different folder structure, and assumed that your friend used another device to delete the mail and folders that no longer exist on the server, as flabdablet described above.
posted by brianogilvie at 3:48 PM on October 29, 2020
In trying to switch, after he created the second account on his Mac (same email address, but different remote hostname, username, and password), all his email from the old account seems to have disappeared.There's a crucial question: which of the following did he do?
1. He opened Accounts; pressed + to create a new account; gave it the same name as his old account; entered server, userid, and password; changed any other settings that he might have wanted to change. Then he selected the other (old) account with the same name, clicked "inactive," and lo and behold, it disappeared!
2. He opened Accounts, selected his current account, and changed the server, userid, and password.
If he did #1, he should be able to do what I suggested above. Mail.app allows you to have two (or more) accounts with the same name; it actually stores data in folders with lengthy, unique hexadecimal names.
If he did #2, his old email is probably toast, or at least overwritten and difficult to recover: Mail.app will have queried the new server, discovered a completely different folder structure, and assumed that your friend used another device to delete the mail and folders that no longer exist on the server, as flabdablet described above.
posted by brianogilvie at 3:48 PM on October 29, 2020
(By "overwritten," I really mean removed from the file directory and marked as available space on the disk/SSD.)
posted by brianogilvie at 3:50 PM on October 29, 2020
posted by brianogilvie at 3:50 PM on October 29, 2020
>If I had access to his computer ... I would now try to poke around on the command line
You might consider using one of any number of remote access apps to connect to his computer. TeamViewer is my goto, but there are many others (and I think MacOS may have a VNC server built in?). You would then be able to use the machine as if you were physically present. All you'd have to do is talk him through installing/enabling it and giving you the relevant information so you can connect. Whether he'd consider that short and non-technical is an open question.
posted by yuwtze at 6:02 PM on October 29, 2020
You might consider using one of any number of remote access apps to connect to his computer. TeamViewer is my goto, but there are many others (and I think MacOS may have a VNC server built in?). You would then be able to use the machine as if you were physically present. All you'd have to do is talk him through installing/enabling it and giving you the relevant information so you can connect. Whether he'd consider that short and non-technical is an open question.
posted by yuwtze at 6:02 PM on October 29, 2020
Best answer: He doesn't have Time Machine, and no backups. ... he's not going to try if there's not a short, non-technical procedure (or, apparently, any procedure) for getting things back.
Just giving no shits at all about data retention, aka the Konmari Data Policy, is a perfectly cromulent backup strategy and one I will always draw to the attention of people I'm helping take their first steps into IT.
In general I won't ever go so far as to recommend it, but it does save quite tremendous amounts of money, time and angst if you can make it work.
posted by flabdablet at 8:37 PM on October 29, 2020 [1 favorite]
Just giving no shits at all about data retention, aka the Konmari Data Policy, is a perfectly cromulent backup strategy and one I will always draw to the attention of people I'm helping take their first steps into IT.
In general I won't ever go so far as to recommend it, but it does save quite tremendous amounts of money, time and angst if you can make it work.
posted by flabdablet at 8:37 PM on October 29, 2020 [1 favorite]
Response by poster: Thanks for the input, all. I've asked for concrete details on the steps he took in creating the new account, but it's quite possible he won't find it worth his time to go through his memory and explain it. Like many things, this may end up being a thing I cared about more than the person directly affected (and more than I should), in which case "Just giving no shits..." may get best answer here.
posted by mistersix at 12:06 AM on October 30, 2020 [2 favorites]
posted by mistersix at 12:06 AM on October 30, 2020 [2 favorites]
« Older How to not ruin my life post breakup | I’d never worked before. Then came COVID. Now what... Newer »
This thread is closed to new comments.
posted by rockindata at 4:22 AM on October 29, 2020