Comments on: How do I export or convert my Thunderbird inbox into the Unix mailbox format?

Question: How do I export or convert my Thunderbird inbox into the Unix mailbox format?

FakeOutdoorsman — Wed, 11 Apr 2007 12:47:19 -0800

How do I export or convert my Thunderbird inbox into the Unix mailbox format?

I need to compile a spam corpus to test a Beysian spam filter. Thunderbird mailbox looks like a Unix type mailbox, but it adds some extra headers: X-Account-Key, X-UIDL, X-Mozilla-Status, X-Mozilla-Status2.

By: twiggy

twiggy — Wed, 11 Apr 2007 13:06:04 -0800

My understanding is that Thunderbird mailboxes are already in unix mailbox format.

By: twiggy

twiggy — Wed, 11 Apr 2007 13:07:53 -0800

Sorry, I clicked submit on accident. If you really don't want those headers there, you would have to write a script to remove them (via Perl or any other language)...

There's really no need to, though. Since they're present in all of your messages good and bad, the Bayesian filter shouldn't look at them as bad or anything. It's already in valid unix mailbox format, it just has some extra headers. That doesn't make it incompatible though.

By: cmiller

cmiller — Wed, 11 Apr 2007 13:35:14 -0800

Fwiw, instead of "Unix format", you should describe it as "mbox", assuming that's what you mean (instead of "Maildir", e.g.). You'll get more search hits that way.

By: sergeant sandwich

sergeant sandwich — Wed, 11 Apr 2007 16:43:02 -0800

to strip the unwanted headers:

grep -v X-account-key yourfile | grep -v X-UIDL | ... etc ... > yournewfile

assuming, of course, that you don't have those strings in any of the body of your messages.

By: sergeant sandwich

sergeant sandwich — Wed, 11 Apr 2007 16:44:44 -0800

there's probably a cleverer way to do that using fgrep though.

By: sergeant sandwich

sergeant sandwich — Wed, 11 Apr 2007 16:45:06 -0800

egrep i mean. argh

By: oats

oats — Wed, 11 Apr 2007 17:50:46 -0800

egrep -v '^(X-account-key|X-UIDL|X-Mozilla-Status)' yourfile > yournewfile

Forcing it to find only at the beginning of the line ('^') is both safer and faster.

By: cmiller

cmiller — Sat, 14 Apr 2007 11:21:15 -0800

*Ahem*, it's legal to have those in the body of the message. Imagine, e.g., what happens when my email contains an encoded attachment with a line that starts with "X-UIDL".

You need a state machine. You can do it with sed.

By: cmiller

cmiller — Sat, 14 Apr 2007 11:23:26 -0800

Oh, and RFC822 (or is it 823?) headers can span several lines.

Header: one two three next line starts with whitespace
four five six grep cannot help with this.

I hate to be a spoil-sport here.