Email auditing and backup, by date?

May 16, 2006 7:20 PM

I've been asked to set up a mail server for a small company with a specific auditing requirement. How do I insure their requirement "show me all emails from May 30, 2005 to June 30, 2005", for example, is fulfilled?

I need to be able to store copies of all the emails, and these need to periodically get backed up to external media. The backups also need to be stored by date, and easily retrievable by non-computer users in the even of an audit. CD-R's are fine, I think, since mail traffic is low...but how do I do this? Is there a mail server or extension/plugin/feature to an existing server which will do this?

Thanks.

posted by bkudria to computers & internet (4 comments total)

Depends. Microsoft Exchange does have a lot of audit-able stuff built into it.
How small is the company? Do they want Linux? Windows? Do you have an existing mail server? What have you researched so far?
There are many ways to fry a turkey, my friend!

* Exchange can do stuff
* Postfix (on Linux) can do this as well
... that will also include Mac OS X.

There's an option called "always_bcc = [e-mail]" in Postfix that people often use to do stuff like this. All email gets CCd to a particular address, and the mailbox for that address gets backed up.

The more information that you can give us, the better the answers will be.
posted by drstein at 7:25 PM on May 16, 2006

Yeah, I don't really think you need a specific mail server, what you need is a system that can sort through the mail after the fact. Any decent system will be able to route copies of all mail into that sorting system. Me personally, I'd just use Postfix and Courier IMAP, and create a special IMAP mailbox just for this mail. Then just use some kind of magical client that can do the sorting/searching.
posted by autojack at 7:48 PM on May 16, 2006

I'll suggest that mail server processes essentially function to process (or create, in the case of outbound mail) SMTP messages into (or from) mail store files, according to instructions maintained in server configuration files that specify connection, routing and filtering rules. So, depending on the audit requirements, you might need to be making a record of the mail server logs, as well as the stored content of various mailboxes and server spools. Depending on the configuration options of a particular site, "backing up" a mail server can be a fairly complicated process, even for a small site.

For example, suppose your mail server implements various filter rules for receiving mail addressed to local mailboxes, such that you deny connections from all IP addresses listed on various RBL's, accept connections and messages from all other sites but filter them for viruses and spam, and hold all mis-addressed messages for your domain on the delivery spool for 48 hours, pending administrator intervention. This is a pretty common configuration setup for small businesses running their own mail server.

So, to back up a server with such a configuration, and show an auditable record of the server's actions for handling all email on a given date, you are going to have to:

1) Backup the server connection logs, to show connections dropped before mail could be exchanged, due to the remote host being listed on an RBL. RBL lists change constantly, and a legitimate can sometimes go on an RBL for several days, due to egregious user behavior which the site corrects. But during that time, your server will be treating a correspondent machine as a leper, and your logs will be the only evidence that this was occurring.

2) Backup the delivery spool, which will have messages that are pending delivery to local mailboxes, but are immediately undeliverable, due to misspellings of user name, bcc envelopes which are defective and can't be properly expanded or evaluated, have relay address instructions your domain rules can't process, etc. Generally, these messages are sitting on the delivery spool, awaiting administrator action, and unless your backup process only runs if your delivery spool is flushed, you'll need to back these up to capture the state of messages which haven't yet been actioned.

3) Backup the mailbox stores. Usually, this will be where the bulk of the mail server's "messages" are stored. There are a number of ways, depending on the server in use, on how the store is maintained, that affect backup strategy. In mail servers like MS Exchange, the mail store is actually an RDBMS database, so that making a backup requires taking a "snapshot" of the database, including a backup of the database and its state information, including transaction logs. Backup software for these kinds of mail servers generally must be integrated into the operation of the mail server and database, and must use Application Programming Interfaces (API's) provided by the server software to make reliable backup and restores.

In other mail server software, the mail stores are simple flat files, much like ordinary text files, which can be backed up with normal file level backup utilities, providing the mail server process isn't simultaneously trying to deliver to the mail store files while they are being backed up.

In either case, the mail stores are generally user accessible, and in terms of audit questions, you have to decide whether it is sufficient to allow deletions and edits by the user to affect the audit record, or not. Suppose that your organization says they want to be able to prove an objectionable message was received and deleted by a user subscribing to a hate mail list. In that case, you are going to have be able to show from backups that a message was received, delivered, read and deleted, and maybe have a record of the message itself. So, simply taking the mail store view once a day, may not be sufficient. You may need to mark "deleted" messages as "eligible for deletion after backup" and have post backup processes which run after your backup process to clear messages users flag for deletion, while showing the users that the messages have been deleted during the interval where they are actually still on the server, awaiting backup and mailbox cleanup.

4) If your mail server implements any delivery control features, such as mailbox size quotas, these controls need to be properly handled by your backup strategy, as they can definitely affect the audit trail as well.

Now, the above discussion is just a simple strategy for a simple inbound delivery machine. If you need to show outbound mail delivery activity, you need to add actions for handling the outbound filtering and delivery mechanisms, as well. Generally, there are different delivery spools and logs to backup, which your audit software would look to for answers to audit inquiries, but the general ideas are the same, and also depend on your mail server software and configuration.

You can't really use the simple suggestion made by autojack to realistically "back up" any practical mail machine, and such a suggestion is quite likely to complicate problems with delivery loops, etc.
posted by paulsc at 1:43 AM on May 17, 2006

Thanks for the detailed response, paulsc.

This is a compnay of approx. 20 people, but email traffic is very low, perhaps 5-10 messages per week. I'm purchasing a Linux server, and will probably use Postfix...unless anyone has any practical objections? I can't believe I forgot to add this in.

paulsc: Seems like I can cover 1 by backing up the server logs, and 2 and 3 with the BCC solution. 4 isn't really relevant. Everything except 3 is not needed, technically.

So, I can BCC all outgoing mail to [e-mail], copy all incoming mail to that address, and then backup those messages periodically to CD and clear that mailbox.

Any other potential problems? Again, thanks for the detailed coverage, paulsc.
posted by bkudria at 4:04 AM on May 17, 2006