Amazon S3 for log backups?

October 22, 2009 10:04 PM

I'm looking for a step by step recipe for log rotation from a web server to Amazon S3.

Caveat: I am an utter newbie to Amazon S3 and EC2.

I currently have a backup process that works well. It's a cronjob that copies server logs on a nightly basis from an external machine that I do not control to a linux VPS I do control via scp.

The cost of storage is killing me (I keep > 1 years worth of logs) so I'd like to use Amazon S3 as the destination for the logs, while keeping this as unixy as possible.

What are the steps I need to take to do this? Baby steps, please. I'm new to Amazon's S3 and EC2.

I would prefer something like a nightly job where I run:

scp [e-mail]:logfile [e-mail]:destination

or the rsync equivalent.

I do not need more than 99% reliability. Simple is best. Installing new software on the outside machine is difficult, so a standard scp or rsync method preferred.

I can install software on one of my servers, but I presume it would not be involved in the transfer as I'd take a 3x bandwidth hit (outside machine to mine, mine to s3 x 2: upload from my side, download from theirs)

Over time, I will have more than one outside machine to back up. For now, it's just one.

posted by anonymous to Computers & Internet (7 answers total)

You can't scp or rsync files to S3, but S3 allows POST uploads from Web forms, which you can emulate using 'curl' which is probably already installed on your external servers. You'll have to create an AWS account and an S3 bucket, then define a policy which gets encoded and signed and included in the POST data. Here's an article about this.
posted by nicwolff at 12:32 AM on October 23, 2009

And here's a handy tool that will help you encode the policy and generate the form data.
posted by nicwolff at 12:49 AM on October 23, 2009

I think you could do scp/rsync from an external server to an EC2 instance, and then store on S3. IIRC there's no bandwidth charge between EC2 and S3. I'm not sure exactly how you would do this though.
posted by zippy at 2:27 AM on October 23, 2009

s3sync?

Disclaimer: I just started using this yesterday.
posted by oaf at 4:58 AM on October 23, 2009

I heartily endorse s3sync, and I've been doing this for ages!
posted by soma lkzx at 6:32 AM on October 23, 2009

Are you bound to EC2?

rsync.net looks right up your alley.
posted by bensherman at 1:04 PM on October 23, 2009

I've found s3tools to be quite nice. Works similarly to rsync and is bog-simple to set up.
posted by zippy at 1:20 AM on October 25, 2009