How can I get a handle on home digital archiving?

August 15, 2011 12:22 PM

I'm tying myself in knots trying to come up with a sensible backup and archiving procedure for my files. This includes work files, personal photos, music, etc. What is your method for making sure that backups are where they are and that your archive works for you?

My two biggest categories are work files and photo files which are two different kinds of beasts. I don't scrapbook and I have thousands of digital images. I use Flickr fairly frequently but I worry about keeping and accessing my photos down the road. Should I be making DVD discs of photos by year and then delete them from my computer? I do need the space. I think about how I will access these in the future and I really have no idea. What will last? What will be easiest?

For work files, I have time machine setup with an external drive. I worry about that drive failing. So, I'm thinking I need to periodically back up files to DVD. But am I missing something there? Many of these files will never be looked at again and my clients don't want them or haven't paid for them. On the other hand, very occasionally, I do pull out discs of old work and pull things off of them. However, finding these old files is a real chore.

Are there any websites exploring this stuff in depth?

posted by amanda to computers & internet (17 answers total)

For work files or other stuff that changes frequently, you can just use Dropbox, SugarSync, UbuntuOne, or whatever file sync tool you want. They all do pretty much the same thing.

For a photo or music library I use Amazon S3. Those files are generally static so once they are backed up you don't need to worry about them too much. I just manually copy them up to S3 anytime I add music or photos to my NAS server. Amazon S3 is designed to survive the simultaneous failure of 2 separate datac enters, so I'm fairly comfortable that the files will be there when I want them.
posted by COD at 12:32 PM on August 15, 2011

A lot of it will depend on what OS you're using, I expect.
posted by Ghostride The Whip at 12:32 PM on August 15, 2011

Mozy, Carbonite, or CrashPlan? I've used all three and they all work, but my current favorite is CrashPlan. $5/mo (less if you do it a year at a time) for cloud-backup, plus it has options to backup to a friend's computer or external hard drives.
posted by alaijmw at 12:32 PM on August 15, 2011 [1 favorite]

A lot of it will depend on what OS you're using, I expect.

Question states Time Machine, implying OSX.

I use Time Machine, but I also back up specific folders to rsync.net.
posted by odinsdream at 12:34 PM on August 15, 2011

I use a two phase plan.

I backup to TimeMachine for a local backup. This is good because I have instant access to my files if I do have a hard drive failure (on my local machine)

I also remotely backup to the cloud via BackBlaze

This covers my ass from a TimeMachine failure - and it keeps backing everything up when I'm on the road or away from my TimeMachine. (I keep the TimeMachine in my office, so its not available on weekends)

This, for me, is the perfect setup. YMMV
posted by bitdamaged at 12:36 PM on August 15, 2011

I echo alaijmw. I'm also using CrashPlan after dumping Mozy because of their new terms of use. So far it's backing up like a champ. It's all I need and I have around 400gb backed up.
posted by Blue.Squares at 12:44 PM on August 15, 2011

Just dropping into the conversation: I used to use BackBlaze, until their abysmal customer service (3 days to respond to an emergency issue) made me jump ship for Mozy...

... Mozy has reconfigured their payment plans, essentially tripling my usage costs, and additionally has software written that blatantly punishes anyone who dares back up more than a gig or two. Before Mozy software can do anything, even something as small as cancelling an in-progress backup, it must poll your entire backup set for changes. EVERYTHING in Mozy takes 2-20 minutes to initiate for "heavy users" such as myself.
posted by IAmBroom at 12:47 PM on August 15, 2011

You definitely need to add off-site backups to the mix.

I've settled on Arq - it encrypts your stuff locally then sends it off to Amazon S3. This can work out to be cheaper than Mozy, Carbonite, Backblaze, CrashPlan, &c. depending on how much stuff you're backing up/how often you access the backups.

I also make a bootable clone of my hard drive every night (automatically) with Carbon Copy Cloner, as well as using Time Machine.

Basically, you can't have too many backups, they should be completely automated, and at least one backup copy should be physically far away from your computer.
posted by jack_mo at 12:47 PM on August 15, 2011

Here at The House Of Introp we:
1. local backup to a NAS box in the closet (also, all our music, financial documents, etc. go here)
2. it backs itself up weekly to an external hard drive
3. nightly job syncs (over ssh) the NAS box to a remote encrypted storage service
4. monthly check of certain files on the remote backup to make sure it's working and can be restored

Only step 3 is custom. None of the encrypted-before-you-send-it services offered a Linux client at the time we rolled all this up so we had to do something special.

Services like CrashPlan which clearly state that (if you use security feature X) they can't even read your data would now be at the top of my list if the current system ever gives me trouble.
posted by introp at 12:51 PM on August 15, 2011

Dan Benjamin has written extensively about his backup strategy. Start here, then read this, and if you want more, this.

He's a smart guy and has requirements similar to yours. Long story short, he uses a Drobo (which has inherent redundancy, like a RAID) as his Time Machine target, and uses a NewerTech Voyager Q with a succession of naked internal hard drives as his backup-backups. His backups (apparently) are all on-site, which he would probably admit is a weakness, but it also sounds like he's just got too damn much stuff to back it all up over the net.
posted by adamrice at 1:49 PM on August 15, 2011 [1 favorite]

For my 100GB+ of family photos (from the first six years of the kids' lives!) plus 60GB+ of music and 40GB+ of digitized movies, I have a fairly straightforward plan:

Redundant local storage

Previously, I had a Linux server running two drives in a RAID 1 array. In layman's terms, that means two identical drives in the computer, each storing exactly the same information at the same time, automatically. Now, I have a Mac Mini doing the same thing. This is good for short-term backing up: let's say I dump my digital camera files onto my computer, then delete the pictures from the camera. Ten minutes later, one of my hard drives dies. No worries, because the data is still available on the other hard drive.

Daily offsite backups (work files)

When my laptop is plugged in at work, it uses Time Machine to back up changes once a day. Then, I bring the laptop home. Now, if my office burns down OR my house burns down with my laptop in it, I have a backup copy in the other location.

Weekly offsite backups (personal files)

Once a week, mid-week, I bring home a portable drive and run a full backup via Time Machine on my Mac Mini (previously, I backed up the Linux server with a portable drive via rsync.) The next morning, I bring it back to work. Same coverage as the daily offsite backups I do for work, but I'm too lazy to bring the drive home more than once a week.

Does this strategy protect me from everything? Not quite, but it is a very cost-effective solution. Hard drives are $60ish, portable backup drives are $90ish, and the work/home computers are ones I would have had anyway. For me to lose data (assuming I stick to my backup schedule), I would have to leave my offsite backup drive at home (or leave my laptop computer at work) on a night that someone broke in and stole everything or the place burned down. I could mitigate that by adding one more drive to the mix, so that I have an independent backup in both locations.
posted by davejay at 1:52 PM on August 15, 2011

Oh yeah, and backing up over the 'net just isn't practical in this age of internet bandwidth capping, plus movie files can be quite large. That's why I don't even bother.
posted by davejay at 1:53 PM on August 15, 2011

Daily continual back up to CrashPlan for all files.

Nightly back up to 2 external hard drives: 1 for personal and 1 for work.

I have considered doing more, but it's so expensive. And, if for some reason my computer, my external drives, AND crashplan fail, i probably have bigger problems.
posted by misanthropicsarah at 2:01 PM on August 15, 2011

My solution only pertains to personal files, but I was using both Time Machine and Jungle Disk for a while. The initial upload to Jungle Disk took forever because, like davejay, I have a ton family photos and home videos.

Eventually I realized it would take another forever for me to download all that stuff again should both my Mac and Time Machine drives die. Now I keep a drive in my safe deposit box and refresh it with all my data every so often. Cheaper than a Jungle Disk subscription, too.
posted by DakotaPaul at 4:49 PM on August 15, 2011

I use Time Machine (OS X) for local backups and Backblaze (OS X/Windows) for off-site backups. It's a good, affordable system, and it's entirely automated, so I can't forget to do it.
posted by wheat at 6:08 PM on August 15, 2011

Ah, yes, I'm on a Mac. Sorry for not specifying.

So, do any of you feel like retrieval is a problem? Like, say, you want to put together a bunch of photos for a family reunion or something so you're going through lots of older photos. Do you feel like you can fairly easily get to them?
posted by amanda at 8:28 PM on August 15, 2011

So, do any of you feel like retrieval is a problem? Like, say, you want to put together a bunch of photos for a family reunion or something so you're going through lots of older photos. Do you feel like you can fairly easily get to them?

It sounds like you're maybe mixing up backing up with archiving? I don't have a problem retrieving photos because every digital photo I've ever taken is in my iPhoto library - having everything 'live' is much more convenient than, eg., sticking a year's pics on a DVD and deleting them, as you suggest in the question.

Obviously if you're only wanting to look at your old photos very rarely, a big hard drive might not be worth the expense, but I tend to think the convenience compared to wrangling loads of DVDs (and labelling them accurately, storing multiple copies safely, &c.) is worth the cost of a few terabytes of external hard drive space.
posted by jack_mo at 2:39 PM on August 16, 2011