Join 3,442 readers in helping fund MetaFilter (Hide)

Digital Storage Advice?
September 19, 2011 10:55 AM   Subscribe

My Lightroom catalog is finally hitting the terabyte mark, which means I either need to double what I'm doing now (internal 1TB drive, which is backed up to an external 1TB drive) or something different. Any tips on how to manage large digital image collections?

I shoot a lot of photos and edit them in Lightroom; Lightroom manages my image library. I have a 2009 Mac Pro with three drives installed, including the 1TB drive that holds all of my images. I have several 1TB drives that I use for various backups, including a Super Duper! clone of the photo drive.

I shot a lot of images recently, and I'm not going to have enough space to import them all. What to do?

1) Get another 1TB internal drive for the available bay in my Mac Pro, and another 1TB external drive to copy it to; or
2) ?

RAID? Something else?

If RAID is something I should do, can you explain it to me? I know nothing about RAID or how to set it up. Would it just connect over USB 2 on my Mac Pro? I'm used to buying bare drives and slapping them in enclosures, but that's as techie as I get.

Specific Newegg recommendations for RAID (if that's what I should be doing) would be greatly appreciated.
posted by Admiral Haddock to Computers & Internet (17 answers total) 4 users marked this as a favorite
If RAID is something I should do, can you explain it to me? I know nothing about RAID or how to set it up.

RAID is basically a way to treat multiple physical hard drives as one logical drive. There are various ways of configuring it, including setting up both drives to be exact mirror copies of one another at all times. In general though, it's not used for what you would want to use it for (for backup purposes), and it will probably just result in extra headaches for you from a configuration perspective.
posted by burnmp3s at 11:04 AM on September 19, 2011

In general though, it's not used for what you would want to use it for (for backup purposes)

To be clear, that's not what I'm asking--I need a new system to store, edit, and manage my photos, not just back them up. Obviously, I can just keep adding one 1TB drive at a time to expand the LR library, but is that the best practice? Or should I use something like a RAID to have a gigantic 4TB collection (and if so, how? And how do I ensure that the images are safe? Two redundant RAID arrays?)?
posted by Admiral Haddock at 11:10 AM on September 19, 2011

There's really no sense thinking about RAID when you can buy a 3TB drive for $120 or whatever they're down to now.
posted by Jairus at 11:12 AM on September 19, 2011

There's really no sense thinking about RAID when you can buy a 3TB drive for $120 or whatever they're down to now.

One of my coworkers thought that too. He lost the drive and huge chunk of data too. RAIDs aren't backups, but they do offer some extra security.

One of the big benefits of using a RAID is, properly configured, if a single disk dies, it doesn't take your pictures with it. Generally, it's also easy enough to add more space as you go. With most configurations, it's possible to swap out one of the drives with a larger one and have the RAID just keep working.

The downsides are that a RAID needs some extra space to have the redundancy to do this. You don't get 4TB of raid with 4 1TB drives. You may get 3 or even 2, depending on how many redundant discs you want to use. They also add a fair cost premium for the RAID hardware itself.

The consumer-grade RAIDs are fairly easy to set-up IME. Easier than, say, a router.
posted by bonehead at 11:18 AM on September 19, 2011

For storage, you may want some big NAS. NAS can be raid or not. (NAS: network attached storage).

NAS can also be a backup solution, if you do something else for your storage.

I use a QNAP Turbo NAS for both. Store data on it as a networkshare, but also use it to back up data from other machines.

You can get into back up of backups, because they'll all die on you eventually. Are you throwing all your data at the 1tb, and not purging stuff ? Carrying all the layers/intermediate files as well ? TB are cheap, but should decide if you want to back everything up, or just the keepers/raw/final shots etc.
posted by k5.user at 11:21 AM on September 19, 2011

bonehead: What you're basically saying is: Raid isn't backup, but let me tell you why RAID is a good backup!

...which is bad advice. Buy a 3TB drive and buy a 2nd 3TB drive for backups. Or use CrashPlan and mail them a 1TB drive to seed an online backup. But relying on RAID for backups in any measure is a bad idea. RAID is for availability.
posted by Jairus at 11:22 AM on September 19, 2011

....availability for people who already have a backup solution.
posted by Jairus at 11:24 AM on September 19, 2011 [1 favorite]

How do you ensure the images are safe? Safe from what? Define the parameters of safety you want, then devise a risk mitigation strategy suitable for it.

Do you want your images safe from hard disk failure? In that case you want RAID 1 (or some other RAID level that supports redundancy). RAID is not hard to set up these days; you can buy a USB hard disk enclosure that takes a pair of SATA drives and does the RAID for you, or if you use internal drives, your OS's built-in RAID software is probably sufficient for such use.

Do you want your images safe from user error (e.g. deleting the wrong file or saving over a version of a file you meant to keep)? Then you want regular on-site backups, probably to another disk or RAID the same size as your main storage.

For example, do you want your images safe from fire, flood, and theft? Then you want regularly-updated off-site backups. What I'd recommend is hooking it up to your main computer to make the initial backup, then hook it up to an Internet connected computer elsewhere (perhaps at a friend's house, or, with permission, at work) to do backups over the net as you add and change material. Situations which you must carry a backup drive back and forth somewhere are prone to forgetting.

Do not simply install a backup solution and assume it is working. Test your backup(s) regularly to make sure you can recover files.

Buy your drives from different suppliers (or even from different manufacturers) to avoid the likelihood that you'll get drives from the same manufacturing batch. This will mitigate the risk that all the drives have a similar manufacturing flaw that will cause them to fail at about the same time.
posted by kindall at 11:29 AM on September 19, 2011 [1 favorite]

If photography and image editing is something you do professionally then you may need to consider getting more hard drives, RAID, backups and off-site backups. The point is storage AND redundancy. You have a 3% chance of a hard drive dying in a year, give or take. Time and usage only make it worse.

I would consider a 4x2TB RAID 10 setup, which would result in 4TB space, striping to increase speeds and mirroring. I would also do weekly backups to external hard drives. And also I would keep an extra hard drive to swap in when a disk in the RAID fails. Also, if you have the bandwidth to do so (depends on your ISP, your account) get an online backup of some sort started. Google Docs will do 1TB at $256/year.
posted by Mister Fabulous at 11:30 AM on September 19, 2011

I'm using a NAS for all my data (a Synology cs407e in my case). I got 4 1TB disks in it, which in RAID5, gives me roughly 3TB of storage space. RAID5 means one of the drives can fail and be replaced without data loss.

I find this setup to be fast enough to use it live rather than working with data on an internal drive and then backing it up to the NAS.

To protect important data from myself (accidental deletion, bad editing) I back it up to a separate partition on the same NAS regularly.

To protect important data from the location (fire, flood, burglary etc) I use a Crashplan account with an unlimited plan to which I back up continuously via their client.
posted by Hairy Lobster at 12:04 PM on September 19, 2011

I am thinking you might want to look at getting a Drobo S and populating it with drives, which will give you plenty of storage that is reasonably fault tolerant to use as your primary photo location or getting a large 3TB drive for internal use on the machine and use a drobo as the backup platform.

You're also probably at a point where it makes sense to use something like crashplan and seed your backup with them by sending them a drive so you have an offsite option that is reasonably low interaction after the initial set up.

The Drobo isn't the only option for this, there are a lot of options, but I've been happy with mine so far. I will say that the filesystem on it is proprietary, but appears to be mostly based around the same stuff in zfs based on the features.
posted by iamabot at 12:05 PM on September 19, 2011

On preview: what imabot said. My setup for this is a Drobo with 4x 2TB drives in it, which gives me just about 5.5 TB of usable space with single-drive drive can fail completely and my data is still just fine. You can pop out a single drive, replace it, and the whole thing just keeps chugging along. I use my Drobo for all my media (photos, audio, video) and have Crashplan backup my photo library to their servers. The audio and video is largely recoverable by re-ripping it all...the photos are born digital, so if they are lost, they are gone.

But the Drobo is a great solution, if a bit more expensive than a roll-your-own NAS.
posted by griffey at 12:11 PM on September 19, 2011

Thirding the Drobo (I use a Drobo FS because I need it to hang off my network and be shared by a bunch of machines). I've had two drives die, and I just cursed a lot, yanked them and replaced them with no loss of data, but as others have said that's not a backup solution. I us a separate Time Capsule for backup. Interestingly, you can also use a Drobo as your Time Capsule drive, giving you backup and redundancy. RAIDed backup is not a bad thing to have.
posted by The Bellman at 12:34 PM on September 19, 2011

This is what I do.

1) Active shoots (anything that might want to be edited or reviewed more or less instantly) are on my internal drive.

2) Archived shoots live on the redundant drive NAS which is just a Gig-E away. I move shoots on/off it with Lightroom itself -- it's a breeze and Lightroom updates the catalogue correctly.

3) Time Machine for the workstation & this must include the Lightroom catalogues and their backups.

4) Weekly backups of the NAS onto a portable solution that lives in a fire/flood rated safe.

5) All final photos uploaded into the cloud at full resolution.

6) Everything is on battery backup.

I have never lost any data despite acts of ornery nature, drive failure, and operator stupidity (including some Lightroom horks that destroyed my catalogue; that's why Time Machine is my best bud.)

NetGear or Synology for your NAS solution (I have a four-drive Synology system I'm thrilled with). I discommend Drobo as when I asked for NAS recommendations I got sad stories.
posted by seanmpuckett at 12:45 PM on September 19, 2011 [1 favorite]

Addenda: the workstation Time Machine destination is a stand-alone dedicated drive, not the NAS -- otherwise there'd be some risk exposure if two drives in the NAS died.
posted by seanmpuckett at 12:53 PM on September 19, 2011

I've been using unRAID for 4 years now. Just had a drive crash two weeks ago and didn't lose any data. Overall been very happy w/ it. If you can build your own unit I did it for $300 + the cost of how many hdd's you want. I scored two 2tb hdd's for $60each on newegg the other day so for $480 you could have 4tb's of storage and spiffy new server.
posted by no bueno at 1:04 PM on September 19, 2011

or you can buy one straight off their site for $700
posted by no bueno at 1:05 PM on September 19, 2011

« Older Help this part-time expat do I...   |  I need the perfect public doma... Newer »
This thread is closed to new comments.