Options for secure data storage?

June 8, 2005 7:42 AM

What are my best options for safely storing in excess of 700Gb of data?

Background: I'm involved in a project where we have between 700 and 800 Gigabytes of data we need to make sure isn't lost. The data is made up of the source materials for an independent feature film -- including various rough cuts of the film, plus all the source video, audio, and graphic files. Since this is our first feature length project, we've never had a need to secure as much data. Foir shorter projects we've been able to burn a series of CDs or data DVDs to save in a safety deposit box. Since that isn't a good solution for our current project, we need some suggestions. I tend to think some sort of offsite, colocated data storage company would be our best bet, but have no idea where to start or which storage companies are reputable.

At the moment, the data is just sitting on the array of hard disks in our editing studio, so the sooner we can get it secured the better. FWIW, the original rolls of film and color corrected transfer tapes are in a safety deposit box, and we do temporary backups of sections of the project on data DVDs as we go along. I just don't think that's good enough. Do you have recommendations for specific data storage companies or solutions that have worked for you?

posted by ScottUltra to computers & internet (12 comments total)

Wouldn't the cheapest solution be to buy a couple of large HDs, copy the data, and store the HDs in a safety deposit box?
I remember an article about some Microsoft bigwig who needed to ship like a TB of data and figured out that the cheapest and fastest way was to buy a computer, copy the data and fedex it to where it needed to be, as the cost and speed of transfering that much data over the internet would wbe much worse.
posted by signal at 7:57 AM on June 8, 2005

You could get a tape autoloader off of Ebay, dump the data to tape, and then ship it off to an offsite, climate controlled facility run by a company like Iron Mountain.

If the data is that important, you don't want to just rely on hard drives or CDs.
posted by cmonkey at 7:59 AM on June 8, 2005

I second tape backups, harddrives are very senstive, and one little scratch on that DVD and that full frontal nudity scene youve been saving is gone.
posted by Mach5 at 8:19 AM on June 8, 2005

if you back it up to disk, don't leave the disks online. this may be obvious, but i know someone who did leave them online. they were hacked and *everything* erased.
posted by andrew cooke at 8:20 AM on June 8, 2005

thats probably too much data to transfer over the internet. it sounds like you should probably invest in a tape backup system, hard drives would work but long term idle storage isnt what they are made for, since they have moving parts.

when i think storage, i think storagetek or emc, though the more end-user friendly companies like Dell, HP, or IBM might be easier to deal with, and if you find a decent salesperson they could at definitely give you and idea of whats out there. it wouldnt hurt to chat with IBM and Dell sales folks and then just verifying and checking out their ideas on the web.
posted by yeahyeahyeahwhoo at 8:26 AM on June 8, 2005

Tape backup might not be a bad idea ... any suggestions for Mac-friendly (OSX) choices?
posted by ScottUltra at 8:54 AM on June 8, 2005

Backing up to tape is the most common type of backup for large amounts of data. However, tapes themselves are not a guaranteed solution. Depending on the age of the tapes and the vendor/format, there is risk. If you go with tape, stay away from Exabyte Mammoth 2 units. They are absolute crap. SDLT is pretty good, and currently I have Sony AIT units that work well. Given the amount of data you want to back up, you will need a library unit, unless you plan on being there to change tapes.

Other questions I would ask, are how exactly do you plan on backing the data up? I assume when you burned to DVD/CD it was data in its native format. That will not be the case when backing up to tape. What kind of budget do you have for disaster recovery?

Seeing that the data is on an array is good. You should make sure you have some spare drives for the array on hand in the event you lose a drive. You don't want to lose another drive waiting for a spare to show up.

Good luck with it.
posted by a3matrix at 9:50 AM on June 8, 2005

Scott,

Realistically I think you can do one of two/three things.

Here's how the industry generally does this:

Back up ONLY the graphics etc.

You have a backup of the footage- the TAPES. That's why timecode is so important.

Especially if it came from telecine tapes.

Need to restore the show? Take the sequence, batch digitize the tapes, add the graphics.

Much smaller than trying to store the video.

-----------
A pair of terabyte firewire drives (lacie has a terabyte for $$800)

Why two? redundancy.

---------------
Buy a firewire Raid 5 (has redundancy built in in case a drive goes down.) Buy two.

That's a start.

DVDs are too small...tapes are too slow.
posted by filmgeek at 10:06 AM on June 8, 2005

Raid 5 is not designed for backup, it's designed for uptime.

My suggestion is two 1TB firewire drives, stored in different physical locations. One of them should be somewhere secure, such as a safe, or a safety deposit box. The other one should be somewhere accessible for an easy restore.
posted by Jairus at 10:14 AM on June 8, 2005

filmgeek & Jairus - Our tapes are telecine, so we've discussed doing exactly what you propose. Our 2 main concerns about this method revolved around (1) storing the data on drives with moving parts, and the risk of data corruption/loss that comes with transporting the drive to a storage facility, and (2) the time necessary to rebuild the entire edit from timecode in case we lose everything. Redundant backups would ease our worries a bit, and I agree that backing up to tape seems slow and painful. I appreciate your input, and think we'll probably go this route.
posted by ScottUltra at 10:24 AM on June 8, 2005

I had heard that apple is actually coming out with an iPod that can store that data for you, AND it will let you view and edit it in real time. Of course, the downside is that it will not support OGG Vorbis...
posted by indiebass at 10:33 AM on June 8, 2005

The dual terabyte drives sound good to me. Pack them inside a strong case with vibration dampening foam. Bring one drive online every six months to verify that the contents are still in good order. Keep both drives in different physical locations.

Since you are worried about data corruption, buy a MacMini or ibook (cheapest you can get), and take it with you as you transport each of the drives. When you get to the storage facility, do a sanity check on the data. If the data is bad (unlikely), return the drive for a replacement.

Unless your film is stored in the middle of baghdad, and you your offsite locations are in afghanistan, at least one of your drive should survive the trip to the offsite location. Most likely both will survive with no trouble. If they both survive, then you can again reasonabally expect at least one of them to make it back to the home site, in the event that you need to do data recovery.

I used to carry a dell laptop, 3.5" desktop hard drive in a firewire case, and a CDRW drive in my backpack. I took them all over seattle in cars, on my back, on busses, in taxis, etc. The drive survived with no data loss and is still in use today. I think 3.5" drives typically lock the heads down when the drive is powered off. Sometimes you can even hear a "thunk" as you power the drive on/off.

Other people I know regularly transport powerbooks on motorcycles. If you are extra paranoid, build an array of 100 gb laptop drives, as they are designed to be even less sensitive to vibration.
posted by b1tr0t at 12:16 AM on June 9, 2005