Sixteen GoPros per concert seemed like a good idea at the time

April 25, 2018 9:06 PM

What is the least expensive yet still reliable way to back up 200 terabytes of audio and video files?

I am working with Macs. It is mostly just for archival backups, although I will occasionally take a folder out to edit a video. When I do that, I will not use the hard drive as I edit. I'll just load the files onto my computer in order to minimize wear and tear on the drive.

posted by umbú to Computers & Internet (8 answers total)

How backed up do you need it to be? Are you going for a single copy, or triplicate? Will your data volume grow over time, or are you just trying to figure out how to handle a fixed amount that you already have? Is it just you working with this, or will you need multiple people able to access the data?

Assuming that it's just you and a fixed 200TB dataset, you're right around the crossover point for tape / spinning disk for a single copy, and well over it for a redundant backup system. Tapes are a bit of a pain to use, but your workflow sounds pretty compatible with them.

Ballpark numbers:
* Hard drives will run you ~$30/TB.
* LTO-7 tape will run you ~$3,000 for the drive, and then ~$10/TB for the media.
posted by Metasyntactic at 9:30 PM on April 25, 2018 [2 favorites]

Tape.

Or head over to your local server provider and order a 500TB unit, which is the size you need to reliably store 200TB of data (mirrored), while still leaving a small bit of room for future data. It will be the size of a small fridge and cost as much as a car, but you will be able to edit off of it directly.

If you are making money off of your work, this is the standard cost of doing business. If it sounds like incredible overkill, then I assume this is just a hobby. In that case you should really question why you need to preserve that much data in the first place. Not everything in the universe needs to be saved forever. Finish your project and toss the files.
posted by BeaverTerror at 3:17 AM on April 26, 2018 [4 favorites]

Cameras don't usually generate very well compressed video. Reencoding with a modern codec could potentially cut down the size tremendously, with little impact to quality. Storing 200TB in the Cloud would run you $800/month on Glacier, plus transfer costs.
posted by qxntpqbbbqxl at 7:48 AM on April 26, 2018 [1 favorite]

To put some numbers on qxntpqbbbqxl's comment, I just reencoded two videos using handbrake. One that came directly from my iPhone went from 42 MB down to 16 MB. The other had come out of iMovie and went down from 27 MB to 2 MB. So we're talking a factor of 3 in the first case and a factor of 7 in the second case. This is without any degradation to my (admittedly casual) eye.
posted by Winnie the Proust at 8:35 AM on April 26, 2018

For the curious, cloud storage doesn't look like an economical option. For 200 TB Amazon Glacier comes out to about $800/month for 200 TB, and even Wasabi comes out to $500/month. Even if you got it down to 20 TB, Wasabi would still cost $100/month and Glacier would be $80/month.
posted by Winnie the Proust at 8:41 AM on April 26, 2018

Big storage. To get to 200 TB you will need to choose a reasonably priced storage tier, like a 2 or 4 TB hard drive, to keep your cost per GB down. For an off the shelf solution you are looking at a storage array or pod, like a Dell Equallogic or HP Storageworks. Those come in a variety of configurations but you can pick up a refurbished 48 TB unit like this HP one for just under 3 grand, which pushes your total around 12 grand, plus a rack.

For that sort of money you might consider a pod like the ones Backblaze has developed. You can pick up a previous generation kit for about 2.5 thousand and then fill it the 45 odd hard drives, which you should be able to grab for around 150 a piece, for a total of just over 2grand for the drives. You'll be out the door for less than 5, and then only have to pay the energy bill when you fire it up.
posted by zenon at 10:03 AM on April 26, 2018

Suggest an external storage device, either directly connected to the computer or on your network. You may want to simple compress the files for storage. Then when you need to use a file move back to the computer and uncompress. This should preserve the quality of the original file and save disk while being stored.
posted by tman99 at 10:07 AM on April 26, 2018

Best answer: The people at r/datahoarder might have some other suggestions. This is what they apparently live for.
posted by OmieWise at 8:00 AM on April 27, 2018 [1 favorite]