Wanted: bombproof, redundant, fast, accessible data storage
April 8, 2013 3:46 PM   Subscribe

I have tens of thousands of photo, video and music files. I want to store them in a way that is reliable and rapidly accessible. More details inside.

What I have: Tens of thousands of image, video, and music files, many of them quite large. I am an amateur photographer, so most of my images files are in in RAW format; I like to have my music files in lossless format; and I am a musician, so I have a lot of large sound files (.wavs) that I use for recording. They are currently spread across several 1 TB drives and the hard drive on my 4-year-old macbook. i will be upgrading my computer (brand new MacBook Pro or desktop), and want to upgrade my storage options while I'm at it.

What I want: I want to be able to store all these files in one place, in such a way that they can be accessed as quickly as possible (e.g. by Ableton when playing back an audio composition, or when editing large photos in photoshop, without watching spinning beach balls all the time). Ideally, I would like to be able to access them from multiple computers over a home network (e.g. so that my partner can down- and upload photos on her own). I'd like the storage format to be as reliable as possible. Yes, I know that hard drives fail (been there, done that) and I will back up using Backblaze, but I'd also like the hardware itself to be dependable.

I'd like to buy one piece of hardware that will get this done, and will last for years (e.g. might be capable of upgrading as more space is needed). Price is not an issue. I assume that NAS is the way to go here (maybe something like this?), but I'm not sure. I'm computer literate and willing to learn something new, but not an expert, and somewhat overwhelmed by the options out there. What am I looking for?
posted by googly to Computers & Internet (11 answers total) 13 users marked this as a favorite
You want NAS for backup/access by remote computers, but you want Thunderbolt or eSATA interface for your working machine. Otherwise it would be achingly slow to work with a file that lives on your RAID.
posted by supercres at 3:52 PM on April 8, 2013

I wouldn't consider access to a NAS via gigabit ethernet to be "achingly slow". Older NASes with crappy CPUs did qualify for that description, but I wouldn't say that for newer ones.
posted by zsazsa at 3:59 PM on April 8, 2013 [4 favorites]

Best answer: Synology diskstation with RAID. something like a ds4xx with a few multi terabyte drives slapped in.

It's more than fast enough to handle intense, multi gigabyte database activity from multiple systems at the same time. They're not SSD fast, but the speeds generally seem to approach "good internal hard drive on a desktop" in general, with large single file copies sometimes going a bit higher.

They're not cheap, but also not horrendously expensive. be prepared to spend in the mid-high $100's on this. But out of the pile of them i've worked with, i've never seen one actually fail. They're easily upgradeable, seem to get software updates forever which have noticeably improved performance and added features, and just generally seem well made by a company that's not staffed with assholes. They also aren't storing the files on some weird proprietary BS system like a drobo. It's just ext2/3 standard linux stuff with a normal, actual raid controller. They're also capable of rebuilding a disk or the array in a reasonable human amount of time unlike some crappy small NAS units that have weak CPUs.

What i will say though, is that with ableton(which i have a lot of specific experience with) you really need a good local cache drive where it can unpack all of your samples so that it doesn't have to go back to the disk where the compressed(or even WAV) sample is stored. Ableton wants to unpack samples in it's own little raw way in to a folder that you can choose from the preferences menu. A setup like this will just make you angry without that, but then again so would any sort of external hard drive. I would probably count on getting max RAM and a larger internal disk for the ableton machine as part of this upgrade to your whole setup(and that's what, $100-120 now for 8gb of ram and a 1tb laptop internal drive? bonus points if you just get a 256gb ssd which is infinitely more satisfying)

A synology should meet, but not really exceed the speed of a good external drive plugged straight into the machine. That is a double edged sword for that specific use though.
posted by emptythought at 4:19 PM on April 8, 2013 [1 favorite]

I'd advise you to tally up how much you currently have stored, and try to estimate the rate at which your stash is growing. There is no way to give you a solid answer otherwise.

I can say that you'll probably want an x86 based option (none of the ARM based NAS options are likely to have enough power, though that will change soon)

Thunderbolt or eSATA aren't really an option for something that can also share the data with multiple machines. GigE could be a bit of a bottleneck for working with large media files. Local caching by some of the apps you are likely to use will help. If that isn't enough, a lot of the options will have the ability to aggregate 2 or more GigE links. You'll need a GigE switch that supports link aggregation with sufficient ports, and you'll need a client operating system and hardware that will support multiple GigE links.

A 4-bay Synology with dual LAN and an x86 CPU is $669 at newegg. Then you add the drives. 3TB WD Red drives are ~160 each. One will be used for parity, which gives you shy of 9GB of storage or 12GB with pricier 4TB drives. If that can hold you for a couple years you could expand your storage by swapping out drives. if it isn't then you'll need one of the larger models and you'll probably want to use one of the bays for a hot spare.

If it were me and I needed more than 4 bays, I'd probably conclude that my best bet was a small server, running Linux, or FreeNAS withn ZFS or straight FreeBSD, but that is because I have have sysadmin experience, don't place enough value on my time, would want to try using an SSD cache drive.
posted by Good Brain at 5:04 PM on April 8, 2013

Response by poster: Thanks for the answers so far!

Just to clarify: I only need network access for low-intensity tasks such as up- and downloading pictures. Intensive use of large files ala Ableton or Photoshop would be done via a direct (Thunderbolt or eSATA) connection.

Tough to estimate the growth rate on my stash, as it has been limited up to this point by the knowledge that I have finite storage capacity. I need approximately 2.5 TB right now; maybe an additional .75-1TB/year in the future?

All that said, the 4- or 5-bay Synology NAS looks good, and the price range is acceptable. And that's probably at the upper end of my technological sophistication.
posted by googly at 5:10 PM on April 8, 2013

You're not going to find a device that lets you connect both over the network and via eSATA/Thunderbolt.

One could imagine one that would let you connect via one or the other, but to my knowledge none exist. One that would let you connect via both simultaneously is out of the question - it'd involve multiple devices mounting the volumes which is unheard of outside of enterprise-level stuff.
posted by zsazsa at 5:33 PM on April 8, 2013

Best answer: Gigabit ethernet is not going to be a bottleneck with video files. I have a setup where a network attached tuner device can serve three uncompressed HD streams to a single PC without ever really breaking its stride, network utilization-wise. Single core ARM might be sluggish, but dual core should be fine. A single purpose machine like that doesn't really need a lot of processing power. RAID parity calculations and Samba file serving really don't take all that much CPU at all. The important thing is the throughput of the busses- you want DDR3 ram and a SATA controller that has plenty of bandwidth. I've got an an old DDR2 core2duo machine hosting a 3tb RAID and three virtual machines and it only ever bogs down if all three of the VMs are hitting the disk hard.

If you had 10 users all editing video off the network, yeah, maybe. But for home use, gigabit ethernet is fine. Even wireless N should be adequate, provided your WAP is strong enough to handle the throughput.

If price is no object, then yes, by all means buy something like that 4 bay sysology. I built myself a homebrew setup like that, and it has served me well. I usually buy hard drives when they are on sale or when one fails, and usually it sort of works out that I have enough space.

The very important thing is this:

If your hardware fails, will you be able to cram those drives into another enclosure and see your data? Backup is great, but not needing it is even better.

Secondly, make sure whatever you choose does two things: alerts you loudly when it has a failing drive, and has a cron job to do a raidcheck once a week. You don't want to miss a failed drive and then find a bad spot on a second drive. No es bueno.

As for hard drive purchases, I use WD greens in mine. They are cheap, run cool, don't use a lot of power and are plenty fast enough. But they don't always play nice with NAS solutions, so make sure the device you choose likes them before you buy any. I also buy whatever drives are the cheapest per GB. I think that's 2tb right about now. 4 of those will get you 6TB, which should last you two years. In two years, 4tb drives will probably be at the price sweet spot.
posted by gjc at 5:44 PM on April 8, 2013

As with many things in life, there are three characteristics to this solution and you can only choose two.

1. Highly safe/reliable
2. Fast access
3. Cheap.
posted by DWRoelands at 8:12 PM on April 8, 2013 [1 favorite]

While the ioSafe comes close none of these are strictly speaking bombproof. Wouldn't you want some sort of online cloud backup to supplement your NAS? Like crashplan or Amazon S3 or something?

ok ok the title was probably a joke but I'm just saying...
posted by Wretch729 at 8:34 PM on April 8, 2013

Please pay attention to emptythought's answer RE: Ableton. I would highly recommend NOT using a Synology NAS for your .als and .wav files.
posted by kuanes at 6:55 AM on April 9, 2013

Response by poster: Thanks for all the great answers, everyone! Looks like I'm going with the Synology 412+ and 4 2TB drives, and will make sure to get a big SSD with my next computer to handle Ableton .wavs.
posted by googly at 7:48 AM on April 9, 2013

« Older YANML: After disability, can't do a single aspect...   |   Flip flop help, please Newer »
This thread is closed to new comments.