Fastest way to transfer files between two external hard drives?
April 13, 2017 12:20 AM   Subscribe

I need to design a fast and cost effective storage solution for an on location video project. The video camera uses 2TB 2.5" SSDs, which hold about two hours of footage per drive. At the end of a day of shooting we typically have two of these 2TB drives to offload onto external storage. For data safety purposes, the external storage should be a RAID.

We have experimented with transferring from the SSD to an external 3.5" drive via a recent model MacBook Pro and USB3 enclosures for both drives, but this takes the entire night for 2TB of data (bottlenecked by the internal USB controller?). This speed is inadequate for our purposes. Since this is an on location project we are limited to the MacBook Pro.

What's the best external storage solution for our use scenario?
posted by BeaverTerror to Computers & Internet (8 answers total) 6 users marked this as a favorite
Similar question here
posted by Kwadeng at 12:58 AM on April 13, 2017

You are bottlenecked by the hard drive. Hard disks are only ~100MB/sec. A SSD can write 2-5x that rate. You want a SSD in a usb3 enclosure for your destination if you want anything close to native speed of the source ssd.

If you need a raid enclosure, raid slows things down, raid1 might be okay speed wise, but you'd have to find reviews that say if it can handle 500MB/sec not all raid controllers do, a lot of them were expecting HDD level performance. raid0 is the opposite of redundancy, raid5 is super slow.

2TB will still take a long time to copy.
posted by TheAdamist at 5:50 AM on April 13, 2017 [1 favorite]

You could pick up a 5 bay Drobo and load it up with drives. It looks like that Drobo will sustain ~200mb/s write. That would let you write 2tb in about three hours. That would run you about $600 for the Drobo and depending on the drives you pick out maybe $1700 for five 8tb drives. That kind of setup would give you 29tb of storage or if you're shooting 4tb/day enough space to shoot for 7 days. Having your SSD on USB and the Drobo on Thunderbolt might give you a slight speed advantage.

There are some crazy expensive options. That thing says it will write at 1300mb/s, but it looks like it's in the $20,000 range.
posted by gregr at 7:38 AM on April 13, 2017 [1 favorite]

Not an expert on video formats, but off hand a terabyte per hour sounds like a high data rate. Is there any compression that you could perform as you go to reduce the amount of data you eventually have to copy?
posted by floppyroofing at 7:46 AM on April 13, 2017

You need a Thunderbolt to SATA sled (to connect to the SSDs) and a Thunderbolt RAID.

This is a Thunderbolt RAID with 40TB storage (you'd want to probably run it as RAID5 (30TB) or RAID0+1 (20TB) if you can, for some modicum of data safety.

To run this particular enclosure in RAID modes, you'll need to have software RAID drivers.

Here's the Thunderbolt to SATA adaptor I use when I'm doing DIT work.

All of this assumes that your recent-vintage MacBook Pro has more than one Thunderbolt port.
posted by tomierna at 10:56 AM on April 13, 2017 [1 favorite]

If you need more space with the potential for faster throughput and hardware RAID, I suggest these Thunderbolt RAIDs. This is just the enclosure, you put drives in.

Configuring for RAID on these takes a LONG time, so give yourself a couple of days to populate, configure and format.
posted by tomierna at 11:01 AM on April 13, 2017

The Western Digital Black 6TB and Seagate Barracuda Pro 10TB mechanical drives will sustain 200MB/s writes, or about 700GB per hour. If you did a RAID 1+0, you might get a little more speed out of those.

That being said, even cheap small SSDs are twice that speed.
You could hardware RAID multiple SSDs, but I would also be somewhat cautious about consumer level hardware RAID enclosures. They break, and could leave your data in an unknown state if the individual disks aren't readable by the OS.

On some quick research for hardware RAID, it looks like it's Drobo (which in the past have had a reputation of dying with a non-accessible filesystem) or a generic alternative. I'm not up on whether there are Enterprise level enclosures that do this.

If you are willing to risk cheap hardware RAID, you could do with three 2TB SSD drives in a RAID 5, though you'd have to read some reviews to see what performance penalty RAID 5 might do. (You could also do four drives, using three in a RAID 5 plus hot spare, though your enclosure would need to support that configuration.) Another wrinkle is that a cheap enclosure may not be designed to pick up an SSD failure the way it does a mechanical drive failure. Again, research would be required.
You might be able to use Mac OS software RAID, which should have the advantage of a single disk being readable if the enclosure or second disk fails (but I don't know the enclosure part for a fact). To get more than 4TB, you'd need four 4TB SSDs ($1,400 each), since MacOS will only mirror drives.
Obviously, if you never need more than 4TB, two drives would do the job.

Be aware that NAS devices are going to be limited by the speed of your single Ethernet port (assuming you even have one), which is ~120MB per second.
If it were me and I had a lot of money, I'd probably go MacOS software RAID with SSDs, and I'd be inclined to test it by intentionally pulling a disk, then shutting the whole thing down and putting the remaining disk(s) into a separate enclosure. Connect via Thunderbold or USB 3.0.

The cheapest option is an external enclosure with fast(ish) mechanical drives and software RAID. In this scenario, 4TB transfers in 5.5 hours, absolute best case.
posted by cnc at 11:21 AM on April 13, 2017

Just a couple small notes, in the spirit of all of us need to remind each other to lift with the knees :-)

Raid is NOT safety, offsite backups are the ONLY digital safety. It can be in a drawer at a friends house but duplicate copies physically separated.

If you're buying anything, have you called the rental houses? Just saying, there's some fast hardware optimized for the problem and copying the two 2TB ssds on separate systems will beat the fastest big box.
posted by sammyo at 1:27 PM on April 13, 2017

« Older What are today's standards around casual sex?   |   How are days in a notice to vacate counted? Does... Newer »
This thread is closed to new comments.