Optimizing an external hard drive
February 3, 2011 8:13 AM Subscribe
Let's say I want to buy a new external hard drive. I can pick a case with a particular interface (USB2/USB3/FW400/FW800/eSATA) and a mechanism with certain performance characteristics (RPM, latency, transfer rate, etc). Is there a straightforward heuristic that will tell me "if you're using interface X, don't bother buying a mechanism faster than Y."?
I do know that an interface's putative speed doesn't necessarily translate to effective speed—I've always found FW400 to be faster in use than USB2, even though the latter has a higher nominal throughput.
I do know that an interface's putative speed doesn't necessarily translate to effective speed—I've always found FW400 to be faster in use than USB2, even though the latter has a higher nominal throughput.
Wiki has a breakdown of theoretical speeds for each of the different interfaces. Mister Fabulous thoroughly explained it. As you've noted, most of these are theoretical values and do not match up with actual interface speeds, which are dependent upon protocol, controller and other factors.
I think certain hard drive manufacturers like the GoFlex line have external hard drives with modular "docks" so that if you buy a hard drive now with USB 2.0, you can simply buy the USB 3.0 dock when it comes out.
The major problem with the newer external interfaces is that they're not standard on most computers yet. Even eSATA, which has been around for a while, is a premium feature. However, USB3 and FW3200 are backward compatible with their previous generation so you don't have to worry about not being able to access the drives outside of your own computer whereas you would with eSATA.
Newer drives are generally faster than older ones, even comparing newer 5k to older 7200 rpm drives so this doesn't serve as a very good point from which to start. It's easier to take a look at the actual performance of the drive, but in general spinning disks all fall within a similar range that saturate USB2 and are full speed at USB 3
posted by palionex at 4:42 PM on February 3, 2011
I think certain hard drive manufacturers like the GoFlex line have external hard drives with modular "docks" so that if you buy a hard drive now with USB 2.0, you can simply buy the USB 3.0 dock when it comes out.
The major problem with the newer external interfaces is that they're not standard on most computers yet. Even eSATA, which has been around for a while, is a premium feature. However, USB3 and FW3200 are backward compatible with their previous generation so you don't have to worry about not being able to access the drives outside of your own computer whereas you would with eSATA.
Newer drives are generally faster than older ones, even comparing newer 5k to older 7200 rpm drives so this doesn't serve as a very good point from which to start. It's easier to take a look at the actual performance of the drive, but in general spinning disks all fall within a similar range that saturate USB2 and are full speed at USB 3
posted by palionex at 4:42 PM on February 3, 2011
It also depends on a couple of other things:
1- The processing power of the external enclosure circuitry. (I don't know if this is a real-world issue, of just theoretical. But it is real in other arenas- wireless routers, for example, can get bound up by their processing power, not the external links.) I've seen this with USB flash drives- the USB connects up at USB2 just fine, and random, quick reads and writes are fast, but sustained drops like a rock. The little chip in there that translates the flash information to USB just can't handle it. So make sure whatever you buy seems good, or has been benchmarked and doesn't have any obvious flaws.
2- Your computer's processing power and internal bus structure. USB uses up processor time, where firewire doesn't. So on a slower computer, firewire will seem faster. Because your bottleneck isn't the speed on the wire, but the ability of the computer to process the data fast enough.
3- The internal bus structure of the computer's motherboard. Every connector on the computer is connected to the processor and memory by a data bus. Which is just a logical "cord" that connects them up. Older, cheaper computers, for example, had some of the external ports wired up to the old ISA bus, because that was easier and cheaper to design. So even though everything else was fast enough, the data passing internally was bottlenecked. I had one motherboard that touted onboard RAID, back when that was a novelty. Except, performance sucked. After doing some research, I found that it was wired up to the PCI bus, and so it was sharing bandwidth with the NIC and all the other ports. Whereas, the standard, non-raid controller was right on the main controller chip and had an internal, native connection to everything.
You can't control most of that, of course, but that's where performance mysteries turn up. The best answer is to buy the fastest you are willing to pay for and hope for the best.
posted by gjc at 7:42 AM on February 4, 2011
1- The processing power of the external enclosure circuitry. (I don't know if this is a real-world issue, of just theoretical. But it is real in other arenas- wireless routers, for example, can get bound up by their processing power, not the external links.) I've seen this with USB flash drives- the USB connects up at USB2 just fine, and random, quick reads and writes are fast, but sustained drops like a rock. The little chip in there that translates the flash information to USB just can't handle it. So make sure whatever you buy seems good, or has been benchmarked and doesn't have any obvious flaws.
2- Your computer's processing power and internal bus structure. USB uses up processor time, where firewire doesn't. So on a slower computer, firewire will seem faster. Because your bottleneck isn't the speed on the wire, but the ability of the computer to process the data fast enough.
3- The internal bus structure of the computer's motherboard. Every connector on the computer is connected to the processor and memory by a data bus. Which is just a logical "cord" that connects them up. Older, cheaper computers, for example, had some of the external ports wired up to the old ISA bus, because that was easier and cheaper to design. So even though everything else was fast enough, the data passing internally was bottlenecked. I had one motherboard that touted onboard RAID, back when that was a novelty. Except, performance sucked. After doing some research, I found that it was wired up to the PCI bus, and so it was sharing bandwidth with the NIC and all the other ports. Whereas, the standard, non-raid controller was right on the main controller chip and had an internal, native connection to everything.
You can't control most of that, of course, but that's where performance mysteries turn up. The best answer is to buy the fastest you are willing to pay for and hope for the best.
posted by gjc at 7:42 AM on February 4, 2011
This thread is closed to new comments.
If you are using USB 2.0 or FW400, the interface is the limiting factor. The lowest end of hard drives available should push at least 70-80MB/s (the "green" 5400 RPM hard drives, older generation hard drives). This is above the interface throughput of both USB 2.0 (60MB/s) and FW 400 (50MB/s).
FW800 is mixed. It will be a slightly limiting factor as the throughput is around 100MB/sec, so it can slow down a hard drive at the top end of hard drives, but won't slow down the bottom end. If you used an external on FW800 you probably wouldn't notice the difference between it and the next group of interfaces.
USB 3.0, eSATA, SATA I, SATA II, SATA III, FW1600 and FW3200 connections are not limiting factors. That is, all hard drives are slower than any of these interfaces (the slowest interface is SATA I at 150MB/s, which is faster than the top end of common consumer hard drives at 135MB/s).
I haven't touched on latency, but for hard drives vs interface it's kind of moot. If we are talking about sustained transfers (the kind that will hit the maximums) then we don't notice latency as the hard drive starts reading and just continues with no latency. If we are talking about unsustained transfers (such as many small files, heavily fragmented files or drives or the 4K random tests often done on benchmarks), then no hard drive will come remotely close to the throughput on any interface. For any external hard drive, the lower the latency, the better.
If you change the discussion here to Solid State Drives, all of the above goes out the window. Most people won't be using a SSD for an external drive due to the price (typically $1.50-$2.00/GB for SSD instead of $0.06-0.15/GB for HDD). SSD's fully saturate the throughput on USB 2.0, FW400, FW800. Current SSDs nearly saturate eSATA. The newest SSDs that will be coming out over the next few months will saturate USB 3.0, FW1600 and FW3200. It all depends on the speed of the SSD, which is generally listed as one of the specifications.
posted by Mister Fabulous at 9:27 AM on February 3, 2011 [5 favorites]