How to speed up a file transfer between two external hard drives?
October 21, 2009 1:11 PM Subscribe
Why is it that when I transfer files from an external hard drive to my laptop's hard drive via USB 2.0, the speeds are so much faster than transferring files from one external hard drive to another (5 mins vs. a few hours for 1.5 GB)? I think it might have to do with the fact that the external drives use different file systems (NTFS vs. FAT32). Is there any software that could speed things up?
USB is a shared bus, so you're halving the throughput to begin with.
posted by djb at 1:15 PM on October 21, 2009
posted by djb at 1:15 PM on October 21, 2009
The filesystem difference has nothing to do with the speed.
If both drives are connected to the same root hub (same pair of ports on many motherboards), their bandwidth will be severely crippled. Even if they're not on the same root hub, USB is just overall a relatively inefficient protocol and slinging that much data around is going to result in slowdowns.
TeraCopy is probably a good program to try. It tries to be more efficient in the way it reads and writes, which should lower the time it takes to copy.
posted by Rendus at 1:36 PM on October 21, 2009
If both drives are connected to the same root hub (same pair of ports on many motherboards), their bandwidth will be severely crippled. Even if they're not on the same root hub, USB is just overall a relatively inefficient protocol and slinging that much data around is going to result in slowdowns.
TeraCopy is probably a good program to try. It tries to be more efficient in the way it reads and writes, which should lower the time it takes to copy.
posted by Rendus at 1:36 PM on October 21, 2009
The drives that are sold in USB enclosures are typically not high performers, and doing bulk transfer between two USB devices attached to the same controller is pretty much going to suck no matter what software you use, but it might help if you described how you're currently performing the copy.
posted by scatter gather at 1:37 PM on October 21, 2009
posted by scatter gather at 1:37 PM on October 21, 2009
Why not jsut copy to your internal hdd then out to the other external hdd from the internal hdd.
posted by majortom1981 at 1:51 PM on October 21, 2009 [1 favorite]
posted by majortom1981 at 1:51 PM on October 21, 2009 [1 favorite]
Your bottleneck is USB which is slow. Copying to or from the internal drive only involves one trip over USB. Using two external drives involves two trips. There's not much you can do to speed it up.
posted by chairface at 3:03 PM on October 21, 2009
posted by chairface at 3:03 PM on October 21, 2009
"The filesystem difference has nothing to do with the speed."
Not necessarily true: A common tool for writing to NTFS volumes with OSX (and BSD, possibly Linux now that some of the other drivers are abandoned) is the FUSE driver ntfs-3g -- which reads like lightning but is sometimes abysmally multi-hour painfully turtle slow on large writes. The filesystem very much can be a factor on non-Windows platforms, and the platform hasn't been stated.
So hey, matkline: can you tell us a tiny bit about the system with which you're having this problem?
posted by majick at 3:09 PM on October 21, 2009
Not necessarily true: A common tool for writing to NTFS volumes with OSX (and BSD, possibly Linux now that some of the other drivers are abandoned) is the FUSE driver ntfs-3g -- which reads like lightning but is sometimes abysmally multi-hour painfully turtle slow on large writes. The filesystem very much can be a factor on non-Windows platforms, and the platform hasn't been stated.
So hey, matkline: can you tell us a tiny bit about the system with which you're having this problem?
posted by majick at 3:09 PM on October 21, 2009
The listed transfer times (5 minutes vs. several hours) seem way out of proportion to me. The USB protocol alone couldn't account for that much of a bottleneck, could it?
Try running a couple of tests:
1) Compare the times when copying a large file from each of the external drives to the internal drive (read test on the external drives). These times should be roughly the same.
2) Compare the times when copying a large file from the internal drive to each of the external drives (write test on the external drives). These times should also be roughly the same.
3) When you copy between external drives, is it always in one direction (i.e., drive A to drive B)? If so, what happens when you reverse the direction?
If any of the times are grossly larger than the others, that's a good pointer to the problem.
One last thought: when you copy from one external drive, you're using one USB port. When you copy between external drives, are you using a hub or two ports on your laptop? If the latter, are you sure the second port is USB2?
posted by joaquim at 3:41 PM on October 21, 2009
Try running a couple of tests:
1) Compare the times when copying a large file from each of the external drives to the internal drive (read test on the external drives). These times should be roughly the same.
2) Compare the times when copying a large file from the internal drive to each of the external drives (write test on the external drives). These times should also be roughly the same.
3) When you copy between external drives, is it always in one direction (i.e., drive A to drive B)? If so, what happens when you reverse the direction?
If any of the times are grossly larger than the others, that's a good pointer to the problem.
One last thought: when you copy from one external drive, you're using one USB port. When you copy between external drives, are you using a hub or two ports on your laptop? If the latter, are you sure the second port is USB2?
posted by joaquim at 3:41 PM on October 21, 2009
A few things:
SATA drives (SCSI too), support having multiple disk operations "in flight" at once, drives take advantage of this + write caching to reorder writes to optimize performance. I'm pretty sure this is not supported with USB connected drives.
Second, for data safety reasons, write caching is usually disabled by on external drives, while it is used on internal. With the caches on, writes can happen at something close to the sustained write speed of the drive, with caches off, it can take a brutal toll on performance. I suspect the performance hit is even greater for a journaled file system like NTFS where each write of data involves one or two writes to the journal. FAT32 isn't journaled.
Third, if both drives are connected to the same USB controller via a hub, they will be competing for bandwidth. At best, this is going to halve throughput, and depending on how contention is dealt with, it could be even worse.
posted by Good Brain at 4:37 PM on October 21, 2009
SATA drives (SCSI too), support having multiple disk operations "in flight" at once, drives take advantage of this + write caching to reorder writes to optimize performance. I'm pretty sure this is not supported with USB connected drives.
Second, for data safety reasons, write caching is usually disabled by on external drives, while it is used on internal. With the caches on, writes can happen at something close to the sustained write speed of the drive, with caches off, it can take a brutal toll on performance. I suspect the performance hit is even greater for a journaled file system like NTFS where each write of data involves one or two writes to the journal. FAT32 isn't journaled.
Third, if both drives are connected to the same USB controller via a hub, they will be competing for bandwidth. At best, this is going to halve throughput, and depending on how contention is dealt with, it could be even worse.
posted by Good Brain at 4:37 PM on October 21, 2009
I don't think it's especially clear what peopel are saying about USB being a shared hub.
Your PC might have 6 usb slots, but chances are it only has 2 or 3 channels. Each "channel" can support (theoretically) up to 128 devices, but on each "channel" the total bandwidth available is (theoretically) shared between the other devices on that channel.
Ergo...
If you've got two drives plugged in to adjacent USB ports, you're likely on the same channel, which is owning you. Try switching one to a far away port (such as one on the front and one on the back)....of course, sometimes the front ports are simply splits off one of the ones in the back.
Also, windows SUCKS at file transfer when you try to move 2 files simultaneously. You know, you go to copy 1 movie and it says "3 minutes..." and then you grab another and now both say "4 hours...". This is something that Teracopy does better than windows. It channels the data differently, AND it doesn't fail like the windows transfer can sometimes, and if it DOES, it's like an FTP client in that it's able to pickup where it left off.
I have a cheap WD 250Gig external hdd that was $60 at walmart. My write times to it are really pretty surprising, especially for a fairly low spec drive.
Try Teracopy, try making sure they're on different channels, and if that fails, copy to an intermediary drive. (external to C, then C to external.)
posted by TomMelee at 6:48 PM on October 21, 2009
Your PC might have 6 usb slots, but chances are it only has 2 or 3 channels. Each "channel" can support (theoretically) up to 128 devices, but on each "channel" the total bandwidth available is (theoretically) shared between the other devices on that channel.
Ergo...
If you've got two drives plugged in to adjacent USB ports, you're likely on the same channel, which is owning you. Try switching one to a far away port (such as one on the front and one on the back)....of course, sometimes the front ports are simply splits off one of the ones in the back.
Also, windows SUCKS at file transfer when you try to move 2 files simultaneously. You know, you go to copy 1 movie and it says "3 minutes..." and then you grab another and now both say "4 hours...". This is something that Teracopy does better than windows. It channels the data differently, AND it doesn't fail like the windows transfer can sometimes, and if it DOES, it's like an FTP client in that it's able to pickup where it left off.
I have a cheap WD 250Gig external hdd that was $60 at walmart. My write times to it are really pretty surprising, especially for a fairly low spec drive.
Try Teracopy, try making sure they're on different channels, and if that fails, copy to an intermediary drive. (external to C, then C to external.)
posted by TomMelee at 6:48 PM on October 21, 2009
This thread is closed to new comments.
posted by TomMelee at 1:13 PM on October 21, 2009 [2 favorites]