Batch transfer of ~1TB from 8 drives over LAN

October 18, 2009 11:03 AM

Are there any better tools for transferring about a terabyte of files over a local network than just dragging and dropping in Windows? Specifically, is there a tool that can pick up and double-check files once they've been copied over?

I'm moving all my stuff to a new NAS. Lots of stuff scattered all over a lot of computers, and I'm wondering if there's a way to "automate" this, rather than just drag and drop. As I'm doing some of these transfers over wireless, the network occasionally drops, and the transfer gets interrupted, which means I have to start all over again. I'd love to find a software with some kind of "transfer resume", so I can just start it and forget it instead of checking it every hour or so.

posted by saysthis to computers & internet (15 comments total)

Unison might work for you.
posted by Monday, stony Monday at 11:12 AM on October 18

Never used it myself, but I've heard good things about TeraCopy.
posted by aheckler at 11:15 AM on October 18

MS makes a tool for this called Robocopy.

Besides allowing restart and verification it also makes sure all the NTFS attributes are copied properly kif that is desired.
posted by Mitheral at 11:22 AM on October 18 [2 favorites]

I use TeraCopy for almost everything. It's saved my bacon more than once.
posted by niles at 11:39 AM on October 18

As a unixy-nerd, I would always use rsync, though I don't know if it can be run without installing Cygwin.
posted by bsdfish at 11:51 AM on October 18

xcopy /S C /O /X /-Y /Z /B /J e:\project\ \\server2\share\
posted by SirStan at 11:54 AM on October 18

2nding TeraCopy.
posted by deezil at 12:05 PM on October 18

I have found that Robocopy is sometimes more reliable on windows than various rsync ports, especially if you have pathnames that exceed 255 chars. (I don't really understand Windows internals, but I think the deal is that NTFS supports 32k-character pathnames but certain parts of the Windows API only support 255, meaning that depending on how an app is written, it may choke on a file even though it's technically under the NTFS limit.) I have seen it used to copy terabytes of data over fairly small interfacility network connections, running overnight or on weekends unattended.

I've only ever used it to copy from one Windows-based host to another, never from a Windows machine to a non-Windows one running SAMBA or similar. (I am assuming your NAS runs Linux or some a proprietary embedded OS, not Windows.) Depending on how it implements SMB, this may be much slower than installing rsync on your Windows machine and doing the transfer that way, just due to protocol overhead and lack of compression.

It'd be worth giving it a try anyway, though.
posted by Kadin2048 at 12:06 PM on October 18

Your best bet is Robocopy, IMO.
posted by velvetsmog at 12:20 PM on October 18

Microsoft's own SyncToy.

XXCopy - most flexible CLI copier on Windows.
posted by meehawl at 12:31 PM on October 18

Saysthis, please excuse the piggyback: with all the united computing power concentrated in this thread would anybody know how to solve the exact same problem on a mac?
posted by mathiu at 1:13 PM on October 18

would anybody know how to solve the exact same problem on a mac?


On a mac, it's simple.

rsync -avP /path/to/source/directory /path/to/target/directory
posted by pompomtom at 3:47 PM on October 18 [1 favorite]

(and then I like to run it a second time, for voodoo...)
posted by pompomtom at 3:48 PM on October 18

Teracopy, or Richcopy. The latter, amongst other things, is multithreaded.
posted by tra at 6:17 PM on October 18

If I had to do this kind of transfer, I would use rsync for Windows that comes with Cygwin. If you want to avoid the command line, you can try a clickety version of rsync called DeltaCopy. Both of these are free, open source, GPL programs.
posted by gmarceau at 10:44 PM on October 18