Verifying copy utility?

October 5, 2005 12:04 PM

Good file copying utility that verifies integrity?

When I transfer files (specifically compressed ones) from my computer over the network to my housemate (both running XP), they always end up corrupted and unable to decompress. I'm guessing this happens with all the files that I transfer but image and movie formats are more forgiving than .iso and .zip files. Do you know of any copying utility that checks the integrity of the files as they're copying? Windows should do this automatically but experience has shown otherwise.

posted by pantsrobot to Computers & Internet (13 answers total)

Response by poster: This could be caused by my rapidly failing hard drive, but I'm able to play with the files fine on my own computer
posted by pantsrobot at 12:06 PM on October 5, 2005

Best answer: My solution for any kind of copying problem is always rsync. When run on the command line with the --checksum flag, it should catch any file integrity issues.

Though I haven't used it personally, it does appear to exist for Windows.
posted by Eamon at 12:44 PM on October 5, 2005

Automatically, no, but you can still md5sum it at either end (there is an md5 EXE for Windows, somewhere).
posted by cyrusdogstar at 1:10 PM on October 5, 2005

Seconding rsync. As a bonus, you can use it to "heal" files corrupted in transit without transferring the whole file over again. Getting a server working on Windows was kind of a pain when I tried about five years ago; maybe it's easier now that Cygwin has matured. This seems to be a decent tutorial.
posted by zsazsa at 1:35 PM on October 5, 2005

Best answer: Unison is a wonderful cross-platform file synchronizer and verifier. It is basically the modern replacement for Rsync, using a more efficient algorithm and without requiring a listening demon. Instead it uses SSH transport for greatly enhanced security and reliability.
posted by Invoke at 1:37 PM on October 5, 2005

Note, no special server is needed for Unison to work, just SSH and the unison executable on both sides. This is much easier to set up on Windows.
posted by Invoke at 1:38 PM on October 5, 2005

Invoke, rsync operates over ssh as well. The only time you'll see an rsyncd these days is on anonymous "rsync servers" used for wide distribution, like mirroring a Linux distribution.
posted by mendel at 1:51 PM on October 5, 2005

In a Windows Command Prompt you can use the command copy /v sourcefile targetfile which does a verification after copying; use '*' as a wildcard. Likewise xcopy /v if you want to copy subdirectories. Typing copy /? (or xcopy /?) will show you the command usage.
posted by anadem at 1:56 PM on October 5, 2005

Response by poster: I like that Unison has a GUI; don't want to install Cygwin on housemate's computer. I tried to use SCP once but the transfer would fail when it hit something bad (i.e. it would just stop, rather than trying to redownload the bad part), will rsync and unison work the same way?
posted by pantsrobot at 2:32 PM on October 5, 2005

I'm not sure about what it will do in your case with iffy connectivity. I think at the worst it will skip the problem file.

I note the manual mentions this argument:

-retry n (retry failed sync n times)
posted by Invoke at 2:50 PM on October 5, 2005

Response by poster: argh from here
Using /v : Windows XP does not use this command. It is accepted only for compatibility with MS-DOS files.
posted by pantsrobot at 3:36 PM on October 5, 2005

one of you could have bad ram/overclocking/something else.

I used to have the same problem with two older computers, now with two newer computers, no problem.
posted by Iax at 1:02 AM on October 6, 2005

PAR2 lets you generate error correction files that will cope with fairly arbitary levels of corruption, up to and including missing files, depending on how many blocks you generate.

Basic operation after generating your error recovery files is to grab the ~50k .PAR2 file with your data, run your favourite PAR2 tool over it to find how many blocks you need to repair, then you just download sufficient PAR2 files to cover it. It's effectively userspace RAID.
posted by Freaky at 12:57 PM on October 6, 2005