How can I FTP only the recently modified files from within a deep file structure?

September 15, 2009 8:54 AM

How can I FTP only the recently modified files from within a deep file structure?

I work for an organization with a fairly large website (~20,000 discrete items - .asp, .txt, and .pdf). Recently it was decreed that we would change the way some of our links work (inserting intermediary warning page in new targeted window). I was able to write a find+replace script to switch out the old link for the new one, without messing up anything else. It works a treat, but after running it on my local development version of the site I don't particularly want to FTP the whole thing over to the live site, all 20,000 files, when only maybe 3,000 have been changed. Particularly when some of the .pdf's (none of which are affected by this) are giant - 1,000's of MB. Doing it manually would be nuts as these files are sprinkled throughout a fairly complex and deep file structure.

Is there a way to FTP only the recently changed files, while maintaining the file structure? I normally use Flash FXP, but would be fine with some fancy command line FTP expressions, or any other cheap/free solution.

posted by dirtdirt to computers & internet (15 comments total)

This sounds like a job for rsync!
posted by Oktober at 8:55 AM on September 15

Yep, 2nding rsync.
posted by jquinby at 8:56 AM on September 15

Seconding rsync, but if that's not an option, FileZilla and many/most other FTP clients will allow you to only overwrite files whose modified date varies, or whose size varies...

You'll still have to let the FTP client scan through every directory and queue up the files, only to decide not to upload them, but it'll get the job done.
posted by twiggy at 8:57 AM on September 15

Me three on rsync.
posted by cellphone at 8:57 AM on September 15

You could use an xcopy with the /d flag to copy over to some other tree just those files which are recent, then upload that new tree, if rsync isn't your thing.
posted by adipocere at 9:21 AM on September 15

This also sounds like something that could be handled with Mod Rewrite at the higher directory levels, where it will be inherited into the subdirectories. I've mainly used it on Apache, but if you're IIS there are ways to get equivalent functionality.
posted by odinsdream at 9:29 AM on September 15

If you had the files on a *nix system, and you wanted to find things that had changed in the last 10 days, you could do: find . -type f -mtime -10

The "-10" can be changed to "-5" for more recently than 5 days, etc. Then you could use the -exec switch in find to run your ftp transfer.
posted by beerbajay at 9:48 AM on September 15

Transmit (Mac FTP app) can synchronize only changed files in subdirectories (upload or download).
posted by kirkaracha at 10:02 AM on September 15

<CowboyCoder>Just run the script on production!</CowboyCoder&gt
posted by nomad at 10:07 AM on September 15

<CowboyCoder>Just run the script on production!</CowboyCoder>

Oh, man, I know. The best part of that is then I could just overwrite the dev site with the live, and not have to worry about accidentally publishing stuff that should only be on dev to the live site. What could POSSIBLY go wrong?

Forgot to mention that I'm a Windows guy - rsync looks cool but scary(in that way that EVERYTHING *nix looks scary), and the windows wrapper version (DeltaCopy) looks a bit iffy, and I'm not doing any cygwin stuff.

Right now I'm leaning towards biting the bullet and doing it manually, just avoiding directories that I know are full of stuff that won't be changed.
posted by dirtdirt at 10:27 AM on September 15

I've used DeltaCopy without issue. If you run into problems feel free to MeMail me.
posted by odinsdream at 10:54 AM on September 15

seconding Filezilla. Just configure the account and drag the modified (local) directory tree on the remote tree. It'll take a while to scan your disk and then as it encounters the first duplicate file, it'll ask you what to do. You want to overwrite all files that are newer or different (and check the box that says "repeat for all files") and that's it. You can also leave it running overnight and set it up so that it shuts down your computer when it's done.

notice: I assume that everything that's on the production server has been already backed up.
posted by _dario at 1:42 PM on September 15

+1 on Rsync
posted by edmo at 2:07 AM on September 16

WinSCP has an 'only transfer changed files' flag. (And it's free.)
posted by bhance at 1:02 PM on September 16

Are the files' timestamps a reliable indicator of whether they should be updated or not? If so, you can use Total Commander to search for files newer than a certain timestamp and then copy only those. It has a built-in FTP client.
posted by Nameless at 5:21 PM on September 16