Syncing files on a remote web server with Windows 7

August 14, 2013 10:51 AM

I have an interesting file syncing problem. As part of my responsibilities, I have to keep track of changes in the US export regulations. The Dept. of Commerce makes the regulations available in two formats, HTML and PDF. The HTML versions are oddly formatted so I prefer the PDF versions even if they are harder to search. They're available in PDF on this page. I have been regularly going out to the page and downloading relevant documents that have changed but I'd like to automate this if I can. But there are some gotchas.

First of all, the page doesn't actually host the PDF files, it just links to them on another page that's not so easily accessed. Second, I'm on Windows 7 so getting something like rsync to work will take some extra effort. Is this something that's possible? Is there a better choice than rsync? It's not the end of the world having to do this by hand but it just seems like something that should be automated.

posted by tommasz to Computers & Internet (3 answers total)

I think rsync is only going to be usable for this if the URLs for each of the files stays the same as they are updated. I'm not sure if this is actually true or not. Writing a script to do it is no major task, though, if you're good at that sort of thing.
posted by tylerkaraszewski at 11:05 AM on August 14, 2013

Perl script that looks at the linked page, pulls the URLs and date modified from the table, and saves all the PDFs to disk. running it again will only download files in the case that they were added to the table, or the "Date Last Modified" has changed.

Link to code on pastebin.

Edit: changed code to pastebin link, as metafilter formatting screws it up.
posted by tylerkaraszewski at 12:04 PM on August 14, 2013 [1 favorite]

Use PowerShell and Windows Task Scheduler, it will run a scheduled task so you don't have to do it by hand. Something like this should work.
posted by rada at 2:39 PM on August 14, 2013