Make me an IT hero!

April 6, 2009 11:52 AM

I am wanting to automate a SFTP daily download.

We’re looking for an automated solution that will:

Allow us to daily download a directory of PDFs using SFTP from read only directories. This can be a GUI or a CL solution running on a Mac or a PC. The Mac would have 10.4 or 110.5 most likely, the PC some flavor of XP server. I am more conversant with Macs and Unix.

It will need to draw files from 6 different sub directories. We would prefer to grab only new files (ones not already downloaded).

We are behind a firewall, but I can grab the files with a GUI.

posted by cjorgensen to computers & internet (11 comments total)

Are the files named in a predictable fashion?
posted by odinsdream at 11:59 AM on April 6

The files have a complex naming convention like:

P2352000-0-X01200-1--------XA.pdf
P2352000-0-X01300-1--------XA.pdf
etc.

I can probably find out how they go about deciding the file name. They change daily.

I failed to mention these will all be PDF files under 20 mb.
posted by cjorgensen at 12:02 PM on April 6

1. Do an ls on the directory.
2. Compare it with your list of already downloaded files.
3. Download the files you don't already have.
posted by devnull at 12:06 PM on April 6

why not just use rsync? wheel has already been invented.
posted by Mach5 at 12:17 PM on April 6

On Windows I would use Syncback and use that to sync your online and offline directories but for SFTP you need the paid version.

Otherwise, look at rsync on Unix/Linux systems.
posted by IAr at 12:24 PM on April 6

Use rsync over SSH (with key files, so a cron job can run it) and you will be trying on capes by next week.
posted by wenestvedt at 1:00 PM on April 6

Looking into rsynch now. I do not have access to the remote box, and cannot write any files to it. Just an FYI, since the docs mention writing a public key to the server.
posted by cjorgensen at 1:03 PM on April 6

rsync over whatever (ssh probably) would be my first choice. It's tough to beat. But if you really are stuck with SFTP you can look into using "expect" to automate the process. If that isn't tough enough, you can use the "pexpect" module under python which should give you god-like powers.
posted by chairface at 5:27 PM on April 6

If you don't have the ability to set up a shared key, take a look at lftp which will let you specify a username/password. I realize that SSH doesn't allow scripted passwords for a reason, but sometimes you just need to make it work (and I'm not a big fan of expect.)
posted by sysinfo at 6:27 PM on April 6

there are two things called sftp and I'd just like some clarification. are you FTP wrapped in an SSL layer? or are you using scp(ssh secure copy) wrapped in an FTP command layer?

if you don't know what I'm talking about, an approximate answer could be found in what port(s) you have open in your firewall to complete this transation.

if you're using sftp that is actually over ssh, rsync would completely work for you out of the box.

if you're using the first one, FTP wrapped in SSL, I suspect the direction to use an expect script will be your best course of action.
posted by jrishel at 6:40 AM on April 7

Port 22 is the port we're connecting over. We had multiple people working on this, and the solution that was selected was ja-sftp. Seems to be pretty slick, but not what I was thinking. I did learn quite a bit about Unix man pages, and now has a script in place that downloads my weblogs for my personal sites every day, so not a total loss.

Thanks everyone.
posted by cjorgensen at 11:18 AM on April 12