Host-to-host megatransfer

May 12, 2010 6:23 PM

I have a client with 8GB of data I need to transfer from old web host (Powweb) to new one (Dreamhost). Old web host has no shell access and kills single FTP transfers at ~500MB. There are tons of files in tons of folders. Both hosts allow php and perl. How should I do this?

Need some clarity...thanks for your help.

posted by circular to computers & internet (11 answers total)

Take a look at lftp - it does have mirror functionality over FTP, and may well handle reconnects as well.

At the very least, you'd be able to mirror in 500M increments using it.
posted by swngnmonk at 6:37 PM on May 12, 2010

Interesting. Would I need shell access on both ends for that?
posted by circular at 6:48 PM on May 12, 2010

I think you could use wget to recursively copy the files.
posted by zippy at 6:59 PM on May 12, 2010

New webhost has shell? wget or curl also have mirror & reconnect ability. None of these should need shell access on the other side.
posted by fings at 7:02 PM on May 12, 2010

wget --mirror --convert-links will save you a lot of trouble here.
posted by mhoye at 7:10 PM on May 12, 2010

Does gftp not reconnect automatically?
posted by yoyo_nyc at 7:15 PM on May 12, 2010

Will wget know how to recurse folders when they're not linked to each other? Not sure I understand how wget can discover all the folders without my FTP password.

Interesting thought about gftp, but isn't that a desktop client? I'd like to go host to host direct.
posted by circular at 7:23 PM on May 12, 2010

If you are using wget for ftp, it will need your password. The other option would be to use wget over http, if you are mirroring static files not dynamic content.

The insecure way (possibly reveals your password to others if you are on a shared system):

wget --mirror --user=youname --password=yourpass ftp://your.site/dir

The more secure method involves setting up a protected .wgetrc with the line "passwd = yourpassword" in it.
posted by fings at 7:36 PM on May 12, 2010

flashfxp is great for server to server transfers. Just queue everything up and go. Once the connection breaks, you can have the client autoresume.
posted by wongcorgi at 11:00 PM on May 12, 2010

Yeah, FXP is what I'd try first. FlashFXP is probably the best client for Windows, although I'm sure there are apps for other platforms.
posted by neckro23 at 12:14 AM on May 13, 2010

Thanks everyone for the help. I tried flashfxp but it just shot a bunch of generic errors at me and refused to transfer the files. Feeling lazy, I switched over to the console, pasted the code above using wget and it's working fine so far. Excellent, and I've always loved the wget progress bar.
posted by circular at 6:20 PM on May 14, 2010