Host-to-host megatransfer
May 12, 2010 6:23 PM Subscribe
I have a client with 8GB of data I need to transfer from old web host (Powweb) to new one (Dreamhost). Old web host has no shell access and kills single FTP transfers at ~500MB. There are tons of files in tons of folders. Both hosts allow php and perl. How should I do this?
Need some clarity...thanks for your help.
Need some clarity...thanks for your help.
Response by poster: Interesting. Would I need shell access on both ends for that?
posted by circular at 6:48 PM on May 12, 2010
posted by circular at 6:48 PM on May 12, 2010
New webhost has shell? wget or curl also have mirror & reconnect ability. None of these should need shell access on the other side.
posted by fings at 7:02 PM on May 12, 2010
posted by fings at 7:02 PM on May 12, 2010
wget --mirror --convert-links will save you a lot of trouble here.
posted by mhoye at 7:10 PM on May 12, 2010
posted by mhoye at 7:10 PM on May 12, 2010
Response by poster: Will wget know how to recurse folders when they're not linked to each other? Not sure I understand how wget can discover all the folders without my FTP password.
Interesting thought about gftp, but isn't that a desktop client? I'd like to go host to host direct.
posted by circular at 7:23 PM on May 12, 2010
Interesting thought about gftp, but isn't that a desktop client? I'd like to go host to host direct.
posted by circular at 7:23 PM on May 12, 2010
Best answer: If you are using wget for ftp, it will need your password. The other option would be to use wget over http, if you are mirroring static files not dynamic content.
The insecure way (possibly reveals your password to others if you are on a shared system):
wget --mirror --user=youname --password=yourpass ftp://your.site/dir
The more secure method involves setting up a protected .wgetrc with the line "passwd = yourpassword" in it.
posted by fings at 7:36 PM on May 12, 2010
The insecure way (possibly reveals your password to others if you are on a shared system):
wget --mirror --user=youname --password=yourpass ftp://your.site/dir
The more secure method involves setting up a protected .wgetrc with the line "passwd = yourpassword" in it.
posted by fings at 7:36 PM on May 12, 2010
flashfxp is great for server to server transfers. Just queue everything up and go. Once the connection breaks, you can have the client autoresume.
posted by wongcorgi at 11:00 PM on May 12, 2010
posted by wongcorgi at 11:00 PM on May 12, 2010
Yeah, FXP is what I'd try first. FlashFXP is probably the best client for Windows, although I'm sure there are apps for other platforms.
posted by neckro23 at 12:14 AM on May 13, 2010
posted by neckro23 at 12:14 AM on May 13, 2010
Response by poster: Thanks everyone for the help. I tried flashfxp but it just shot a bunch of generic errors at me and refused to transfer the files. Feeling lazy, I switched over to the console, pasted the code above using wget and it's working fine so far. Excellent, and I've always loved the wget progress bar.
posted by circular at 6:20 PM on May 14, 2010
posted by circular at 6:20 PM on May 14, 2010
This thread is closed to new comments.
At the very least, you'd be able to mirror in 500M increments using it.
posted by swngnmonk at 6:37 PM on May 12, 2010