How to do a massive upload
November 16, 2007 6:20 AM Subscribe
How do I upload a large amount of data (multiple GB) to my Dreamhost account without tying up my DSL line day and night?
I was thinking maybe there's a high speed data center someplace where I can "rent" a DVD drive and then do FTP between the servers, but I don't know if anyone offers this. I don't have any local access to any higher speed Internet without driving to a city, and even there, I don't know anybody who can help.
I was thinking maybe there's a high speed data center someplace where I can "rent" a DVD drive and then do FTP between the servers, but I don't know if anyone offers this. I don't have any local access to any higher speed Internet without driving to a city, and even there, I don't know anybody who can help.
What's wrong with transferring it while you're sleeping?
posted by jeffxl at 7:00 AM on November 16, 2007
posted by jeffxl at 7:00 AM on November 16, 2007
Set up scheduled uploads for when you don't care about the line speed (at work, sleeping, etc). I think a lot of the non-free ftp clients (and maybe some free ones as well) can resume an upload from where it left off. It might take a few days or even a week, but it'll get done eventually.
posted by COD at 7:01 AM on November 16, 2007
posted by COD at 7:01 AM on November 16, 2007
Is there a specific reason that you can't or don't want to "tie up" your DSL line? It is packet-based you know - applications can share it.
I recently subscribed to an online backup service (Mozy) and when the entire initial upload was finished, I had uploaded 85GB. It took a while, but I was able to browse the web, etc without any problem while it was uploading in the background.
posted by GuyZero at 7:02 AM on November 16, 2007
I recently subscribed to an online backup service (Mozy) and when the entire initial upload was finished, I had uploaded 85GB. It took a while, but I was able to browse the web, etc without any problem while it was uploading in the background.
posted by GuyZero at 7:02 AM on November 16, 2007
I'm not sure of any places like this that rent. If you were to drive into the city is there a college or university? If you can gain a guest account you could take advantage of their likely fast connection.
The Uni. where I work provides network logins for community members. I think it may be coupled with a small library card fee.
posted by imjosh at 7:02 AM on November 16, 2007
The Uni. where I work provides network logins for community members. I think it may be coupled with a small library card fee.
posted by imjosh at 7:02 AM on November 16, 2007
What operating system do you run at home? WinSCP lets you throttle your upload speed, so you could start the upload and just adjust the slider downwards when you'd like more bandwidth for browsing, etc. Then kick it back up before you go to bed or while you're at work.
I assume that at least some of the similar Mac SFTP programs (Cyberduck, Fugu) have similar features, but I can't verify that while I'm at work.
posted by ckolderup at 7:05 AM on November 16, 2007
I assume that at least some of the similar Mac SFTP programs (Cyberduck, Fugu) have similar features, but I can't verify that while I'm at work.
posted by ckolderup at 7:05 AM on November 16, 2007
Response by poster: GuyZero said Is there a specific reason that you can't or don't want to "tie up" your DSL line?
Mostly it's because I am worried that I will hit some sort of limit on DSL traffic and our provider (Embarq) would pull the plug on us. We're in a DSL monopoly town, so that would suck.
So yeah, it's not really speed that's an issue but hammering our DSL line with this transfer.
posted by calhound at 7:09 AM on November 16, 2007
Mostly it's because I am worried that I will hit some sort of limit on DSL traffic and our provider (Embarq) would pull the plug on us. We're in a DSL monopoly town, so that would suck.
So yeah, it's not really speed that's an issue but hammering our DSL line with this transfer.
posted by calhound at 7:09 AM on November 16, 2007
Can you create a torrent file and upload that? Not sure how many people you envision needing to access this file, but this would spread the uploading burden among x number of people, instead of putting it all on you and then on whatever host you might find.
posted by stupidsexyFlanders at 7:21 AM on November 16, 2007
posted by stupidsexyFlanders at 7:21 AM on November 16, 2007
I wouldn't worry about you exceeding a limit on your DSL line. They set an upload/download maximum rate for each customer and you have no control over (and can't exceed), and you're well within your rights to max it out entirely.
Further, DSL is built so that it doesn't interfere with your neighbor's transfers (unlike cable), so a) you should feel no guilt about using a service you're paying for and b) they've got no reason to pull the plug on you.
posted by jeffxl at 7:22 AM on November 16, 2007
Further, DSL is built so that it doesn't interfere with your neighbor's transfers (unlike cable), so a) you should feel no guilt about using a service you're paying for and b) they've got no reason to pull the plug on you.
posted by jeffxl at 7:22 AM on November 16, 2007
re: my previous comment. Wow. Proof read, please.
Correction: I wouldn't worry about exceeding a limit on your DSL line. They set an upload/download maximum rate for each customer that you have no control over (and can't exceed), and you're well within your rights to max it out entirely.
posted by jeffxl at 7:24 AM on November 16, 2007
Correction: I wouldn't worry about exceeding a limit on your DSL line. They set an upload/download maximum rate for each customer that you have no control over (and can't exceed), and you're well within your rights to max it out entirely.
posted by jeffxl at 7:24 AM on November 16, 2007
Of course, plenty of service providers offer "unlimited" use until you hit a secret amount, then they threaten to drop you. I don't know if this happens with DSL, due to the typically lower bandwidth, but it certainly happens with cable internet access and cell phone data plans.
posted by knave at 7:34 AM on November 16, 2007
posted by knave at 7:34 AM on November 16, 2007
Dreamhosts supports rsync which allows bandwidth limiting via the --bwlimit= option ( via the command line). It should already be installed on macs/*nix's but is available for other platforms (deltacopy is a windows based wrapper). If you want to try it like this the following should work without to much of a noticable impact on your other usage.
rsync --bwlimit=50 --timeout=300 --no-blocking-io --stats -azSH /path/to/files/on/local/machine host.machine.com:/path/to/directory
That's measured in Kbytes/per second, and is doing on the fly compressing so you don't need to worry about compressing stuff first.
posted by tallus at 7:46 AM on November 16, 2007
rsync --bwlimit=50 --timeout=300 --no-blocking-io --stats -azSH /path/to/files/on/local/machine host.machine.com:/path/to/directory
That's measured in Kbytes/per second, and is doing on the fly compressing so you don't need to worry about compressing stuff first.
posted by tallus at 7:46 AM on November 16, 2007
Your small town DSL provider is not going to drop you for using the bandwidth which they have already so graciously allowed you access to via your mandatory contract and probably extortionist monthly fee.
Use it. Use it like you stole it.
A few other thoughts:
- Some one recommends using your local university for the transfer. This will probably require you to misappropriate a student's login information and necessitate criminal trespass. Further, most universities are well aware that students would love to upload huge files, so they blacklist any really abusive users. My school zaps the mac address of anything transferring more than 8 gigs in a 24 hour period.
- Don't waste your time with stupidsexyFlanders bittorrent suggestion. Bittorrent works because multiple hosts have the same large file. Since you don't have multiple hosts (it's just you and your files) torrent is about the stuoopidest way you can go.
- Follow the suggestions above regarding using a scalable, schedulable FTP client. (I use Filezilla.) I uploaded about 80gb to my Dreamhost account over the course of about two weeks on my wimpy DSL line...
posted by wfrgms at 7:48 AM on November 16, 2007
Use it. Use it like you stole it.
A few other thoughts:
- Some one recommends using your local university for the transfer. This will probably require you to misappropriate a student's login information and necessitate criminal trespass. Further, most universities are well aware that students would love to upload huge files, so they blacklist any really abusive users. My school zaps the mac address of anything transferring more than 8 gigs in a 24 hour period.
- Don't waste your time with stupidsexyFlanders bittorrent suggestion. Bittorrent works because multiple hosts have the same large file. Since you don't have multiple hosts (it's just you and your files) torrent is about the stuoopidest way you can go.
- Follow the suggestions above regarding using a scalable, schedulable FTP client. (I use Filezilla.) I uploaded about 80gb to my Dreamhost account over the course of about two weeks on my wimpy DSL line...
posted by wfrgms at 7:48 AM on November 16, 2007
I agree that you should use it to its fullest extent. If they bitch, at least we can have a nice internet outrage campaign.
posted by Mr. Gunn at 8:34 AM on November 16, 2007
posted by Mr. Gunn at 8:34 AM on November 16, 2007
GuyZero:
IIRC the mozy client also has throttling/scheduling available.
wfrgms brings up a good point with the "use it like you stole it". Got any open WAPs around your place? :)
posted by jbroome at 10:48 AM on November 16, 2007
IIRC the mozy client also has throttling/scheduling available.
wfrgms brings up a good point with the "use it like you stole it". Got any open WAPs around your place? :)
posted by jbroome at 10:48 AM on November 16, 2007
IIRC the mozy client also has throttling/scheduling available.
Since it was the initial upload, I cranked it to the max. The email still got through.
But per other commenters, just do it. If they want to they can throttle you without cutting off your service. This is what you bought an internet connection for after all.
posted by GuyZero at 10:55 AM on November 16, 2007
Since it was the initial upload, I cranked it to the max. The email still got through.
But per other commenters, just do it. If they want to they can throttle you without cutting off your service. This is what you bought an internet connection for after all.
posted by GuyZero at 10:55 AM on November 16, 2007
« Older We need some phone and internets for a small... | how meta is metafiction on AskMetafilter? Newer »
This thread is closed to new comments.
posted by cmiller at 6:55 AM on November 16, 2007