New York to China Data Transfer Problem
November 15, 2007 7:01 AM Subscribe
I have to transfer digital files that are 4 to 8 gigabytes from New York City to a location in Hong Kong and a location in Xiamen, China.
As I will have to do this repeatedly over the next several days, and time is of the essence, I can't just UPS or FedEx DVDs, I need an internet way to do this. In New York, where I upload, the speeds are pretty slow to upload (700K), and on our existing computers in Hong Kong and Xiamen the download speeds are way slower than the upload speeds here in New York.
What's the best way to do this? I am open to any solution that I can arrange quickly whether it is buying software that is effective at pushing and pulling data that can be used on both ends, or even taking DVDs to some place in New York where they can get quickly uploaded and locating a place in Xiamen and HK that can quickly download.
As I will have to do this repeatedly over the next several days, and time is of the essence, I can't just UPS or FedEx DVDs, I need an internet way to do this. In New York, where I upload, the speeds are pretty slow to upload (700K), and on our existing computers in Hong Kong and Xiamen the download speeds are way slower than the upload speeds here in New York.
What's the best way to do this? I am open to any solution that I can arrange quickly whether it is buying software that is effective at pushing and pulling data that can be used on both ends, or even taking DVDs to some place in New York where they can get quickly uploaded and locating a place in Xiamen and HK that can quickly download.
Even 700kbp/s is still a pretty good upload if you're using a non-commercial connection, and no matter how fast you can upload, their downloads can't magically go faster. Tracking down an internet cafe or somesuch is your best bet.
posted by Tomorrowful at 7:18 AM on November 15, 2007
posted by Tomorrowful at 7:18 AM on November 15, 2007
Wacky solution: bittorrent:
posted by smackfu at 7:35 AM on November 15, 2007 [1 favorite]
- Checks hashes, so you know the data integrity is there.
- Is commonly used for 4-8 GB files (ISOs)
- Gracefully handles interrupted transfers.
- Can do the uploading from multiple local computers, if you can get them on separate links.
posted by smackfu at 7:35 AM on November 15, 2007 [1 favorite]
You could try using 7-zip to compress the files, in LZMA, but decompression is slow, and if these are media files (like movies), then they won't compress much.
In NYC, most office buildings will have very fast connections. If some corporation is connected to this, talk to their IT dept. Also, try calling Verizon or AT&T and explaining the situation. They may have a service where you can lease a high bandwidth pipe for a month.
On the China end, they will need someone with a fast pipe. Perhaps the phone company there could help?
posted by Pastabagel at 7:44 AM on November 15, 2007
In NYC, most office buildings will have very fast connections. If some corporation is connected to this, talk to their IT dept. Also, try calling Verizon or AT&T and explaining the situation. They may have a service where you can lease a high bandwidth pipe for a month.
On the China end, they will need someone with a fast pipe. Perhaps the phone company there could help?
posted by Pastabagel at 7:44 AM on November 15, 2007
Response by poster: Thanks so far, regarding bizwank's question, yes, I did not mean 7Mbp/s I meant 700kbp/s.
Regarding bittorrent, I could only upload from multiple computers on the same wifi connection / pipeline could that still help?
In terms of Internet cafe's, will look into this, but if anyone can offer anything more concrete (i.e. is there a commercial service that is made for addressing this problem?) I will be grateful!
posted by extrabox at 7:46 AM on November 15, 2007
Regarding bittorrent, I could only upload from multiple computers on the same wifi connection / pipeline could that still help?
In terms of Internet cafe's, will look into this, but if anyone can offer anything more concrete (i.e. is there a commercial service that is made for addressing this problem?) I will be grateful!
posted by extrabox at 7:46 AM on November 15, 2007
Response by poster: Files are tens of very high res photos that need to stay hi-res for printing reasons.
posted by extrabox at 7:48 AM on November 15, 2007
posted by extrabox at 7:48 AM on November 15, 2007
There are pay sites like MegaUpload and RapidShare which provide a direct service you seek- you can upload a file to their central server which can be downloaded elsewhere. For files of your service, I think you'll need a "premium" account.
However, your downloader will also need a premium account for bigger files, though I'm guessing they could use your credentials if you purchase an account. I only have experience with their free service, which is filled with adds, pop-ups, and you need a download manager for resuming because they will crap out.
posted by jmd82 at 7:58 AM on November 15, 2007
However, your downloader will also need a premium account for bigger files, though I'm guessing they could use your credentials if you purchase an account. I only have experience with their free service, which is filled with adds, pop-ups, and you need a download manager for resuming because they will crap out.
posted by jmd82 at 7:58 AM on November 15, 2007
^ Uploading the file somewhere than retrieving it is an unnecessary step and would take quite a bit longer than the other solutions offered. The issue here is bandwidth, not storage.
There would be no benefit to running multiple bittorrent clients on the same pipe, but if you could spread copies around onto different internet connections you would then benefit from their combined bandwidth. This would only help with the uploading though, you would still need a single fast connection at the receiving end to assemble the pieces.
Looks like Kinkos offers computer rentals and "technology services", and has branches in NY and China. That might be your easiest/fastest solution as all the tech is already in place.
posted by bizwank at 8:07 AM on November 15, 2007
There would be no benefit to running multiple bittorrent clients on the same pipe, but if you could spread copies around onto different internet connections you would then benefit from their combined bandwidth. This would only help with the uploading though, you would still need a single fast connection at the receiving end to assemble the pieces.
Looks like Kinkos offers computer rentals and "technology services", and has branches in NY and China. That might be your easiest/fastest solution as all the tech is already in place.
posted by bizwank at 8:07 AM on November 15, 2007
Bittorrent is a cool technology, but it's not going to make your upload or download faster if everyone on the upload/download side is using the same pipe (shared connection). You want more pipes or fatter pipes, and there's no getting around it.
You've got a 4 GB payload. The Chinese side is the limiting factor on throughput. You don't say how fast they run, but assuming it's 384 Kbps, it'll take 23 hours to download 4 GB under optimal conditions and allowing for no network overhead. If they can install two lines in their office, you halve that. And so on. With 3 or 4 lines, it starts to look like a feasible amount of time. You'd obviously need to partition the payload appropriately so that each download station got an equal chunk. And you'd probably want to add at least one line on the NY side and set up a second upload station.
If the Chinese office sends 10 people out to 10 different Internet cafes (and assuming each person is getting the same 384 Kbps DL speed, which seems unlikely), each person will spend 2 hours 20 minutes staring at a progress bar, which seems like a poor use of personnel.
posted by adamrice at 8:13 AM on November 15, 2007
You've got a 4 GB payload. The Chinese side is the limiting factor on throughput. You don't say how fast they run, but assuming it's 384 Kbps, it'll take 23 hours to download 4 GB under optimal conditions and allowing for no network overhead. If they can install two lines in their office, you halve that. And so on. With 3 or 4 lines, it starts to look like a feasible amount of time. You'd obviously need to partition the payload appropriately so that each download station got an equal chunk. And you'd probably want to add at least one line on the NY side and set up a second upload station.
If the Chinese office sends 10 people out to 10 different Internet cafes (and assuming each person is getting the same 384 Kbps DL speed, which seems unlikely), each person will spend 2 hours 20 minutes staring at a progress bar, which seems like a poor use of personnel.
posted by adamrice at 8:13 AM on November 15, 2007
Best answer: You haven't yet said whether these high-res photos are compressed or not, but based on your assertion they are for high-res print purposes, I'd bet they are uncompressed.
How big is each file? What file format are they in? Are they PSD with layers? Can they be flattened?
If they are indeed uncompressed, you can do lossless compression on them. Maybe use tar + bzip2. You could see from 30% up to around 80% compression, and it will be lossless, unlike JPEG compression.
posted by tomierna at 8:52 AM on November 15, 2007
How big is each file? What file format are they in? Are they PSD with layers? Can they be flattened?
If they are indeed uncompressed, you can do lossless compression on them. Maybe use tar + bzip2. You could see from 30% up to around 80% compression, and it will be lossless, unlike JPEG compression.
posted by tomierna at 8:52 AM on November 15, 2007
What's your budget?? Are you going to have to do this again over the next year? Based on service availability of China, I would spend some money and get fat wireless pipes installed at all three locations.
posted by kaizen at 8:55 AM on November 15, 2007
posted by kaizen at 8:55 AM on November 15, 2007
Best answer: Another thought comes to mind. I have no idea how you're using these photos or what your workflow is, but does your situation make it feasible to send low-res FPO images electronically, and overnight a DVD or two to arrive later?
posted by adamrice at 9:16 AM on November 15, 2007
posted by adamrice at 9:16 AM on November 15, 2007
I'd second using either 7zip or bz2 compression to try and reduce the amount that needs to be transfered.
that said, if you still can't get a decent connection speed in China, you might want to reconsider the shipping of a DVD.
If the Chinese side can't download faster than 128 Kbps, DHL says they can move a 4GB of data faster than the internet can.
sources:
download speed calculator
DHL, get a quote
posted by jrishel at 9:26 AM on November 15, 2007
that said, if you still can't get a decent connection speed in China, you might want to reconsider the shipping of a DVD.
If the Chinese side can't download faster than 128 Kbps, DHL says they can move a 4GB of data faster than the internet can.
sources:
download speed calculator
DHL, get a quote
posted by jrishel at 9:26 AM on November 15, 2007
Best answer:
posted by jenkinsEar at 10:02 AM on November 15, 2007 [1 favorite]
Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.—Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall, 83. ISBN 0-13-349945-6.
posted by jenkinsEar at 10:02 AM on November 15, 2007 [1 favorite]
You are going to be limited by the lowest bandwidth of the two sites.
That said, sometimes it is faster to upload to an intermediate site, if both sites have good bandwidth to the intermediate host but not to each other. Unlikely, and hard to find, but possible.
I'd suggest using something like rsync for the transfer. It won't make the transfer any quicker (and can make it a touch slower) but it's nice to be able to resume a broken download. In cases where the data files might get updated after initial transfer, it can also save a ton of time/bandwidth.
You might want to try syncing more than one at a time. Depending on the network setup, sometimes the aggregate bandwidth of two connections can be better than a single connection.
posted by alikins at 11:23 AM on November 15, 2007
That said, sometimes it is faster to upload to an intermediate site, if both sites have good bandwidth to the intermediate host but not to each other. Unlikely, and hard to find, but possible.
I'd suggest using something like rsync for the transfer. It won't make the transfer any quicker (and can make it a touch slower) but it's nice to be able to resume a broken download. In cases where the data files might get updated after initial transfer, it can also save a ton of time/bandwidth.
You might want to try syncing more than one at a time. Depending on the network setup, sometimes the aggregate bandwidth of two connections can be better than a single connection.
posted by alikins at 11:23 AM on November 15, 2007
It's possible I'm missing something here, but what about just compressing them and sending via senduit?
posted by yoga at 1:06 PM on November 15, 2007
posted by yoga at 1:06 PM on November 15, 2007
FTP FTW!!!
Err.. Rather, I suggest using FTP.
Get an FTP server setup. No, you don't need a separate or special computer, just a computer that's got a full-time net connection. Google for and install some FTP (or tFTP, if security is a consideration) server software. Then, simply provide your IP address and a username/password to your client in china, and they can download the files wherever they can get the best download speeds.
(Back in the day, WarFTP was free and easy to use for the server side of things - probably still is. Google it)
posted by terpia at 2:45 PM on November 15, 2007
Err.. Rather, I suggest using FTP.
Get an FTP server setup. No, you don't need a separate or special computer, just a computer that's got a full-time net connection. Google for and install some FTP (or tFTP, if security is a consideration) server software. Then, simply provide your IP address and a username/password to your client in china, and they can download the files wherever they can get the best download speeds.
(Back in the day, WarFTP was free and easy to use for the server side of things - probably still is. Google it)
posted by terpia at 2:45 PM on November 15, 2007
Response by poster: Thanks everybody for suggestions so far. We are going to try a mixture of compressing some of our uploads (Thanks for the suggestions of specific compression methods) and DVD's via UPS and/or Continental Airlines which will take our DVDs Newark airport to HK airport for about twice what UPS charges.
DHL was slower than FedEx and UPS.
We tried the FedEx / Kinkos business office to business office solution but came into contact with people who knew far less than we do about this stuff which is not saying a whole lot.
We're also going to look into local internet cafes to see if someone has a lot of bandwidth.
(Right now we are uploading compressed files to our website's server, and then having our people in China download from there)
posted by extrabox at 2:56 PM on November 15, 2007
DHL was slower than FedEx and UPS.
We tried the FedEx / Kinkos business office to business office solution but came into contact with people who knew far less than we do about this stuff which is not saying a whole lot.
We're also going to look into local internet cafes to see if someone has a lot of bandwidth.
(Right now we are uploading compressed files to our website's server, and then having our people in China download from there)
posted by extrabox at 2:56 PM on November 15, 2007
This thread is closed to new comments.
posted by bizwank at 7:12 AM on November 15, 2007