Send Receive Large Data Everyday

November 6, 2007 6:52 AM

I need to move serveral gigs of data everyday from/to around the world.

First, I have no idea how server or web sites work.

Second, My computer skills are install new o.s./software/etc... use zonealarm, anti-virus program... and carry around some encrypted folders around... I can create fairly secure home wireless network.

So.. I am an average joe with some computer skills. If possible I would like to get some step by step recommendations.

I have one location in U.S. where I get average home internet connection with PC with big Hard Drive.

I need to travel around the world and be able to send few giga bytes of data almost everyday to the US computer. Some times I have to get giga bytes of data from the same PC in U.S.

The data need to be transfered securely..

My usual internet connection speed seems to be similar to the ones in my US home. I've used GoToMyPC and LogMeIn to send or receive 600mb of data taking up to four hours each way... It seems to be same time frame where in the world I am in.... This is not acceptable to do it everyday....

What can I do?

posted by curiousleo to computers & internet (29 answers total)

This is a pretty tall order. If your home Internet connection can transfer 600 mb of data in 3-4 hours, you're seeing a speed of 40 - 60 kilobytes/sec. If that isn't fast enough, your first order of business is to upgrade your home connection to one fast enough to transfer the data in the time you have available. You may want to consider colocation, where you rent a server and a fast Internet connection in a data center and use that remotely.

Don't forget that you will also be limited by the speed of the Internet connection in the places you travel to. No matter how fast your home connection is, if you only have a 20 kilobyte/sec connection where you're staying, that's as fast as you can go.

My suggestion is that you think of alternatives, in case you can't guarantee that the connections where you're travelling will be fast or reliable enough. Why are you transferring the data home? If it's for processing, maybe you can do that processing on a laptop you carry with you. If it's for backup, consider burning a DVD at each location and sending it home by courier, plus putting the data onto a portable backup drive. If you can give more details on what you want to do, maybe AskMe can give more specific advice.
posted by pocams at 7:07 AM on November 6, 2007

Do you need the data that day, or is this more of a backup solution? Because if it is just for backup, it might be easier to mail physical media.
posted by DMan at 7:08 AM on November 6, 2007

You're generating gigabytes of new data every day on a home pc? Something seems wrong; it's quite unlikely that you would generate that much unique new data every day.

If it's mostly the same data, and only some of which changes day to day, you can use rsync.

Otherwise: that's a hard problem, and an expensive one to solve. You can bring in a dedicated T3 or DS3 to your house, and then go to the fastest pipes you can find in the countries you visit.

But, assuming a budget that's in normal human range (since you appear to have a regular single PC in a regular house), the answer is: you don't do that. There is no way to cheaply transfer gigabytes of new data every day to anywhere in the world.
posted by Malor at 7:10 AM on November 6, 2007

Are you sure you need to send all the data or just changes? We use a complicated rsync nightmare to do online backups, but rsync is smart enough to just sync our changes instead of the whole thing. That cuts the size of the transfers by 85%.

If you do need to move all the data, strong crypto plus a cd plus fedex is the way we give our customers their backups. Might that work?
posted by mrbugsentry at 7:12 AM on November 6, 2007

Your bottleneck is the upload speed of the connection that you're sending from. There's no way to send something faster than that connection permits.

Most home Internet providers allow pretty steamy download rates (5Mb/s is not uncommon), but cap the upload speeds to very low rates, like 50kbps. It sounds like this is the case in your scenario.

Even if Computer A (in the US) can download 5Mb/s+, it will only download as fast as Computer B (in the other location) can upload it. So that's your bottleneck.

Transfer speeds aside, it sounds like you need to setup a simple FTP server on your home PC. There are countless articles on how to do this; here's the first good-looking one that Google revealed.
posted by sprocket87 at 7:13 AM on November 6, 2007

Actually, I don't think it is my home Internet connection.
I have tried to send files through gotomypc and logmein to and from different locations to different internet connections in different pcs.. most of the times it was around 3-4hours for 600mb... it could be the software thing or some people tell me that it is the upload speed that are bad.. (which means most home type internet connctions (dsl/cable) limits the upload speed..)

So.. how does other pros(like photographers, IT people, spies,etc.)???

burning and sending via mail is not a possibility for me. Beside, I would like to set it up so that i can constantly get and put large data to one place where in the world I am...
posted by curiousleo at 7:15 AM on November 6, 2007

Well, You could easily handle this using a T1, and you would saves thousands a month over the cost of a T3.

Anyway, I do this already. I have a colocated server and a backup server at my house. The server at my house uses ssh and tar to do daily incremental and weekly full backups.


I would suggest that you use some form of FTP. pure-ftpd is a nice ftp server (assuming you are running some flavour of unix). Another thing, unless this positively has to be secured I would stay from using encryption. It is going to cost you a lot in terms of bandwidth. Otherwise, you can use WinSCP (on your laptop) and pureftpd (on your server back home) over SSL.

I'm with the others in wondering if your travel stops will have the necessary bandwidth.
posted by ydnagaj at 7:19 AM on November 6, 2007

If I use FTP thing.. would it enable me to upload faster?
Is FTP the fastest way to transfer data?

I uploaded a simple website via FTP... I had no idea what I was doing... the software simple seems to upload it the site...(it was simple single page one)...

Is FTP safe?

THE DATA are HD video type of data... most range from 350mb to 600mb as WMV/AVI/MP4 etc... trying to shrink them is not possible. (too much time for too many files...)
posted by curiousleo at 7:21 AM on November 6, 2007

So.. how does other pros(like photographers, IT people, spies,etc.)???

I'm an IT person and a photographer (though not a spy), and I certainly don't have the need to transfer 600mb+ every single day. Sure, I do a decent amount of uploading to my FTP server, but not that much on a daily basis. Furthermore, my work connection offers 5Mb upstream, so I have the luxury of uploading as fast as the FTP server can download.

The people you're imagining send that much would have to have blazing fast connections at both ends, but especially on the sending side. That's simply the only way to do this - you can't send data through faster than the pipe allows!

Without finding an ISP that affords you greater upload speeds, you're stuck with whatever they do give you. Sorry.
posted by sprocket87 at 7:21 AM on November 6, 2007

By the way it is not all BACKUP use... I will say 70% or more is for transfering video files... obviously I will use the setup for the backup too....

By looking at some of the answers.. T1 or T3 setup at home may work.. BUT... it would only help me downloading from far away... 80% or more times... I will need to upload the files.....

any ideas?
posted by curiousleo at 7:24 AM on November 6, 2007

FTP is simply a protocol for sending data. It's not going to be (noticeably) more or less efficient than any other method for your purposes.

If you use SSL with FTP, it is secure.

Again, re: your transfer speeds - it is completely dependent on what the ISP (Internet Service Provider) at the sending location gives you. There's no clever hack to circumvent the physical bandwidth limits.
posted by sprocket87 at 7:26 AM on November 6, 2007

Another thing... from my experience.. most places out side of U.S... seems to have much faster download speed than anywhere in U.S.... I assume the upload is little faster.. but as i said before... my gotopc and logmein tells me otherwise..
posted by curiousleo at 7:26 AM on November 6, 2007

You don't specify what type of data it is, but if it's not video or photos, you *need* to use some compression on the files to reduce their file size before you send them.

This means make use of a program that can make a "zip file archive" of the data you want to send.

Then, at the other end, when you want to view the file, you 'unzip' the archive file to see the original contents.

For security, the 'zipping' process optionally supports password-protecting the archive file.
posted by Wild_Eep at 7:27 AM on November 6, 2007

Perhaps you can find a local Internet Cafe that you can use for an hourly rate. If there are some nearby, just ask what their upload/download speeds are (not to mention their policies on such transfers). If you can find one with a blazing fast connection, it might be worth the couple/few bucks to use it.

Then again, if you're doing this every day, I'm sure you could find a much faster ISP for the price of daily Internet Cafe usage.
posted by sprocket87 at 7:30 AM on November 6, 2007

I don't know about gotomypc, but transferring stuff via logmein is terribly slow on my computer and I have a decent connection. Do you have to use one of those services?
posted by yerfatma at 7:40 AM on November 6, 2007

The roaming part is the hard part. You can probably install enough bandwidth at home for your transfers, but you will probably have a hard time uploading while on the road. I don't believe you're going to be able to do this unless you pay serious coin to a series of service providers along your travel route: the upstream bandwidth you'll find at the typical cafe probably won't be adequate to the task at hand, not to mention whatever policies the cafe might have in place to keep people from clobbering their connections.

Can you do this via the post office? Basically, burn DVDs as you go, drop them in the mail, etc. Remember, nothing quite beats the bandwidth of a minivan loaded up with backup tapes/disks/etc. This would be analogous: if you don't need close to realtime access, the disk in the mail may be the best way to do it.
posted by chengjih at 7:47 AM on November 6, 2007

Not going to happen. I have ADSL 1.5M down, 386K up, my ADSL terminates at my workplace, no Internet to worry about in between. It would take me about 9.5 hours to download 5GB of data and 4 times as long to upload it.

I also happen to work for a University IT department that is connected to many high speed research networks that span the globe, 10Gb/s pipes across the US and to some countries, 1Gb/s pipes to other places around the globe.

Even if you were on these research networks, you would have a hard time doing what you want in a timely manner without tuning your computers TCP stacks and using strange high performance network tuned programs.

You're not going to be able to visit coffee shops or free wifi places, or hotel internet connections in random places around the world and do what you're asking.

Invest in a DVD burner and mail your data back, co-locate a server in a data center, have a friend upload data, whatever. You're going to have to be creative in solving your needs.
posted by zengargoyle at 7:48 AM on November 6, 2007

If you're traveling around the world and want the data to be where you are, it's really tempting to say sneaker net. Looking at newegg, you can get a 1000GB external hard drive that just goes with you for about $350, 500GB/$120, 250GB/$80.
posted by a robot made out of meat at 8:04 AM on November 6, 2007

I'm guessing once you look at the cost of setting up an FTP server that will work for you and getting a T1 to your house, you'll realize that fedex overnighting DVDs will be much less expensive and only slightly slower. Regardless of how much you can get your "home" setup for, you'll eventually find yourself uploading files from somewhere at such a slow rate that you should have just fedexed them anyway.
posted by pwb503 at 8:07 AM on November 6, 2007

>sneakernet

Sending hard-drives via UPS/FexEd is still the fastest, most reliable method of multi-GB information transfer.

(This speaking as someone who just had to spend two weeks support a national telecom/ISP provider trying to move 50GB of data every-day over a distance of only 1400 kilometers - in the end, guess which method was used?)

Sorry ;-)
posted by jkaczor at 9:01 AM on November 6, 2007

Read up on S3 and Jungledisk.
posted by ill3 at 9:32 AM on November 6, 2007

curiousleo,

Is the problem with GoToMyPC that you have this big window open, sending files? Or do you actually need it to take less than four hours?
posted by effugas at 9:33 AM on November 6, 2007

Nth-ing fedexing DVDs for backup purposes.

But, just to clarify: You're *also* having to transfer multi-gigs of video files in a daily basis, just to do your work? That's definitely getting into "not really viable for telecommuting" territory, no matter how you slice it.
posted by Tomorrowful at 9:35 AM on November 6, 2007

You need to assess (or at least describe) why you need to transfer this much data so often. There is likely a better way, but without more information it's difficult to give advice.
posted by rhizome at 9:43 AM on November 6, 2007

FedEx
posted by jannw at 12:38 PM on November 6, 2007

Let's assume for the sake of argument that you find yourself somewhere with the basic consumer-grade DSL at 192Kbps (little-b = bits, big-B = bytes) upstream. IME DSL circuits usually operate at about 80% efficiency, so working through a bit of math we can say that the best-case upload rate from that connection will move 1.125 MB in one minute. At that rate a 1GB file will take 1152 minutes (just over 19 hours) provided nothing goes wrong. A 350MB file would take six and a half hours, Your report of 3-4 hours for 600MB looks like about 768Kbps upstream, which isn't a number you can count on getting everywhere.

You also have to consider that in many locations, unless you have a known office you're working out of, you might have to deal with usage caps and/or charges. If you're working out of other people's houses, the limitations of their network connection will be unknown and unpredictable. If you're planning on using internet cafes, you'll also run into the issue of time, since you may not be able to stay connected for long enough to complete your file transfers (either because they have a time limit, or because they have to close for the day). If you get lucky and find a location to transfer your files with a generous upload speed (some cable modem or fiber connections offer 1Mbps or more) you could transfer that same theoretical 1GB file in under four hours (while sitting and babysitting it, of course). Assume in most cases you're not going to get that lucky, though.

You have two distinct problems, really: (1) you need access to an entire hard drive full of content that's at home when you're not, and (2) you need to be able to send large amounts of data back home. The first problem is pretty easily solved if the data at home isn't changing (except for what you're adding to it remotely): buy a portable external hard drive (or drives) and copy all the data onto it (or them) before you go. The size of the necessary drives depends on exactly how much data you need to access. The second problem is best solved by FedEx, as pointed out by everybody above. In the worst case scenario (more than a gigabyte to move and a small pipe with which to upload) FedEx will actually be faster.

You can also get something like this to accelerate compression of that video to H.264 (there are other accelerators available for other compression methods), but that's not much of a solution since the compression would still take time you apparently don't have.

The absolute fastest way to handle transfer of that much data is to ship hard drives back and forth (laptop drives are small enough to fit into a FedEx box with enough packing to protect them). DVDs will be cheaper to ship as they can go in envelopes rather than boxes, but then you have to account for the time it takes to burn them (a laptop probably has a 4x burner, so if you have two hours of video it'll take 30 minutes to burn). If your business revolves around this, buy a half dozen little USB or FireWire drives and overnight them. Insure them for what they're worth, including the loss of revenue you'd incur if they don't arrive or are damaged in transit. If you can spare the time, burn DVDs before shipping and send them too, for redundancy. If you can spare more time, burn multiple DVDs and ship them separately so if one gets lost, the other still has a chance of getting there.

Consumer high speed internet connections are not yet reliable and common enough to make doing this over the internet a viable business. If you need to move that amount of data that quickly, FedEx is pretty much it.
posted by fedward at 1:49 PM on November 6, 2007

I am with some of the others in that, maybe you should explain to us what it is that you want to achieve (thereby allowing us to suggest other alternatives) instead of maybe trying to solve the specific data transfer question. At work, we have to move several hundred GB of data on a regular basis to another location, and portable hard disks are best.

Also, I am confused as to your use of logmein/gotomypc in this context -- while they allow you to remotely control your computer you would still have to use a method of data transfer (ftp) to move the data to your current location. How do you actually move the data now -- again, if it is possible tell us what it is that you are doing (having to move several GB may not even be the solution you need)
posted by cusecase at 2:50 PM on November 6, 2007

any chance of compressing the data before transit?
posted by browolf at 4:15 PM on November 6, 2007

The files can not be anymore compressed.

At the moment I am using gotomypc to (drag and drop) 600mb of video files in/out. Works pretty well.. except it takes around 3 hours.

The problem with compressing before transfer is that it also takes about an hour or two just to recompress... thus wiping out the time saved during the transfer...

Maybe gigabytes of data everyday was too extreame...

I would be happy, if i can get about 1 gig a day back and forth...

I thought about streaming... .. but that doesn't really help storing the file.
posted by curiousleo at 1:11 AM on November 7, 2007