Temporary SFTP/Rsync for traveling videography
March 20, 2012 10:40 AM Subscribe
Temporary Cloud Backup on the road: I'm traveling this summer for 2 months in Europe and will be shooting HD footage at a rate of at most 64 GB a day (4 hrs) documenting music festivals. What would be the cheapest solution to do daily backups to a hosted service? By the end of the trip I at most will have 3 TB of video.
I currently have a BackBlaze account and have them for my desktop at home, but I worry that they won't be able to 'catch up' nightly if I tried to use them for on the road backups for a massive amount of files.
I have looked into Rsync and Amazon S3, but was curious if there is a better hosted solution that I can SFTP my raw footage up to nightly (or whenever I have access to WiFi) so that if my laptop dies I at most would have lost 1 day of video.
This would only be needed for 2 months, and after that I will transfer my footage back onto my computer, delete the unnecessary crap and then let BackBlaze 'catch up' with the remaining files.
I currently have a BackBlaze account and have them for my desktop at home, but I worry that they won't be able to 'catch up' nightly if I tried to use them for on the road backups for a massive amount of files.
I have looked into Rsync and Amazon S3, but was curious if there is a better hosted solution that I can SFTP my raw footage up to nightly (or whenever I have access to WiFi) so that if my laptop dies I at most would have lost 1 day of video.
This would only be needed for 2 months, and after that I will transfer my footage back onto my computer, delete the unnecessary crap and then let BackBlaze 'catch up' with the remaining files.
seconding flab here
you're going to have a very tough time uploading that much data on a publically available connection at a hotel/hostel/cafe
64 Gb USB drives are $45 or so a pop, I'd buy a case of them and mail them to myself at regular intervals.
posted by Oktober at 10:48 AM on March 20, 2012 [1 favorite]
you're going to have a very tough time uploading that much data on a publically available connection at a hotel/hostel/cafe
64 Gb USB drives are $45 or so a pop, I'd buy a case of them and mail them to myself at regular intervals.
posted by Oktober at 10:48 AM on March 20, 2012 [1 favorite]
Yea, uploading 64 gig of data a night from a hotel is an aggressive target. Uploading with a client that speaks S3's language to a European S3 endpoint is probably your best bet, but I'd focus on having a big backup drive back at the hotel or in the car, as well as 3-4 small USB drives that you mail home every two weeks.
posted by jeffkramer at 10:50 AM on March 20, 2012
posted by jeffkramer at 10:50 AM on March 20, 2012
Honestly, trying to reliably upload that kind of data will be migraine inducing.
In my last job, we had remote data collection vans that would do about 200 gigs per day, a TB per week, or so. We messed about with various upload schemes, but the simple fact is that you can't beat the bandwidth of a UPS truck with a couple hard drives in it.
We used a bunch of external drives, mailed out daily and weekly back to home base where they were copied up to the primary storage array for processing.
We lost drives sometimes to damage and just getting lost. But with a copy in the van, and copy in the weekly, we were OK.
Based on that, I'd recommend you get a friend to work as home base, use some USB and external drives to mail back and forth. They can upload to a storage system of your choice using a faster, more stable connection.
Think multiple copies - thats the key. Assume some will get lost and build your system to work around that. Use MD5 Hashing to verify data integrity - we saw some corruption occasionally.
posted by Pogo_Fuzzybutt at 10:50 AM on March 20, 2012
In my last job, we had remote data collection vans that would do about 200 gigs per day, a TB per week, or so. We messed about with various upload schemes, but the simple fact is that you can't beat the bandwidth of a UPS truck with a couple hard drives in it.
We used a bunch of external drives, mailed out daily and weekly back to home base where they were copied up to the primary storage array for processing.
We lost drives sometimes to damage and just getting lost. But with a copy in the van, and copy in the weekly, we were OK.
Based on that, I'd recommend you get a friend to work as home base, use some USB and external drives to mail back and forth. They can upload to a storage system of your choice using a faster, more stable connection.
Think multiple copies - thats the key. Assume some will get lost and build your system to work around that. Use MD5 Hashing to verify data integrity - we saw some corruption occasionally.
posted by Pogo_Fuzzybutt at 10:50 AM on March 20, 2012
Response by poster: Yeah, thinking about it needing that much bandwidth, it might be cheapest to buy 10-15 SDHC cards to supplement, and 3 cheapo USB drives to FedEx home at intervals.
posted by wcfields at 10:54 AM on March 20, 2012
posted by wcfields at 10:54 AM on March 20, 2012
That seems like the most reasonable solution, yes. Hell, even assuming that you'd never be booted off the network by connectivity or "please stop monopolizing our bandwidth, sir" reasons, I still very much doubt there's enough time in a day to upload 64 GB of data over a publicly available connection.
posted by Holy Zarquon's Singing Fish at 11:55 AM on March 20, 2012
posted by Holy Zarquon's Singing Fish at 11:55 AM on March 20, 2012
Just to do the math on that:
That's around the bandwidth of five MiniDV tapes, which is unwieldy. Uploading 64 gig in 8 hours requires a couple of megabytes a second of bandwidth. The usual for a DSL or cable connection is around an order of magnitude less than that.
posted by Pronoiac at 1:59 PM on March 20, 2012 [1 favorite]
That's around the bandwidth of five MiniDV tapes, which is unwieldy. Uploading 64 gig in 8 hours requires a couple of megabytes a second of bandwidth. The usual for a DSL or cable connection is around an order of magnitude less than that.
posted by Pronoiac at 1:59 PM on March 20, 2012 [1 favorite]
it might be cheapest to buy 10-15 SDHC cards to supplement
For the price of X gigabytes of flash storage you can usually get roughly 10X gigabytes of hard drive storage, and a typical USB hard drive will easily outpace a typical SDHC card reader/writer. 2.5" hard drives are small, light, strong enough to survive being mailed in a padded envelope and don't need additional power packs. I think you'd be better off making more copies on cheaper media.
posted by flabdablet at 10:27 PM on March 20, 2012
For the price of X gigabytes of flash storage you can usually get roughly 10X gigabytes of hard drive storage, and a typical USB hard drive will easily outpace a typical SDHC card reader/writer. 2.5" hard drives are small, light, strong enough to survive being mailed in a padded envelope and don't need additional power packs. I think you'd be better off making more copies on cheaper media.
posted by flabdablet at 10:27 PM on March 20, 2012
Also: you can use PAR2 to mitigate occasional data corruption.
posted by flabdablet at 10:33 PM on March 20, 2012
posted by flabdablet at 10:33 PM on March 20, 2012
Yes, you'll probably need hard storage (drives or flash-based): I can't imagine how you'd get access to that kind of bandwidth, short of knowing people with access to I2 or fat corprate networks. If you can afford it, I would make two copies of each night's dump and ship them separately, or maybe hold one and ship one home.
(Also, this serves to support the old "never underestimate the bandwidth of a stationwagon fulll of backup tapes" line. :7)
posted by wenestvedt at 1:15 PM on March 21, 2012
(Also, this serves to support the old "never underestimate the bandwidth of a stationwagon fulll of backup tapes" line. :7)
posted by wenestvedt at 1:15 PM on March 21, 2012
This thread is closed to new comments.
Surely a little clutch of USB hard drives would work better for this.
posted by flabdablet at 10:45 AM on March 20, 2012 [2 favorites]