Recommendations for synchronizing 1TB of data on 3 computers?
January 15, 2011 10:15 AM Subscribe
Recommendations for synchronizing 1TB of data on 3 computers?
I have about a terabyte of data (images and video) that I keep synchronized between 3 computers--a laptop and 2 desktops--for my work. All of these computers are usually in different geographic locations, so a local server is not an option. To synchronize I have been using Dropbox.com, but this will not synch more than 100GB. I am thus left with a logistical pain in the butt of constantly shuffling files in and out of the dropbox, or using portable hard drives.
My human propensity for error often results in 3 slightly different file sets on each of the computers--a constant nightmare to sort out. I am thus hoping to find a fully automated solution that works over the Internet. Are there services or software that you recommend for this type of task? It would need to be installed on Microsoft Windows machines. Any suggestions would be most appreciated.
I have about a terabyte of data (images and video) that I keep synchronized between 3 computers--a laptop and 2 desktops--for my work. All of these computers are usually in different geographic locations, so a local server is not an option. To synchronize I have been using Dropbox.com, but this will not synch more than 100GB. I am thus left with a logistical pain in the butt of constantly shuffling files in and out of the dropbox, or using portable hard drives.
My human propensity for error often results in 3 slightly different file sets on each of the computers--a constant nightmare to sort out. I am thus hoping to find a fully automated solution that works over the Internet. Are there services or software that you recommend for this type of task? It would need to be installed on Microsoft Windows machines. Any suggestions would be most appreciated.
Seconding rsync.
posted by dougrayrankin at 10:38 AM on January 15, 2011
posted by dougrayrankin at 10:38 AM on January 15, 2011
Moving 1 TB over the network will be slow and expensive. Use rsync to synchronize each computer to its own portable, external hard drive.
posted by Blazecock Pileon at 10:41 AM on January 15, 2011
posted by Blazecock Pileon at 10:41 AM on January 15, 2011
I have been using Mozy successfully. They have unlimited storage for paying customers. And they have just improved the interface so you can go online and download as little as one file faster and with less hassle (I guess before it was more "restore my backup hard drive" and now it's more "just let me download one damn file").
The procedure was easy for me to learn in about 15 minutes.
* I do not work for Mozy or own Mozy stock : )
posted by cda at 10:42 AM on January 15, 2011
The procedure was easy for me to learn in about 15 minutes.
* I do not work for Mozy or own Mozy stock : )
posted by cda at 10:42 AM on January 15, 2011
You can accomplish this while carrying around 1 portable drive for syncronization, you could do something like the following (assuming that you keep your data structure/organization unchanged between all PCs):
Make a batch file on all three PCs called Sync.bat (or sync.cmd, the name doesn't matter itself) with the contents:
----
rem File sync batch file
xcopy /d /e /v /y "(drive letter and path on backup drive)" "(drive and media folder on PC)"
xcopy /d /e /v /y "(media folder on PC) "(path on backup drive)"
----
So what's basically happening here is, all the newer files from the portable drive are being copied to the PC. If you had in there for example, xcopy /d /e /v /y "e:\media" "c:\users\bob\my documents" it would copy ever file in e:\media and its subdirectories (the /e switch) which date is newer than the target c:\users\bob\my documents (the /d switch), verify the file is copied fully (optional /v switch) and not prompt for overwriting the older file (the /y switch).
Afterwards, the opposite is done from the PC to the drive to put the drive in sync.
What's important however, is to make sure you keep the same file/directory structure in all locations to avoid duplicates. Also make sure your PCs clocks are set to the right date/time before syncronization, as this script is only making judgements on the datetime stamp of the files themselves. Hope this could help as a cheap Windows solution....rsync is still worth looking into as it give a little more flexibility.
posted by samsara at 10:57 AM on January 15, 2011
Make a batch file on all three PCs called Sync.bat (or sync.cmd, the name doesn't matter itself) with the contents:
----
rem File sync batch file
xcopy /d /e /v /y "(drive letter and path on backup drive)" "(drive and media folder on PC)"
xcopy /d /e /v /y "(media folder on PC) "(path on backup drive)"
----
So what's basically happening here is, all the newer files from the portable drive are being copied to the PC. If you had in there for example, xcopy /d /e /v /y "e:\media" "c:\users\bob\my documents" it would copy ever file in e:\media and its subdirectories (the /e switch) which date is newer than the target c:\users\bob\my documents (the /d switch), verify the file is copied fully (optional /v switch) and not prompt for overwriting the older file (the /y switch).
Afterwards, the opposite is done from the PC to the drive to put the drive in sync.
What's important however, is to make sure you keep the same file/directory structure in all locations to avoid duplicates. Also make sure your PCs clocks are set to the right date/time before syncronization, as this script is only making judgements on the datetime stamp of the files themselves. Hope this could help as a cheap Windows solution....rsync is still worth looking into as it give a little more flexibility.
posted by samsara at 10:57 AM on January 15, 2011
Windows live Mesh might be worth looking into. It is the successor to Windows Live Sync, and before that, foldershare. It is free, does multi-computer sync over the net. I used Live Sync and Foldershar (albeit for much smaller filesets than you're using) and really loved both. My initial experiences with Mesh are a little problematic, but I'm still hoping they get ironed out.
You might want to check out Wikiepdia's page comparing file sync software
posted by ManInSuit at 11:36 AM on January 15, 2011
You might want to check out Wikiepdia's page comparing file sync software
posted by ManInSuit at 11:36 AM on January 15, 2011
Rsync, or the windows equiv, which is deltacopy. No server necessary.
posted by Brian Puccio at 12:12 PM on January 15, 2011
posted by Brian Puccio at 12:12 PM on January 15, 2011
Moving 1 TB over the network will be slow and expensive
But how much of the data actually changes each day? The whole point of rsync is that it sends only checksums over the network until it finds data that needs to be copied — and it does this even within files, sending only the parts of the files that need to be updated. If the three workstations have almost the same files now, you should just be able to run one rsync to reconcile the differences, and then each night's rsync would just copy over newly changed data.
posted by nicwolff at 12:33 PM on January 15, 2011 [1 favorite]
But how much of the data actually changes each day? The whole point of rsync is that it sends only checksums over the network until it finds data that needs to be copied — and it does this even within files, sending only the parts of the files that need to be updated. If the three workstations have almost the same files now, you should just be able to run one rsync to reconcile the differences, and then each night's rsync would just copy over newly changed data.
posted by nicwolff at 12:33 PM on January 15, 2011 [1 favorite]
If you're using only Windows machines, Live Mesh is probably okay; but if you throw in a Mac in the mix, I'd caution against it--- I've had lots of trouble (and a quick look at the forums shows I'm not the only one).
posted by cuetip at 3:54 PM on February 3, 2011
posted by cuetip at 3:54 PM on February 3, 2011
This thread is closed to new comments.
Info about rsync: http://en.wikipedia.org/wiki/Rsync
Another more detailed tutorial from Lifehacker: http://lifehacker.com/5657283/how-to-make-your-own-open-source-dropbox+like-sync-and-backup-service
posted by msbutah at 10:33 AM on January 15, 2011