Back me up, Scotty!

June 18, 2008 4:02 PM

Recommend a good tool to remotely backup/synchronize large folders to my home NAS.

Here's what I have:

- Laptop with 100+ GB music library; misc documents, photos, and other data that needs backed up
- Netgear ReadyNAS Duo 750GB at home (supports pretty much everything - rsync, (S)FTP, WebDAV, etc)

Here's what I want to do:

For my MP3 library (100GB, 20k files, 4k folders) - mirror or synchronize any changes from my laptop (remote at work) to my home NAS. I've tried a TON of apps already with varying degrees of success. The major problem seems to be that to do a mirror, the apps have to perform a full "audit" scan of the destination (NAS) share to see what needs updated/copied. Problem is, with 20k files, that takes f-o-r-e-v-e-r, at least using FTP or WebDAV.

Ideally, I'd like a real-time backup app that monitors some selected folders (like My Music) and kicks off an automated upload for any changed files. This (a) makes it fully automatic and (b) [in theory] faster - doesn't have to synch each time, just upload based off a good known synch. Problem is, I've yet to find the right tool to accomplish all these things correctly.


Note: I'd be happy as a clam to use rsync, which the ReadyNAS supports, but I'm sending from a Windows client and my options are limited. I've been racking my brain trying to get DeltaCopy to work, but it keeps timing out when attempting to connect to the NAS. I have port forwarding, etc, working fine on my router, so I can't really figure out why rsync is failing.

Note 2: The ReadyNAS does have optional support for SSH and Telnet, I believe, but it involves some trickery to get working and I think it might void my warranty...

posted by sprocket87 to computers & internet (7 answers total)

The trouble is that rsync is really the canonical solution for this problem, so rsync alternatives are uncommon - people don't generally write one because rsync works so well. I suggest focusing on getting rsync working instead of trying a bunch of different programs. Can you bring your work laptop home and test from inside the router? That would eliminate the router and let you fix whatever basic problems might exist before you try to get the port forwarding working.

Other than that, I also recommend you reconsider your "ideal" situation. In my experience, you will use backups to recover accidentally deleted or damaged files much more often than you will use them to recover from hardware failure. If the files are being automatically synced, the damage will be synced to your backups just as fast. A nightly or manual backup is much safer.
posted by pocams at 6:07 PM on June 18, 2008

I don't how your NAS works, but I've had luck using SyncBack freeware. I sync all my music from my work computer to my home computer (both running XP) over a VPN enabled by Hamachi. I've got 10k or so music files and the compare only takes a few seconds. The actual sync takes longer, of course.
posted by tayknight at 7:16 PM on June 18, 2008

Dude, Foldershare will blow your doors, it is awesome. I have 40,000 songs in two locations, work and home, and Foldershare keeps them in sync, PLUS it has a web based part that allows you to sign in and browse any of your Foldershared computers from anywhere in the world.
posted by Cosine at 10:59 PM on June 18, 2008

oh oh oh, and it does it all over the net so no worries about firewalls and such!
posted by Cosine at 11:01 PM on June 18, 2008

and apparently it's down for maintenance right now... lol
posted by Cosine at 11:10 PM on June 18, 2008

@ Cosine: Yup, the pinnacle of reliability ;)
But seriously, FolderShare won't work for me because (AFAIK) it requires its software client to be installed on both sides. In my scenario, I've got a hardware NAS on one side which means I can't run any old piece of Windows software on it. I'm pretty much restricted to the built-in services, with a little bit of add-on flexibility, but it's a big stinkin' (read: *nix) headache.

@ tayknight: Thanks for the tip, I gave SyncBack a try right after posting this question; dunno how I managed to overlook that one. It's VERY impressive and may just do the trick, as it seems to allow me to do an initial synchronize and then just update thereafter, with plenty of granularity in between.

@ pocams: Worthy advice, thanks. You'll be amused to know that I DID end up getting rsync working, and I'm ashamed to admit that the problem was with my port forwarding, despite my claims to the contrary :)
So now I just need to figure out how to make rsync do its thing automatically... Any tips on real-time rsync backups in Windows [with DeltaCopy?] are appreciated.

I'm giving consideration to your advice on the pitfalls of real-time backup as well, though I still think I'd like to employ it for at least part of my backup scheme. But thanks again for your thoughts.


Jesse
posted by sprocket87 at 5:38 AM on June 19, 2008

Update: rsync rocks! Fast and efficient. Only problem thus far is that cygwin/rsync skips any file over 260 characters, and based on a bit of Googling there isn't much I can do about it. So a few of my folders are getting skipped, but I can handle renaming those few (or skipping them, as they're static and I can just manually sync them one time).

Also, it handles weird characters in an odd way, and filenames with a non-standard character (such as the "ö" in "Björk" or "Röyksopp") get treated like a different file altogether, and are re-uploaded based on how rsync handles that character. Not sure if that is Unicode/UTF/whatever, I don't know anything about that stuff (or how to fix it!).

I can live with that stuff though =)
posted by sprocket87 at 7:22 AM on June 19, 2008