Rsync to Merge Disk Images?

August 30, 2009 11:18 PM

Should I use rsync (instead of rsnapshot) to pare down Mac disk image backups?

I've used SuperDuper to make several disk image backups of a boot drive.

Let's say they're called backup-090107.dmg, backup-090114.dmg, backup-090121.dmg, and so on.

I want to merge all these into a single backup directory, RETAINING incremental backups. So if file-a.txt changed on Jan 16 2009 (between backup-090114 and backup-0090121), I'd end up with a copy of how it was on 090114 and also a copy of how it was on 090121.

I tried rdiff-backup but gave up on it due to insane slowness. (It takes SuperDuper 5-6 hours to back up 60-70 GB over USB2; rdiff-backup was taking about 16 hours to back up just the 7GB applications directory. Yikes.)

I've started looking at rsnapshot, but I don't think that's what I want for this initial merge ... although it might be what I want for any subsequent backups, once I've got the disk images merged.

So:

* am I correct in thinking that rsync is best for merging these dmgs?

* would the following commands do what I want to do - so I end up with a single copy of any file that hasn't changed, plus incremental backups of each file that HAS changed?

rsync -a --delete --numeric-ids --relative --delete-excluded --xattrs --exclude=/this/ --exclude=/that/ \
/Volumes/backup-090107 /Volumes/BackupDrive/big-old-merge-directory/backup-090107/

rsync -a --delete --numeric-ids --relative --delete-excluded --xattrs --exclude=/this/ --exclude=/that/ \
/Volumes/backup-090114 /Volumes/BackupDrive/big-old-merge-directory/backup-090114/


* should I switch to rsnapshot after merging, or continue using rsync? If I continue with rsync, would I use something like

rsync -a --delete --numeric-ids --relative --delete-excluded --xattrs --exclude=/this/ --exclude=/that/ \
/MyHardDrive /Volumes/BackupDrive/big-old-merge-directory/backup-todaysdate/

Extra bonus points for lists of directories you like to exclude, or pointers to pages that list appropriate OS X (10.4) directories to exclude. (I DO want to back up even big directories; I don't need to back up the unnecessary or dangerous - for example, I know most folks exclude .Spotlight-V100.)

Once I've merged the backups (and verified them, of course), I'll be deleting the disk images to free up space for future backups. (And I'm doing full, bootable backups to a different drive; this is just my attempt to get good incremental backups.)

Many, many thanks!

posted by kristi to computers & internet (4 comments total)

The canonical article for this (and the one that rsnapshot references) is this one. I think you're going to want to create one directory per .dmg file -- the first copy is a straight rsync, and each subsequent one will use --link-dest to link to the previous. You'll end up with a set of dated directories, each with a complete image, and every file will be hard linked so that only one copy of the file will exist, maximum.
posted by brool at 12:12 AM on August 31

Rsync will work Much Better copying to directories instead of disk images. You can't remove duplicates with hard links between disk images. Switching to directories & hard links should save you a lot of space.
posted by Pronoiac at 10:55 AM on August 31

am I correct in thinking that rsync is best for merging these dmgs?

Rsync is a good choice.


would the following commands do what I want to do - so I end up with a single copy of any file that hasn't changed, plus incremental backups of each file that HAS changed?

No. First of all, be very careful with the --delete option. It will delete files on the destination directory if they don't exist any more in the source. This is not what you want for incremental backups.

With the commands you've listed, you would just be doing a straight copy of each .dmg file to a directory in the merge directory. Use the --link-dest option for incremental backups, to create hard links to files that already exist and are unchanged. For example:

rsync /original-source /backup-monday
rsync --link-dest /backup-monday /original-source /backup-tuesday

The second rsync command will only copy new and modified files to backup-tuesday. Everything else will be a hard link to files in backup-monday, so there's only really one copy on the filesystem.

Man rsync. And the website posted above. :)
posted by lalas at 11:08 AM on August 31

Thanks for the help!

Brool, I had seen that article, but I didn't see a lot of OS X-specific stuff in there, so I wondered if anyone had any Mac-oriented tips to share.

Pronoiac - sorry if I wasn't clear: I have disc images that I am indeed planning to merge into directories. I am not planning to merge TO disc images.

Lalas - thanks so much for pointing out how much I definitely do not want to use --delete. Your example was most helpful!

Thanks again!
posted by kristi at 10:51 AM on September 4