Keeping it backed up and in synch
April 26, 2011 3:55 PM Subscribe
Continuous Backup of external drives OS X 10.6 - best system?
Hardware:
1) 2.8GHz i7 iMac 2009, running fully updated OS X 10.6.7.
2) Drive A - 4TB Data Tale External Drive - 2TB + 2TB in RAID 1 for 2TB capacity
3) Drive B - 4TB Duo Pro External Drive - 2TB + 2TB in RAID 1 for 2TB capacity
I use Drive A as a music server on which I store all my music files organized by iTunes (10.2.2 fully updated), while the iTunes library file is on the iMac. Any music files I add, goes immediately to this drive.
I have had a different 4TB drive organized in RAID 1 have a near failure of the controller, so now I'm paranoid about losing my files to a similar mishap. I copied the files onto a new drive A, which I've been running the past few weeks, but I have also purchased Drive B to act as a backup to Drive A.
Here's a complication: I add music files and also remove music files virtually every day. Therefore, I would like something like a Time Machine function, which automatically (on perhaps an hourly basis?) backs up any changes made on drive A to synch with the backup files on drive B, so that drive B is always a current copy of drive A. Since both drives are live and permanently connected to the iMac, I'd like any such synching to happen automatically and in the background, without my having to take any action. I can use either USB or FW800. And keeping in mind that both drives are individually in a RAID1 configuration.
What is the best protocol to set up this? Do I need to use any particular application to do this?
Hardware:
1) 2.8GHz i7 iMac 2009, running fully updated OS X 10.6.7.
2) Drive A - 4TB Data Tale External Drive - 2TB + 2TB in RAID 1 for 2TB capacity
3) Drive B - 4TB Duo Pro External Drive - 2TB + 2TB in RAID 1 for 2TB capacity
I use Drive A as a music server on which I store all my music files organized by iTunes (10.2.2 fully updated), while the iTunes library file is on the iMac. Any music files I add, goes immediately to this drive.
I have had a different 4TB drive organized in RAID 1 have a near failure of the controller, so now I'm paranoid about losing my files to a similar mishap. I copied the files onto a new drive A, which I've been running the past few weeks, but I have also purchased Drive B to act as a backup to Drive A.
Here's a complication: I add music files and also remove music files virtually every day. Therefore, I would like something like a Time Machine function, which automatically (on perhaps an hourly basis?) backs up any changes made on drive A to synch with the backup files on drive B, so that drive B is always a current copy of drive A. Since both drives are live and permanently connected to the iMac, I'd like any such synching to happen automatically and in the background, without my having to take any action. I can use either USB or FW800. And keeping in mind that both drives are individually in a RAID1 configuration.
What is the best protocol to set up this? Do I need to use any particular application to do this?
Response by poster: jenkinsEar - I don't think this is what Time Machine does exactly - I think TM creates distinct folders with new data, rather than synching. I like the automagical-in-the-background action of TM and would like to emulate that, but it does something different.
odinsdream - doesn't this require me to initiate the synching every time? Or does one set up some kind of cron job? I'd like something I can set up once, and forget, like Time Machine (though TM doesn't synch exactly). Also, I need something pretty precise, because I need all the metadata preserved for iTunes, so in case drive A conks out, I can simply switch to drive B, without a hitch.
The best way to describe it, would be like a RAID 1 where I'm mirroring drive A to drive B, so any changes from A are reflected in B automatically.
posted by VikingSword at 4:11 PM on April 26, 2011
odinsdream - doesn't this require me to initiate the synching every time? Or does one set up some kind of cron job? I'd like something I can set up once, and forget, like Time Machine (though TM doesn't synch exactly). Also, I need something pretty precise, because I need all the metadata preserved for iTunes, so in case drive A conks out, I can simply switch to drive B, without a hitch.
The best way to describe it, would be like a RAID 1 where I'm mirroring drive A to drive B, so any changes from A are reflected in B automatically.
posted by VikingSword at 4:11 PM on April 26, 2011
Time Machine will give you "a current copy of drive A" as you desire. It will also give you past copies of drive A which you are free to ignore. I'm not sure how Time Machine isn't exactly what you want except the name of the drive is funny and you'll need to do a little work to restore.
Disk utility can set up mirrored raids, if that's what want. I've never used it though.
posted by chairface at 4:33 PM on April 26, 2011
Disk utility can set up mirrored raids, if that's what want. I've never used it though.
posted by chairface at 4:33 PM on April 26, 2011
I'd like something I can set up once, and forget
rsync has a daemon mode that can be started automatically with launchd. For a music collection, you probably won't need the version that handles resource forks, but if you do, it's compilable.
For a simpler GUI implementation, you could run SuperDuper with its scheduled "Smart Update" incremental backup.
posted by holgate at 4:36 PM on April 26, 2011
rsync has a daemon mode that can be started automatically with launchd. For a music collection, you probably won't need the version that handles resource forks, but if you do, it's compilable.
For a simpler GUI implementation, you could run SuperDuper with its scheduled "Smart Update" incremental backup.
posted by holgate at 4:36 PM on April 26, 2011
Response by poster: I considered using Disk utility to set up software RAID1 A to B, but I'm somewhat concerned if this would work if both A and B are in turn internally hardware RAIDed - so I just don't know...
Re: TM, perhaps I'm not fully understanding Time Machine - I didn't think you could use it to work between two external hdds... I guess I should investigate what TM does exactly, though I'm not crazy about filling up a drive with extra folders with time stamps.
Re: SuperDuper - I'll look into that, thanks.
posted by VikingSword at 4:41 PM on April 26, 2011
Re: TM, perhaps I'm not fully understanding Time Machine - I didn't think you could use it to work between two external hdds... I guess I should investigate what TM does exactly, though I'm not crazy about filling up a drive with extra folders with time stamps.
Re: SuperDuper - I'll look into that, thanks.
posted by VikingSword at 4:41 PM on April 26, 2011
Best answer: One caveat: mirroring automatically runs the minimal risk of having, say, silent data corruption in Array A mirrored onto Array B. I see a lot of drive redundancy in your ideal setup, but I don't see much redundancy in the actual backup procedure.
You might be better off breaking the RAID 1 in array B, formatting each drive separately, and alternating your backups. In that scheme, B1 is your hourly backup for a week, then becomes the weekly snapshot and B2 becomes the hourly. That way, if something really catastrophic happens in terms of mechanical failure or corruption, you at least have a weekly snapshot. And for paranoia's sake, buy another 2TB drive, sync it monthly, and store it offsite: that's your "house burns down" backup.
posted by holgate at 4:45 PM on April 26, 2011 [1 favorite]
You might be better off breaking the RAID 1 in array B, formatting each drive separately, and alternating your backups. In that scheme, B1 is your hourly backup for a week, then becomes the weekly snapshot and B2 becomes the hourly. That way, if something really catastrophic happens in terms of mechanical failure or corruption, you at least have a weekly snapshot. And for paranoia's sake, buy another 2TB drive, sync it monthly, and store it offsite: that's your "house burns down" backup.
posted by holgate at 4:45 PM on April 26, 2011 [1 favorite]
Response by poster: Thanks, holgate, data corruption wasn't on my horizon so much, I was much more worried about the raid controller failing in some way in software (which is why I avoided using disk utility) or hardware (which is where the second drive B comes in). I suppose breaking B into B1 and B2 as you suggest is possible, but I'm worried about how to automate all this, as I don't want to spend time babysitting syncing processess, copying, backing up etc.
posted by VikingSword at 4:54 PM on April 26, 2011
posted by VikingSword at 4:54 PM on April 26, 2011
I'm worried about how to automate all this
SuperDuper's scheduler makes it easy to automate a scheme like the one I described, alternating between target drives, but it only works for daily, not hourly backups. The creator suggests Time Machine if you want that kind of frequency, though rsync would handle it too: it's your call whether hourly incrementals are a necessity.
I guess I should investigate what TM does exactly, though I'm not crazy about filling up a drive with extra folders with time stamps.
You'd just need to exclude the internal drive/volume on the iMac from the backup, or only include the iTunes library file. The "extra folders with time stamps" really aren't that big of a deal as long as you're working within the TM interface, but it's definitely different from the traditional approach to backup.
posted by holgate at 5:20 PM on April 26, 2011
SuperDuper's scheduler makes it easy to automate a scheme like the one I described, alternating between target drives, but it only works for daily, not hourly backups. The creator suggests Time Machine if you want that kind of frequency, though rsync would handle it too: it's your call whether hourly incrementals are a necessity.
I guess I should investigate what TM does exactly, though I'm not crazy about filling up a drive with extra folders with time stamps.
You'd just need to exclude the internal drive/volume on the iMac from the backup, or only include the iTunes library file. The "extra folders with time stamps" really aren't that big of a deal as long as you're working within the TM interface, but it's definitely different from the traditional approach to backup.
posted by holgate at 5:20 PM on April 26, 2011
Response by poster: Thanks, holgate, I guess I really should look into SuperDuper. For now I have my TM backing up my iMac internal drive to an external hdd, and I just don't have the confidence that I understand TM well enough to have it in addition backup between all these drives. I'm fcsking drowning in drives - I have 8 hooked up to my iMac at all times (some for data storage, some for editing etc.), cables and multiple hubs everywhere, and the whole thing is starting to really depress and get me down; where's my promised cloud with 10TB I can access and stream from at any time, from my one clean untethered iMac?
posted by VikingSword at 5:37 PM on April 26, 2011
posted by VikingSword at 5:37 PM on April 26, 2011
Best answer: Carbon copy cloner does pretty much everything superduper does, but for free. It also has some special things as well. I have used both quite happily. I used them to do pretty much the same thing you do, in addition to maintaining bootable backups. Time Machine(though great at what it does) is not what you want here.
posted by rockindata at 6:13 PM on April 26, 2011
posted by rockindata at 6:13 PM on April 26, 2011
I'm not sure how Time Machine isn't exactly what you want
Time Machine is not a data synchronization system for two different computers. The backup store tracks computers with separate timelines, so you cannot use Time Machine to do what VikingSword wants. You can share a USB or FireWire backup disk between two computers, but they will not share the same timelines.
posted by secret about box at 6:53 PM on April 26, 2011
Time Machine is not a data synchronization system for two different computers. The backup store tracks computers with separate timelines, so you cannot use Time Machine to do what VikingSword wants. You can share a USB or FireWire backup disk between two computers, but they will not share the same timelines.
posted by secret about box at 6:53 PM on April 26, 2011
Right, VS is talking about a single computer with two disks. Sorry, pretty distracted here at work today.
Time Machine can't synchronize one disk to another (e.g., disk A to disk B, in this situation). The closest it can do here is have disk A being backed up and disk B set as the backup disk. That's not really "synchronization", however, because you can't add files to the backup and if you remove files from the backup, those changes won't be reflected on disk A.
If VS just wants a backup of disk A, using disk B as the backup disk, Time Machine can do that.
posted by secret about box at 7:04 PM on April 26, 2011
Time Machine can't synchronize one disk to another (e.g., disk A to disk B, in this situation). The closest it can do here is have disk A being backed up and disk B set as the backup disk. That's not really "synchronization", however, because you can't add files to the backup and if you remove files from the backup, those changes won't be reflected on disk A.
If VS just wants a backup of disk A, using disk B as the backup disk, Time Machine can do that.
posted by secret about box at 7:04 PM on April 26, 2011
Carbon copy cloner does pretty much everything superduper does, but for free.
Thanks for getting me to re-check: I've always liked CCC, but last time I used it, a few years ago, it didn't do incremental backups. Now it does, and offers hourly scheduling too.
posted by holgate at 8:36 PM on April 26, 2011
Thanks for getting me to re-check: I've always liked CCC, but last time I used it, a few years ago, it didn't do incremental backups. Now it does, and offers hourly scheduling too.
posted by holgate at 8:36 PM on April 26, 2011
Nthing CCC. BTW, it uses rsync underneath.
posted by StrawberryPie at 10:02 PM on April 26, 2011
posted by StrawberryPie at 10:02 PM on April 26, 2011
Best answer: I'm not crazy about filling up a drive with extra folders with time stamps
Time Machine is clever about re-using old files if they haven't changed. So if you have 100GB of music on your live drive, and you make a TM backup, then add 1GB of new music, then make another TM backup, your backup drive will now have 101GB of stuff on it, not 201GB. If you then delete 20GB of music from your live drive and make another TM backup, the backup drive will still have that same 101GB stored on it - even though it will appear to have three separate time-stamp folders containing 100GB, 101GB and 81GB of files respectively.
Basically, TM will keep as little on the backup as it has to in order to make it possible for you to restore any of the point-in-time backups you tell it to keep. What's on the backup drive grows when you add new stuff, but only by the size of the new stuff. Files only get removed from the backup drive when you remove all the point-in-time snapshots in which they are included.
The only way to make TM use ridiculous amounts of space is to ask it to back up files that are both large and variable. For example, if you're using a mail client like Thunderbird that keeps all your downloaded mails in a single huge mbox file, and new mails arrive between TM backup sessions, then TM has no option but to grab a fresh copy of the entire mbox; I don't believe it's smart enough to store just the differences within a single file. Which is one of the reasons that Apple's own mail app uses maildir files, with one file per message instead of per mailbox.
posted by flabdablet at 1:09 AM on April 27, 2011
Time Machine is clever about re-using old files if they haven't changed. So if you have 100GB of music on your live drive, and you make a TM backup, then add 1GB of new music, then make another TM backup, your backup drive will now have 101GB of stuff on it, not 201GB. If you then delete 20GB of music from your live drive and make another TM backup, the backup drive will still have that same 101GB stored on it - even though it will appear to have three separate time-stamp folders containing 100GB, 101GB and 81GB of files respectively.
Basically, TM will keep as little on the backup as it has to in order to make it possible for you to restore any of the point-in-time backups you tell it to keep. What's on the backup drive grows when you add new stuff, but only by the size of the new stuff. Files only get removed from the backup drive when you remove all the point-in-time snapshots in which they are included.
The only way to make TM use ridiculous amounts of space is to ask it to back up files that are both large and variable. For example, if you're using a mail client like Thunderbird that keeps all your downloaded mails in a single huge mbox file, and new mails arrive between TM backup sessions, then TM has no option but to grab a fresh copy of the entire mbox; I don't believe it's smart enough to store just the differences within a single file. Which is one of the reasons that Apple's own mail app uses maildir files, with one file per message instead of per mailbox.
posted by flabdablet at 1:09 AM on April 27, 2011
Best answer: Why not use Time Machine on one backup drive and SuperDuper on the other? (Recommended here by John Siracusa, if you have time to listen.) This will get you the file version backups and easy restore of Time Machine plus the redundant, bootable backups of SuperDuper.
SuperDuper (if you pay for it, and do: backups are worth the meager $20) will do a "smart copy" which does a brute force comparison between every file on your drive and the backup and only copies new files. It's pretty quick as things go and you can schedule it. Once an hour is probably not needed if you already have Time Machine doing that. Daily or even weekly will be good enough.
posted by davextreme at 7:08 AM on April 27, 2011 [1 favorite]
SuperDuper (if you pay for it, and do: backups are worth the meager $20) will do a "smart copy" which does a brute force comparison between every file on your drive and the backup and only copies new files. It's pretty quick as things go and you can schedule it. Once an hour is probably not needed if you already have Time Machine doing that. Daily or even weekly will be good enough.
posted by davextreme at 7:08 AM on April 27, 2011 [1 favorite]
Response by poster: Thank you everybody - I've learned a lot. As expected, there are many ways to do this, so I'll have to pick one. I'm leaning toward using either SD or CCC. I learned something from every post here - thank you again, everyone!
posted by VikingSword at 11:21 AM on April 27, 2011
posted by VikingSword at 11:21 AM on April 27, 2011
This thread is closed to new comments.
posted by jenkinsEar at 4:00 PM on April 26, 2011