RAID and backup system for Mac Pro?
October 13, 2009 2:08 PM Subscribe
Best RAID / Backup system for HD video on a Mac Pro?
I was in the process of building myself a nice little external RAID enclosure with four 1 TB drives, when I started digging into the articles and realizing that RAID isn't backup.
Now I have these 4 1 TB drives, a couple of 500 GB drives, 1 TB internal, and 640 GB system drive for my Mac Pro. So I'm looking for the best way to set up these drives in my workflow for producing HD video.
Here are my basic needs:
1. Reliable Daily backups - Daily backups of all my work. I'm still thinking of using the RAID esata box as a RAID1 setup, to prevent downtime if a drive goes down.
2. Long term backups - When a project is finished, moving the data off the working drive and onto two separate hard drives for storage. I'll buy additional 500 GB drives for this purpose as needed.
3. Speed isn't crucial, as I'm not working in uncompressed HD.
Time Machine seems to be a great built in system, and I'd like to take advantage of that. I'm also signing up for Carbonite for online backup of as much information as I can.
Any thoughts on the workflow and the best way to automate the system would be greatly appreciated.
I was in the process of building myself a nice little external RAID enclosure with four 1 TB drives, when I started digging into the articles and realizing that RAID isn't backup.
Now I have these 4 1 TB drives, a couple of 500 GB drives, 1 TB internal, and 640 GB system drive for my Mac Pro. So I'm looking for the best way to set up these drives in my workflow for producing HD video.
Here are my basic needs:
1. Reliable Daily backups - Daily backups of all my work. I'm still thinking of using the RAID esata box as a RAID1 setup, to prevent downtime if a drive goes down.
2. Long term backups - When a project is finished, moving the data off the working drive and onto two separate hard drives for storage. I'll buy additional 500 GB drives for this purpose as needed.
3. Speed isn't crucial, as I'm not working in uncompressed HD.
Time Machine seems to be a great built in system, and I'd like to take advantage of that. I'm also signing up for Carbonite for online backup of as much information as I can.
Any thoughts on the workflow and the best way to automate the system would be greatly appreciated.
Best answer: I'm glad that you've realized that RAID isn't backup
I suppose you know all this, but if you value your data -- if it is your business, livelihood, etc you shouldn't consider hard disk storage as a long-term archival medium. Disks die -- especially if they're not being used. So, RAID, you say. Hard drive controllers also fail. If you blow a controller, your nifty raid array has plenty o' meaningless data written to it. Please consider spending some money for LTO or DLT tape backup -- if you want to be absolutely sure the data isn't going anywhere, grab a SAS tape array, and get a SAS controller. Then, maybe, you can also get a SAS JBOD like this, and a cheap controller card, and you will be happy. I have used Enhance Tech's boxes in the past in super-uber-low budget situations, and they work.
If you've got ~$1500 to spend, we can point you in the direction of an awesome tape drive that will fulfill your archive needs for quite a long time.
posted by Geckwoistmeinauto at 3:15 PM on October 13, 2009 [1 favorite]
I suppose you know all this, but if you value your data -- if it is your business, livelihood, etc you shouldn't consider hard disk storage as a long-term archival medium. Disks die -- especially if they're not being used. So, RAID, you say. Hard drive controllers also fail. If you blow a controller, your nifty raid array has plenty o' meaningless data written to it. Please consider spending some money for LTO or DLT tape backup -- if you want to be absolutely sure the data isn't going anywhere, grab a SAS tape array, and get a SAS controller. Then, maybe, you can also get a SAS JBOD like this, and a cheap controller card, and you will be happy. I have used Enhance Tech's boxes in the past in super-uber-low budget situations, and they work.
If you've got ~$1500 to spend, we can point you in the direction of an awesome tape drive that will fulfill your archive needs for quite a long time.
posted by Geckwoistmeinauto at 3:15 PM on October 13, 2009 [1 favorite]
Reliable daily backups - Time machine. That's what it's good at. Get something high speed and directly attached via fw800.
Drobo seems to get good reviews - keep in mind the big plus with drobo is ease of manageability. Anyone can set up a raid array - but making sure it is maintained properly, and can recover from failures properly is another story. Typically people set up a raid, and then a drive fails (and they don't notice, because it was redundant) and then another drive failes, and they're hosed.
second: Make regular, deliberate, scheduled backups of specified folders to external storage (whether it's offsite or just other network devices) using simple bash scripts you write yourself (rsync, bash, whatever).
posted by TravellingDen at 3:57 PM on October 13, 2009
Drobo seems to get good reviews - keep in mind the big plus with drobo is ease of manageability. Anyone can set up a raid array - but making sure it is maintained properly, and can recover from failures properly is another story. Typically people set up a raid, and then a drive fails (and they don't notice, because it was redundant) and then another drive failes, and they're hosed.
second: Make regular, deliberate, scheduled backups of specified folders to external storage (whether it's offsite or just other network devices) using simple bash scripts you write yourself (rsync, bash, whatever).
posted by TravellingDen at 3:57 PM on October 13, 2009
I like my Drobo for backup.
If you need it faster and larger, go for the new iScsi enabled Drobo Pro.
posted by digividal at 4:28 PM on October 13, 2009
If you need it faster and larger, go for the new iScsi enabled Drobo Pro.
posted by digividal at 4:28 PM on October 13, 2009
Drobo is excellent. Combine that with Time Machine and you have your incremental daily backup solution. For your archival backups, you can get bare 500 GB SATA OEM hard drives dirt cheap, and use a SATA dock like this one or this one.
If you want real security, you should probably also be doing full bootable backups nightly to an external drive using something like SuperDuper. For real REAL security, get two drives, swap them out on a schedule and keep the one you're not using offsite.
posted by joshrholloway at 5:01 PM on October 13, 2009
If you want real security, you should probably also be doing full bootable backups nightly to an external drive using something like SuperDuper. For real REAL security, get two drives, swap them out on a schedule and keep the one you're not using offsite.
posted by joshrholloway at 5:01 PM on October 13, 2009
As an aside, the Mac Pro motherboards have two extra SATA connectors which are not used. You can convert them to eSATA connectors with one of these things. I did it with my Mac Pro, and it works just fine. Much faster than a USB enclosure.
posted by procrastination at 5:54 PM on October 13, 2009
posted by procrastination at 5:54 PM on October 13, 2009
Best answer: We have a NAS but I no longer like RAID for backup. The time to recover after a failure now that the NAS is 4x1TB is too long for comfort.
Anyway, my wife has a Mac Pro. I am obsessive about backup. Here is what we do, YMMV.
Her machine is
0/500GB 1/1TB 2/500GB 3/500GB
external 2TB FW800 #1 (usually turned off and unplugged from both machine and power)
external 2TB FW800 #2 (usually turned off and unplugged from both machine and power)
FW drives are partitioned into INSTALL [10gB]/CLONE[200GB]/BACKUP [remainder]. They are GUID partition scheme volumes.
0 is the boot volume and the install location for applications. There is nothing else on it. A superduper clone of this volume is present on each of the external drives.
1 is used for Time Machine only.
2 and 3 are RAID mirrors of each other.
Each of the BACKUP volumes on the external drives is used for CrashPlan. I attach these to the machine and power, alternating between them, once a week, overnight. One drive is kept off the premises, I swap them.
[Incidentally, the INSTALL partition is a clone of the Leopard DVD and is bootable [as are the clones], this happens to be quite convenient].
I really did not like Carbonite or Mozy; we are trying out (seeded) Crashplan Online now.
posted by rr at 8:51 PM on October 13, 2009
Anyway, my wife has a Mac Pro. I am obsessive about backup. Here is what we do, YMMV.
Her machine is
0/500GB 1/1TB 2/500GB 3/500GB
external 2TB FW800 #1 (usually turned off and unplugged from both machine and power)
external 2TB FW800 #2 (usually turned off and unplugged from both machine and power)
FW drives are partitioned into INSTALL [10gB]/CLONE[200GB]/BACKUP [remainder]. They are GUID partition scheme volumes.
0 is the boot volume and the install location for applications. There is nothing else on it. A superduper clone of this volume is present on each of the external drives.
1 is used for Time Machine only.
2 and 3 are RAID mirrors of each other.
Each of the BACKUP volumes on the external drives is used for CrashPlan. I attach these to the machine and power, alternating between them, once a week, overnight. One drive is kept off the premises, I swap them.
[Incidentally, the INSTALL partition is a clone of the Leopard DVD and is bootable [as are the clones], this happens to be quite convenient].
I really did not like Carbonite or Mozy; we are trying out (seeded) Crashplan Online now.
posted by rr at 8:51 PM on October 13, 2009
Response by poster: Thanks for all the help so far, exactly the sort of suggestions I was wanting.
I looked seriously into Drobo, but for me it doesn't give anything better than RAID even though it is very easy to setup and use. Not really a backup system, just better than running with no protection at all.
Geckwoistmeinauto, I'm definitely looking more into tape as a long-term archive format, although I wouldn't have at first. The downside is of course the price, but I'd ideally like to put together archives in about 50 GB chunks. Each commercial spot ranges from 10 GB - 50 GB of raw footage, project files, etc. Once I have a spot completed and on air, I'd like to be able to put all those files to duplicate long-term storage. Is LTO the best bet, or another option I've overlooked? It seems that most LTO tapes are now in the 200/400GB range if not higher; I could definitely make that work as well, I'm sure it'd be more economical anyway. Originally I put an estimate in our pricing for $25 per commercial spot for archival backup, (mainly for hardware costs with hard drives), but scaling that over a $1500 tape drive will be a bit tough. I am also limited to external drives on the Mac, but I could probably be convinced to revive a PC box sitting around to have the tape backup on another system that doesn't matter for time.
rr, is there a particular reason you didn't like Carbonite? I'm looking for an online service to just use as an extra precaution. The $55 for a year of Carbonite is less than a 500 GB drive, so if it works, great, if not I have all my other backups in place (after this post, hopefully).
posted by shinynewnick at 10:58 PM on October 13, 2009
I looked seriously into Drobo, but for me it doesn't give anything better than RAID even though it is very easy to setup and use. Not really a backup system, just better than running with no protection at all.
Geckwoistmeinauto, I'm definitely looking more into tape as a long-term archive format, although I wouldn't have at first. The downside is of course the price, but I'd ideally like to put together archives in about 50 GB chunks. Each commercial spot ranges from 10 GB - 50 GB of raw footage, project files, etc. Once I have a spot completed and on air, I'd like to be able to put all those files to duplicate long-term storage. Is LTO the best bet, or another option I've overlooked? It seems that most LTO tapes are now in the 200/400GB range if not higher; I could definitely make that work as well, I'm sure it'd be more economical anyway. Originally I put an estimate in our pricing for $25 per commercial spot for archival backup, (mainly for hardware costs with hard drives), but scaling that over a $1500 tape drive will be a bit tough. I am also limited to external drives on the Mac, but I could probably be convinced to revive a PC box sitting around to have the tape backup on another system that doesn't matter for time.
rr, is there a particular reason you didn't like Carbonite? I'm looking for an online service to just use as an extra precaution. The $55 for a year of Carbonite is less than a 500 GB drive, so if it works, great, if not I have all my other backups in place (after this post, hopefully).
posted by shinynewnick at 10:58 PM on October 13, 2009
Backups are funny -- I occasionally do consulting for a wide array of industries, but most of my work is within the research field. Whenever I am given a budget, storage is always accounted for, but not backup, disaster recovery, or what have you. I always bring up the point that it's important to see how much it would cost to recreate your data versus what it would cost to make appropriate backups. There are many times where it's not worth the money to back stuff up because you could just re-run the experiment, or whatever, and get your stuff back. Sometimes it's cheaper.
I think what you're going for here is not so much 'oops' backup, and not really 'disaster recovery', as much as long-term archiving. I will admit that purchasing an LTO3 drive is probably a luxury that really isn't worth the purchase. I do, however, see external SCSI LTO2 drives on EBay for under $500, even $299. That, plus a decent SCSI adapter, will be pretty cheap. Tapes are in the neighborhood of $30 for 200GB native (400GB compressed).
I think this is a bargain for long term archival storage, and if you are willing to run the admittedly minute risk of losing your current job's work on if RAID array crashes, but you've got pretty much everything else up to that on archived tape... Well, that's not so bad. Feel free to MeMail me, and we can talk more.
posted by Geckwoistmeinauto at 9:44 AM on October 14, 2009
I think what you're going for here is not so much 'oops' backup, and not really 'disaster recovery', as much as long-term archiving. I will admit that purchasing an LTO3 drive is probably a luxury that really isn't worth the purchase. I do, however, see external SCSI LTO2 drives on EBay for under $500, even $299. That, plus a decent SCSI adapter, will be pretty cheap. Tapes are in the neighborhood of $30 for 200GB native (400GB compressed).
I think this is a bargain for long term archival storage, and if you are willing to run the admittedly minute risk of losing your current job's work on if RAID array crashes, but you've got pretty much everything else up to that on archived tape... Well, that's not so bad. Feel free to MeMail me, and we can talk more.
posted by Geckwoistmeinauto at 9:44 AM on October 14, 2009
rr, is there a particular reason you didn't like Carbonite? I'm looking for an online service to just use as an extra precaution. The $55 for a year of Carbonite is less than a 500 GB drive, so if it works, great, if not I have all my other backups in place (after this post, hopefully).
I did not care for their SW (or, honestly, their astroturfing on Amazon).
One of the appeals of crashplan is that it can be applied to both local and online backups. This removes one variable...
I did not mention, but I should, that I also back up my laptop to my wife's external crashplan volumes as well as a volume of my own. The ability to do machine-to-machine backups over the LAN at home is quite useful; eventually my intent is to no longer host the external drives on her machine directly at all and instead put a Mac Mini somewhere which can connect to them.
Just to add a bit of justification to my original post -- Mozy was an unbelievable piece of crap. The "restore process" consists of hopefully managing to download a big archive. It was just mind-blowing how bad the app, the code, the process and everything else was.
I should mention that un-seeded online backup is a waste of time unless you can do your initial backup from a place of business that has a very high capacity pipe to the internet; even then most of the online providers (esp. Mozy, but Carbonite as well if I remember correctly) limit your connection to around 1 mbit, which means a 500GB backup is going to take a very, very, very long time.
Crashplan charges a hefty fee for this, but I expect over the next few years for this to become a standard part of the package since the initial backup is a major, major problem for all players, without exception.
posted by rr at 12:50 PM on October 14, 2009 [1 favorite]
I did not care for their SW (or, honestly, their astroturfing on Amazon).
One of the appeals of crashplan is that it can be applied to both local and online backups. This removes one variable...
I did not mention, but I should, that I also back up my laptop to my wife's external crashplan volumes as well as a volume of my own. The ability to do machine-to-machine backups over the LAN at home is quite useful; eventually my intent is to no longer host the external drives on her machine directly at all and instead put a Mac Mini somewhere which can connect to them.
Just to add a bit of justification to my original post -- Mozy was an unbelievable piece of crap. The "restore process" consists of hopefully managing to download a big archive. It was just mind-blowing how bad the app, the code, the process and everything else was.
I should mention that un-seeded online backup is a waste of time unless you can do your initial backup from a place of business that has a very high capacity pipe to the internet; even then most of the online providers (esp. Mozy, but Carbonite as well if I remember correctly) limit your connection to around 1 mbit, which means a 500GB backup is going to take a very, very, very long time.
Crashplan charges a hefty fee for this, but I expect over the next few years for this to become a standard part of the package since the initial backup is a major, major problem for all players, without exception.
posted by rr at 12:50 PM on October 14, 2009 [1 favorite]
Response by poster: Just to follow up, my backup system has progressed very well, but is still evolving. I tried the external SATA Raid box, and it was rather buggy (beta drivers in Snow Leopard) - So I went back to just using the internal bays on the Mac Pro.
System drive 500 GB - OS and programs only
Working drive 1 TB - 2 drives mirrored as RAID 1 to protect against drive failure
Time Machine drive 1 TB - My initial "just in case" backup
I'm backing up my 1 TB working drive to an external drive every day or two, trying to keep it offsite as much as possible.
Future plans - As data from the Working drive is no longer in use (completed projects), I will move them to duplicate hard drives, one on and one off site. I'd also like to develop a weekly backup of my system drive - no important data on it, but it would be nice to have a quick restart if there were problems there. I'm also looking to tape backup as an eventual long term archive, but considering the additional $1000 - $1500 for the drive alone, that will have to wait for the moment.
posted by shinynewnick at 8:05 AM on November 16, 2009
System drive 500 GB - OS and programs only
Working drive 1 TB - 2 drives mirrored as RAID 1 to protect against drive failure
Time Machine drive 1 TB - My initial "just in case" backup
I'm backing up my 1 TB working drive to an external drive every day or two, trying to keep it offsite as much as possible.
Future plans - As data from the Working drive is no longer in use (completed projects), I will move them to duplicate hard drives, one on and one off site. I'd also like to develop a weekly backup of my system drive - no important data on it, but it would be nice to have a quick restart if there were problems there. I'm also looking to tape backup as an eventual long term archive, but considering the additional $1000 - $1500 for the drive alone, that will have to wait for the moment.
posted by shinynewnick at 8:05 AM on November 16, 2009
Carbonite is great if you have less than 200GB to upload.
I am a Carbonite subscriber, and have about 3TB to upload. The first ~200GB uploaded fairly quickly (a few GB/day), but Carbonite severely throttles users who have more than 200GB. Instead of multiple GB/day, I get maybe 100 MB/day. I wish I knew this before signing up, as I will not be renewing when my year is up. For your information.
I looked at Drobo for my Mac Pro as well, but the write times are a bit slow, and I am unwilling to lock myself into their proprietary format for storing data.
The Mercury Elite-AL Pro Qx2 is available without drives for $330, and has hardware 0, 1, 5, and 10. I am thinking of getting this box, I will post back here with thoughts.
posted by santaliqueur at 12:39 PM on November 27, 2009
I am a Carbonite subscriber, and have about 3TB to upload. The first ~200GB uploaded fairly quickly (a few GB/day), but Carbonite severely throttles users who have more than 200GB. Instead of multiple GB/day, I get maybe 100 MB/day. I wish I knew this before signing up, as I will not be renewing when my year is up. For your information.
I looked at Drobo for my Mac Pro as well, but the write times are a bit slow, and I am unwilling to lock myself into their proprietary format for storing data.
The Mercury Elite-AL Pro Qx2 is available without drives for $330, and has hardware 0, 1, 5, and 10. I am thinking of getting this box, I will post back here with thoughts.
posted by santaliqueur at 12:39 PM on November 27, 2009
This thread is closed to new comments.
- Mac Pro has 4 internal drive bays.
- Bay 1 & 2: Stripe of 250GB HDs. Gives 500GB for OS, apps, etc.
- Bay 2 & 4: Stripe of 1TB or larger HDs. Gives 2TB - 3TB of raw storage. Used as a media location for Final Cut Pro.
- Attach your choice of RAID 5 drive arrays for backup. Drobo is okay but has mixed reviews. I like G-Tech and CalDigital. Avoid Lacie until they can get their collective heads out of their rears and fix the longstanding problems with their power supplies.
- Time Machine can easily back up your boot volume. Your media storage volume can be backed up in lots of ways (SuperDuper, CCC, CrashPlan, etc.).
I've set up a few of these and the internal bus is absolutely fast enough for editing uncompressed HD footage. You should probably have a lot of RAM (4GB or more), but that almost goes without saying.
posted by mrbarrett.com at 2:24 PM on October 13, 2009