Digital media backup and storage.
April 27, 2012 8:09 AM Subscribe
You generate a lot of video/audio/etc. You're a professional, but not a network or tv station, so you don't have millions of dollars to spend on media archiving and backup. What are your digital media backup and storage strategies?
I work for a theater company that generates a lot of video. We have a daily video blog in HD, we document most performances in HD, and we document about 5 hours a day of rehearsals in SD.
I don't really have a handle on exactly how much data we generate, different people are responsible for different things, which is part of the problem.
I'm researching storage and backup strategies. We've looked at things like this, and it seems like they're too expensive/too technical for us. (We're essentially a group of artists, we can't really afford to hire a full time IT person).
My current thinking is that we should get some kind of RAID array, and attach it to our network somehow so that people can dump backup copies of whatever media they're generating on their desktop/laptops onto a secure (?) thing we have physically on-site. But then what about off-site data storage? And what do we do when it gets full, buy another one?
Right now we're storing all of our footage on external hard drives with firewire 800/400. I'm trying to get a handle on where we should go from here to improve our strategies, but I'm not even sure where to start. If you know any decent guides online, I'd be happy to read them. I'd also be totally happy with specific suggestions like, "Hey buy this and use it."
I would also be happy to receive suggestions about *people* who can help with this kind of thing, if there's such a thing as a person who would come in and help analyze our situation and give advice. I don't know if these people exist.
Oh, also, we're mac based. Mostly Mac Pro towers, but a few laptops too.
Thanks so much.
I work for a theater company that generates a lot of video. We have a daily video blog in HD, we document most performances in HD, and we document about 5 hours a day of rehearsals in SD.
I don't really have a handle on exactly how much data we generate, different people are responsible for different things, which is part of the problem.
I'm researching storage and backup strategies. We've looked at things like this, and it seems like they're too expensive/too technical for us. (We're essentially a group of artists, we can't really afford to hire a full time IT person).
My current thinking is that we should get some kind of RAID array, and attach it to our network somehow so that people can dump backup copies of whatever media they're generating on their desktop/laptops onto a secure (?) thing we have physically on-site. But then what about off-site data storage? And what do we do when it gets full, buy another one?
Right now we're storing all of our footage on external hard drives with firewire 800/400. I'm trying to get a handle on where we should go from here to improve our strategies, but I'm not even sure where to start. If you know any decent guides online, I'd be happy to read them. I'd also be totally happy with specific suggestions like, "Hey buy this and use it."
I would also be happy to receive suggestions about *people* who can help with this kind of thing, if there's such a thing as a person who would come in and help analyze our situation and give advice. I don't know if these people exist.
Oh, also, we're mac based. Mostly Mac Pro towers, but a few laptops too.
Thanks so much.
The first question, really, is the one you already know you need to answer - how much data are we talking about? You mention an HD daily video blog and another 5 hours in SD. Do you know what codec you use? The codec will drastically affect the data rate and the higher the data rate, the larger the space you'll need, and the quicker you'll need more, and so on.
The second question is are you looking to use it as shared storage (so that multiple editors can work with video at the same time and share bins/projects), or is just saving video into an on-site backup computer sufficient?
The shared storage vendors (Avid ISIS, et al) have the performance and features that make storing video on a server feasible and the ability to share bins is a boon to productivity if you need it, but it does come at a premium. As far as needing an IT guy, I've used ISIS (and have worked for a competitor), and I'd say that after some initial training, the day-to-day operation of the system doesn't need a dedicated IT guy to handle. For more complicated issues, well, that's what tech support is there for.
If you don't need the shared storage aspect and are just looking for somewhere to put your video, then owls' got you covered with the Drobo recommendation, but again, they're not fast enough to edit from.
As far as backup goes, the NBCs of the world still use tape, except that now it's digital video on an LTO tape, with computer files instead of "analog" tape, and yeah, when you run out, you spend $50 for a new tape. The initial outlay is quite a bit more expensive, but the tapes themselves are cheaper than disks. It's also possible that LTO tape is completely overkill for what you need.
There are firms that specialize in setting up video editing suites that would be able to assist you locally to help you figure out your needs.
posted by fragmede at 9:43 AM on April 27, 2012
The second question is are you looking to use it as shared storage (so that multiple editors can work with video at the same time and share bins/projects), or is just saving video into an on-site backup computer sufficient?
The shared storage vendors (Avid ISIS, et al) have the performance and features that make storing video on a server feasible and the ability to share bins is a boon to productivity if you need it, but it does come at a premium. As far as needing an IT guy, I've used ISIS (and have worked for a competitor), and I'd say that after some initial training, the day-to-day operation of the system doesn't need a dedicated IT guy to handle. For more complicated issues, well, that's what tech support is there for.
If you don't need the shared storage aspect and are just looking for somewhere to put your video, then owls' got you covered with the Drobo recommendation, but again, they're not fast enough to edit from.
As far as backup goes, the NBCs of the world still use tape, except that now it's digital video on an LTO tape, with computer files instead of "analog" tape, and yeah, when you run out, you spend $50 for a new tape. The initial outlay is quite a bit more expensive, but the tapes themselves are cheaper than disks. It's also possible that LTO tape is completely overkill for what you need.
There are firms that specialize in setting up video editing suites that would be able to assist you locally to help you figure out your needs.
posted by fragmede at 9:43 AM on April 27, 2012
Response by poster: Thanks, this is really helpful.
One question about the Drobo, that I couldn't figure out from the website: once the drives are full, could I just buy new drives and replace the ones in the Drobo, or would I buy a whole new Drobo?
Maybe the answer to that is so obvious that nobody thought to put it on the website.
posted by hapticactionnetwork at 1:23 PM on April 27, 2012
One question about the Drobo, that I couldn't figure out from the website: once the drives are full, could I just buy new drives and replace the ones in the Drobo, or would I buy a whole new Drobo?
Maybe the answer to that is so obvious that nobody thought to put it on the website.
posted by hapticactionnetwork at 1:23 PM on April 27, 2012
I think you're on the right track with RAID + some kind of offsite backup. (A RAID is not a backup...something I always "knew" but still had to learn the hard way.)
For the offsite backup, a hard drive dock like this might make sense:
NewerTech Voyager Q
Just remember that those hard drives probably won't last more than a few years. LTO tape is a much more reliable offsite solution, but substantially more expensive.
posted by mr frosted at 2:02 PM on April 27, 2012
For the offsite backup, a hard drive dock like this might make sense:
NewerTech Voyager Q
Just remember that those hard drives probably won't last more than a few years. LTO tape is a much more reliable offsite solution, but substantially more expensive.
posted by mr frosted at 2:02 PM on April 27, 2012
Regarding Drobo again - you start with at least two drives of any size, which will give you some amount of storage depending on how large the drives are. You can figure out how much with their capacity calculator.
When you've filled that space, or whenever your budget allows you to buy a new drive you just pop it into the Drobo. It will do it's thing for awhile and you'll suddenly have more space available.
Eventually your machine will be full of drives. When you fill all of that space (the Drobo will warn you when you're close) you can pull one old drive at a time and replace it with a new drive. The Drobo will refill the new drive with what was on the old drive, give you additional space, and off you go.
At a certain point you may get to a Drobo full of the largest disks that it supports (3 TB currently, I think). If you fill that then it's time for another storage solution.
I like the idea of archival tape that folks have been mentioning. That would be a better approach for older material that you don't need active access to at a moment's notice.
Data storage gets hard when you start worrying about backups and redundancy - good luck.
posted by owls at 2:29 PM on April 27, 2012
When you've filled that space, or whenever your budget allows you to buy a new drive you just pop it into the Drobo. It will do it's thing for awhile and you'll suddenly have more space available.
Eventually your machine will be full of drives. When you fill all of that space (the Drobo will warn you when you're close) you can pull one old drive at a time and replace it with a new drive. The Drobo will refill the new drive with what was on the old drive, give you additional space, and off you go.
At a certain point you may get to a Drobo full of the largest disks that it supports (3 TB currently, I think). If you fill that then it's time for another storage solution.
I like the idea of archival tape that folks have been mentioning. That would be a better approach for older material that you don't need active access to at a moment's notice.
Data storage gets hard when you start worrying about backups and redundancy - good luck.
posted by owls at 2:29 PM on April 27, 2012
This thread is closed to new comments.
My main complaint is that it's not particularly fast, even over gigabit LAN. It's good enough for storage, but you wouldn't want to work from it as a primary drive. It's a comparatively expensive solution as compared to other NAS boxes too - you'll want to weigh convenience against price as you make your decision.
Whatever you do, you'll want to step away from using individual external drives if data preservation is your aim. Hard drives will fail. It's just a matter of time. If you don't have a backup plan you will lose your data.
Something like this will handle your local backup. If you're thinking offsite backup you have other things to consider.
You probably need to get a handle on how much storage you're talking about too. Daily HD video will begin to add up as time passes. You'll need to size your storage plan accordingly or you might outgrow your available space.
posted by owls at 9:19 AM on April 27, 2012