Video files are taking over my computer!
September 20, 2011 5:31 PM
Massive video camera files...where to put them?
As a new father I'm videotaping my young son an inordinate amount, and I upload the HD video from the camera to my iMac. I use iMovie and sometimes Final Cut Pro to edit the films.
I want to keep the video files--all of them if I can--but HD video takes up a massive amount of hard drive space. I have a 320 Gb hard drive on my iMac and a 1TB external drive. Both are becoming more and more full with my video files.
Do I compress them? I've tried this with the native Mac OSX compressor, but this shaves off just about 5 or 10 percent off the size--not really enough. Should I just keep buying more and more external drives?
As a new father I'm videotaping my young son an inordinate amount, and I upload the HD video from the camera to my iMac. I use iMovie and sometimes Final Cut Pro to edit the films.
I want to keep the video files--all of them if I can--but HD video takes up a massive amount of hard drive space. I have a 320 Gb hard drive on my iMac and a 1TB external drive. Both are becoming more and more full with my video files.
Do I compress them? I've tried this with the native Mac OSX compressor, but this shaves off just about 5 or 10 percent off the size--not really enough. Should I just keep buying more and more external drives?
The bottom line is that HD video footage takes up a lot of space.
I don't know what camera you're using, but the footage that comes out is probably H264 codec, which is already a pretty good compression. You're not going to do much better without downgrading the quality. You might get some gain by compressing the audio, but not a huge amount.
If you really want to keep *all of it*, you need either a) more and more drives, which will eventually fail b) burn it on discs or c) upload to somewhere like Amazon S3. (This would be extraordinarily time-consuming, thought they do offer a service where you mail them a drive).
When I'm doing a movie, I edit it down to say 15 minutes from hours and hours of footage. Eventually I delete the bad takes and unused footage. Look, I'm not here to tell you to delete footage of your kid. But consider if you will really go back and watch *all* of it- because it sounds like you are accumulating more than will be watchable in a lifetime. Could you possibly edit it down to highlights?
posted by drjimmy11 at 5:41 PM on September 20, 2011
I don't know what camera you're using, but the footage that comes out is probably H264 codec, which is already a pretty good compression. You're not going to do much better without downgrading the quality. You might get some gain by compressing the audio, but not a huge amount.
If you really want to keep *all of it*, you need either a) more and more drives, which will eventually fail b) burn it on discs or c) upload to somewhere like Amazon S3. (This would be extraordinarily time-consuming, thought they do offer a service where you mail them a drive).
When I'm doing a movie, I edit it down to say 15 minutes from hours and hours of footage. Eventually I delete the bad takes and unused footage. Look, I'm not here to tell you to delete footage of your kid. But consider if you will really go back and watch *all* of it- because it sounds like you are accumulating more than will be watchable in a lifetime. Could you possibly edit it down to highlights?
posted by drjimmy11 at 5:41 PM on September 20, 2011
(If you do go the compression route, Mac's Compressor is notoriously poor. I hate to sound like a broken record with this but seriously, google "MPEG Streamclip." It's free and awesome.)
posted by drjimmy11 at 5:45 PM on September 20, 2011
posted by drjimmy11 at 5:45 PM on September 20, 2011
My settings when I recently was uploading audition footage I didn't care about:
Reduced size from 1080p to 720.
Reduced quality to 50%.
Compressed audio (forget which I chose, they're all similar more or less)
This shrunk the files by a very significant amount, to maybe 20% of their original size. They were smaller and looked worse of course, but you could still clearly see and hear what was happening. That's compression.
posted by drjimmy11 at 5:48 PM on September 20, 2011
Reduced size from 1080p to 720.
Reduced quality to 50%.
Compressed audio (forget which I chose, they're all similar more or less)
This shrunk the files by a very significant amount, to maybe 20% of their original size. They were smaller and looked worse of course, but you could still clearly see and hear what was happening. That's compression.
posted by drjimmy11 at 5:48 PM on September 20, 2011
It's valuable footage. You have to back it up. That means at least two copies. Harddrives are cheap and getting cheaper. Raw harddrives in a esata/usb/firewire docking station will probably give you the best bang for buck for back ups. You can make all sorts of attempts to cut down the amount of footage you're backing up but if it's of your kid I'd keep it all.
Also check your render files from FCP, they can eat up lots of space.
posted by jade east at 5:49 PM on September 20, 2011
Also check your render files from FCP, they can eat up lots of space.
posted by jade east at 5:49 PM on September 20, 2011
Hard drive space is cheap. Memories are not. Back up everything at least twice. Back things up to the cloud. Aim for redundancy several times over.
posted by Inspector.Gadget at 6:14 PM on September 20, 2011
posted by Inspector.Gadget at 6:14 PM on September 20, 2011
There's no reason to limit yourself to the HD space you currently have. External USB HD's are amazingly cheap now. (A 1TB USB2 external HD is like $90 now.)
posted by Chocolate Pickle at 6:20 PM on September 20, 2011
posted by Chocolate Pickle at 6:20 PM on September 20, 2011
Since these are videos of your son, you're really looking for an archival storage solution. And I'll second that backed-up hard drives are currently the best choice for archival storage. CD-Rs degrade in under a decade, and for all we know writable Blu-ray discs will go even sooner. Your only other real choice is tape backup, and that's a real pain in the butt. Buy multiple hard drives, and save a cold backup of all footage, preferably in a second location. Copy to fresh hard drives every few years.
As Chocolate Pickle says, 1TB external hard drives are less than 100 bucks a pop, so if you do complete backup copies and store your video at 40mb/s (rougly Blu-ray bitrates) we're talking $14 or so per hour of video. To beat that, you have to trade off either video quantity, video quality, or storage reliability. Your choice.
FWIW, my choice would probably be to store the video at fairly-compressed 720p; I've never been huge on HD myself. DVD-quality is good enough for me, and that's a tenth the bitrate, so we're down to $1.40 per hour.
posted by goingonit at 6:35 PM on September 20, 2011
As Chocolate Pickle says, 1TB external hard drives are less than 100 bucks a pop, so if you do complete backup copies and store your video at 40mb/s (rougly Blu-ray bitrates) we're talking $14 or so per hour of video. To beat that, you have to trade off either video quantity, video quality, or storage reliability. Your choice.
FWIW, my choice would probably be to store the video at fairly-compressed 720p; I've never been huge on HD myself. DVD-quality is good enough for me, and that's a tenth the bitrate, so we're down to $1.40 per hour.
posted by goingonit at 6:35 PM on September 20, 2011
A 1TB USB2 external HD is like $90 now.
Chocolate Pickle is still paying too much! Not a week goes by that I don't see a sale for a 2TB external HD for less than that.
That being said, back up to the cloud. Online backups of essentially limitless size (provided the hard drives are connected to your computer at least once a month, long enough to back up) are available for about $100-200/year. No hard drive is as secure as online backups, IMO.
posted by IAmBroom at 6:42 PM on September 20, 2011
Chocolate Pickle is still paying too much! Not a week goes by that I don't see a sale for a 2TB external HD for less than that.
That being said, back up to the cloud. Online backups of essentially limitless size (provided the hard drives are connected to your computer at least once a month, long enough to back up) are available for about $100-200/year. No hard drive is as secure as online backups, IMO.
posted by IAmBroom at 6:42 PM on September 20, 2011
I don't know, IAmBroom, I'm all in favor of online backups, but when you hit around 1TB, things start to get more dicey. Assuming zardoz gets ~1mbps upstream, 1TB will take a 10 million seconds -- about a third of a year (!) -- to upload.
posted by goingonit at 6:50 PM on September 20, 2011
posted by goingonit at 6:50 PM on September 20, 2011
Sounds like you'd be an ideal candidate for a Drobo.
posted by sockpup at 7:15 PM on September 20, 2011
posted by sockpup at 7:15 PM on September 20, 2011
I'm an editor, so this is my job day in and day out. Here's my recommendation, depending on your budget:
Buy an external dock, three options-
1. Firewire / USB 2.0 & 3.0 / ESATA for $73 - the Firewire connection will be your fastest option on an iMac.
2. USB 2.0 for $32 - not as fast, but half the price. The lower speed might be an issue if you are trying to edit full HD directly off the drive, but I have done it before.
3. USB 2.0 Dual Dock for $48 - This is probably what I would go with, personally, as it will allow you to copy files directly between two bare drives. Perfect for backups! Speed is still a potential issue, but if that's the case I would just edit off of your other external drive and use the dock for raw footage only.
Buy two bare hard drives:
1. I really like the WD Caviar Green series of drives. While not as fast as the Caviar Black, the prices are great and I've had strong reliability over more than 20 drives (one failure, and they replaced it under warranty without any hassle). The sweet spot right now seems to be the $79 2 TB size.
My recommendation on saving footage - backup the camera files as-is directly onto this external drive, and keep the second drive as a mirrored backup. The advantage here is that if you want to do anything with the footage later, you have kept the original files and can do whatever you need with them. The reason your compression doesn't gain much is because the files right off the camera are already highly compressed (likely some variant of the H.264 or AVCHD format). To give you a general idea, I am the director for a weekly national music show where we do a four-camera shoot. I can get two episodes, four cameras worth of raw footage (approximately 7-8 hours of raw) in under 100 GB. That means I can fit 20 episodes worth of raw footage on a single 1 TB drive. If you're shooting more than that - more power to you, man! Let me know if you have any questions.
Final thought: Do not use Blu-Ray or DVD as an archival format. They will not last over any extended period of time (neither will hard drives, but keep backups and rotate them).
posted by shinynewnick at 8:27 PM on September 20, 2011
Buy an external dock, three options-
1. Firewire / USB 2.0 & 3.0 / ESATA for $73 - the Firewire connection will be your fastest option on an iMac.
2. USB 2.0 for $32 - not as fast, but half the price. The lower speed might be an issue if you are trying to edit full HD directly off the drive, but I have done it before.
3. USB 2.0 Dual Dock for $48 - This is probably what I would go with, personally, as it will allow you to copy files directly between two bare drives. Perfect for backups! Speed is still a potential issue, but if that's the case I would just edit off of your other external drive and use the dock for raw footage only.
Buy two bare hard drives:
1. I really like the WD Caviar Green series of drives. While not as fast as the Caviar Black, the prices are great and I've had strong reliability over more than 20 drives (one failure, and they replaced it under warranty without any hassle). The sweet spot right now seems to be the $79 2 TB size.
My recommendation on saving footage - backup the camera files as-is directly onto this external drive, and keep the second drive as a mirrored backup. The advantage here is that if you want to do anything with the footage later, you have kept the original files and can do whatever you need with them. The reason your compression doesn't gain much is because the files right off the camera are already highly compressed (likely some variant of the H.264 or AVCHD format). To give you a general idea, I am the director for a weekly national music show where we do a four-camera shoot. I can get two episodes, four cameras worth of raw footage (approximately 7-8 hours of raw) in under 100 GB. That means I can fit 20 episodes worth of raw footage on a single 1 TB drive. If you're shooting more than that - more power to you, man! Let me know if you have any questions.
Final thought: Do not use Blu-Ray or DVD as an archival format. They will not last over any extended period of time (neither will hard drives, but keep backups and rotate them).
posted by shinynewnick at 8:27 PM on September 20, 2011
shinynewnick, that is a superb answer. I am an editor, too and I didn't even know about those docks.
posted by bz at 10:04 PM on September 20, 2011
posted by bz at 10:04 PM on September 20, 2011
Thanks, everyone, all of you have given helpful answers. shinynewnick, if your show can shoot 7-8 hours in under 100 gigs, then I need to do what you're doing! I have a Sony HD camcorder that shoots in AVCHD format (I think), and when I import that through iMovie, it saves onto the computer as a .mov file. Some seem small, some are amazingly huge. For example, one .mov file is about 12 minutes 30 seconds, the resolution is 1280 x 720, and the size is 3.52 Gigs. Gigs! What in the world am I doing wrong?
posted by zardoz at 12:34 AM on September 21, 2011
posted by zardoz at 12:34 AM on September 21, 2011
When you're buying bare drives in pairs to plug into your dock, use two different brands for the members of any given pair. This guards against data loss due to drive firmware faults that affect whole batches, like this Seagate SD15 firmware fault from a couple of years ago.
Seagate's reputation took a pretty massive hit from the SD15 fault; I know lots of people who reacted to being burned by that who simply swore off Seagate drives for life. I think that reaction is basically superstitious nonsense. Firmware faults, like any other kind of software bug, are a fact of life; I can think of no particular reason why any other manufacturer is less likely than Seagate to suffer a similar problem eventually. But the chances of two manufacturers releasing buggy drive firmware at the same time are very slim (unless they're actually the same manufacturer, as was the case with Seagate and Maxtor at the time of the SD15 fault).
posted by flabdablet at 3:37 AM on September 21, 2011
Seagate's reputation took a pretty massive hit from the SD15 fault; I know lots of people who reacted to being burned by that who simply swore off Seagate drives for life. I think that reaction is basically superstitious nonsense. Firmware faults, like any other kind of software bug, are a fact of life; I can think of no particular reason why any other manufacturer is less likely than Seagate to suffer a similar problem eventually. But the chances of two manufacturers releasing buggy drive firmware at the same time are very slim (unless they're actually the same manufacturer, as was the case with Seagate and Maxtor at the time of the SD15 fault).
posted by flabdablet at 3:37 AM on September 21, 2011
Seconding the "bare drive(s) + dock" plan and adding Disk Tracker to catalog those drives. An HD that is not spinning is pretty safe.
posted by omnidrew at 7:59 AM on September 21, 2011
posted by omnidrew at 7:59 AM on September 21, 2011
Ah, there is the rub I think. I haven't used iMovie very much (we are a FCP shop that is slowly transitioning to Adobe Premiere after the FCP X tragedy), but I believe iMovie transcodes files to the Apple Intermediate Codec (AIC). This is the non-pro version of ProRes; it allows you to edit HD movies without a lot of computing power, but the trade-off is file size as you have seen. There is one shooter/editor I deal with on occasion, and when I have to bring in his projects from Final Cut Express they are 3-4x the size of a comparable project on my system because of the AIC format. ProRes is better in the quality for file size trade-off, but same sort of issue.
What model Sony camera do you have? My recommendation is to find a way to grab the raw files off of the camera memory before you bring it in to iMovie. Here's the workflow I would do:
1. Plug in camera or memory card
2. Copy those files in Finder to one of the external drives. Check against the original files to confirm a successful transfer
3. Then you can import the files into iMovie from the external drive. You will still have those large AIC files while you're working on a project, but they can be deleted when you're finished and you will still have all the original footage.
posted by shinynewnick at 8:05 AM on September 21, 2011
What model Sony camera do you have? My recommendation is to find a way to grab the raw files off of the camera memory before you bring it in to iMovie. Here's the workflow I would do:
1. Plug in camera or memory card
2. Copy those files in Finder to one of the external drives. Check against the original files to confirm a successful transfer
3. Then you can import the files into iMovie from the external drive. You will still have those large AIC files while you're working on a project, but they can be deleted when you're finished and you will still have all the original footage.
posted by shinynewnick at 8:05 AM on September 21, 2011
Thanks for that. So iMovie is the culprit! Damn you IMovie! The camcorder is a Sony HDR-SR11. It shoots in AVCHD format, so importing it is the one big pain of this camera. Otherwise I like it a lot. I do have another program that will import the files, I guess I'll try that instead of iMovie. And give Premiere a shot; it seems marginally more user friendly than FCP. Though they're both more than I really need.
Anyway, thanks all!
posted by zardoz at 4:54 PM on September 21, 2011
Anyway, thanks all!
posted by zardoz at 4:54 PM on September 21, 2011
In theory, Premiere may (or may not) be able to work with the AVCHD files without transcoding. Depends on the file format and your computer. I will say it has been a relative joy to bring in both native Canon 5D (H.264) and Sony EX3 (MPEG2) files without a trip through Compressor.
Transcoding itself isn't a problem, those just aren't the files you want to keep long term. No matter which program you end up using, you still want to bring the raw footage files onto your external drive for future use.
posted by shinynewnick at 6:49 PM on September 21, 2011
Transcoding itself isn't a problem, those just aren't the files you want to keep long term. No matter which program you end up using, you still want to bring the raw footage files onto your external drive for future use.
posted by shinynewnick at 6:49 PM on September 21, 2011
I don't know, IAmBroom, I'm all in favor of online backups, but when you hit around 1TB, things start to get more dicey. Assuming zardoz gets ~1mbps upstream, 1TB will take a 10 million seconds -- about a third of a year (!) -- to upload.
Hrrrm. Point taken. Everything gets harder at these sizes.
posted by IAmBroom at 2:20 PM on September 22, 2011
Hrrrm. Point taken. Everything gets harder at these sizes.
posted by IAmBroom at 2:20 PM on September 22, 2011
« Older In which a Media Mail package gets some sort of... | MacBook Air. Now with a drive no one else... Newer »
This thread is closed to new comments.
posted by EndsOfInvention at 5:35 PM on September 20, 2011