How to compile and compress video for internet distribution
October 7, 2005 10:55 PM Subscribe
I want to make a compilation of clips from a series of (i think Divx-encoded) AVI files -- basically, a 15 to 20 minute "best of" montage taken from about 10 hours of video. The native resolution of the files is something around 500px by 250 or so -- 16:9 ratio -- 1 hour = ~300MBs.
I use a Mac, and I've got access to Final Cut Pro HD and iMovie -- I'm more familiar with iMovie, but I've got a good enough grasp of other pro apps that I'm sure I can make my way through Final Cut for what I need (just cuts, joins, and crossfades).
Where my knowledge falls short, however, is video compression. Every time I've tried something like this in the past, I always end up with a video full of compression artifacts and WAY larger in file size than it should be. I don't want to lose any quality in these files -- I want to import them, cut them up, and save them in the same or similar format. What compression method(s) should I use? Encoding Divx costs money, right? (bleh, unless its the best option by far). H.264 is too new for my purposes -- I want this to be accessible on as many machines as possible. Finally, any tips as far as Final Cut or iMovie are concerned? iMovie is terribly slow to render, but I imagine Final Cut will be relatively slow for me too (iMac G5, 1.6MHz). My gripe w/ iMovie is that I've never been able to find a compression method that didn't make the video look like crap. Suggestions/ideas?
I use a Mac, and I've got access to Final Cut Pro HD and iMovie -- I'm more familiar with iMovie, but I've got a good enough grasp of other pro apps that I'm sure I can make my way through Final Cut for what I need (just cuts, joins, and crossfades).
Where my knowledge falls short, however, is video compression. Every time I've tried something like this in the past, I always end up with a video full of compression artifacts and WAY larger in file size than it should be. I don't want to lose any quality in these files -- I want to import them, cut them up, and save them in the same or similar format. What compression method(s) should I use? Encoding Divx costs money, right? (bleh, unless its the best option by far). H.264 is too new for my purposes -- I want this to be accessible on as many machines as possible. Finally, any tips as far as Final Cut or iMovie are concerned? iMovie is terribly slow to render, but I imagine Final Cut will be relatively slow for me too (iMac G5, 1.6MHz). My gripe w/ iMovie is that I've never been able to find a compression method that didn't make the video look like crap. Suggestions/ideas?
DivX only costs money if you use the divx.com codec. There are lots of others, pretty much covered by joeblough there.
If you don't have any scene transitions, then you might be able to get away without any transcoding and resultant loss of quality. If you were using a PC I'd say to use VirtualDub for this, but I have no idea about OS X. However if you want fades between scenes, then that will require synthesizing new frames and you'll have to re-encode. But if the starting and final bitrates are approximately the same, you shouldn't notice too many added artifacts.
posted by Rhomboid at 1:37 AM on October 8, 2005
If you don't have any scene transitions, then you might be able to get away without any transcoding and resultant loss of quality. If you were using a PC I'd say to use VirtualDub for this, but I have no idea about OS X. However if you want fades between scenes, then that will require synthesizing new frames and you'll have to re-encode. But if the starting and final bitrates are approximately the same, you shouldn't notice too many added artifacts.
posted by Rhomboid at 1:37 AM on October 8, 2005
The problem is...you've already screwed - the quality with the bizzarro AVI files (they're possibly divx). GIGO - Garbage In/Garbage Out.
Where are the original videos?. That's what you want/need/need to learn to edit from, not distribution copies that have had most of the information thrown out and aren't mean't for editing.
Look, I get it. Somehow you ended up with a bunch of clips that you want to put together for others.
Divx, MPEG1, MPEG2, WMV are all Temporal compresssion codecs.
They all have their own unique JPEG flavor to throw out gobs of video quality to make the file smaller...but bonus:
Every frame does not have all the information. The use an "I" frame (Intra frame) that contains all the information...and the a series of frames that describe only the pixels that have changed.
You can "technically" only edit on an I frame.
This is why that trying to edit these distribution copies should not be done.
Why do you get such artifacts when you convert them to an I frame codec? Well, they were pretty lossy to begin with, and then you're converting them (often) to another lossy format, and bonus - you may be blowing up the picture.
Example:
An AVI (divx) encoded with two hours of infomation, crushed to 700mb. You decide you want to edit it. Well you blow up the 500 x 300.
Let's decide to go to DV. DV is a common, inexpensive I frame codec. It converts well later for DVD and requires minimal procession power to edit.
The original Video - 16x9 or 4x3 are the same amount of pixels in DV - 720x480. So those pixels are being uprezzed...but it's not a 2:1 increase, but some bizarro adjustment (a 100% size increase works better than a 37.5%. I'm pulling these numbers out of my ass...It's just the idea of a less than optimum scaler)
Now, DV is a wavlet I-frame compression...running at 5:1 (that is, around 5 min : gig.) So it'll be 2 hours? That's 24 gigs.
Oh, wait, a different compression. iMovie, FCP, (hell, everything) is geared towards video meant for the screen or DVD, meant for the screen.
The "other" compression types you're familiar with (divx, h264, mpeg 2) were never meant to be edited.
joeblough writes "divx/mpeg4 is pretty much state of the art and is used almost universally on the 'net. only h.264 is more advanced than mpeg4 at this point, and you're right, its just too new. divx will probably get you better looking video at lower bitrates than the lavc codec, but its probably not worth the money."
Divx is okay. It's used on the net primarily for piracy - porn and stolen footage. I know of no video based client's website that's not WMV, MPEG, Real, or QT sorenson. h.264 is part of the MPEG-4 specification, and DivX is not Divx is a propritary format that is based on MPEG4, but not part of MPEG4 spec. Yes, I even own a phillips 642 DVD player for 'handling" video that may have come from the net on my TV.
But let's say you bought divx. It's < $50. you're wililng to spend time going over 10 hours of material, but not willing to shell out $50?br>
I encounter this all the time, btw.
Use the original video for editing, in an I frame format (HDV is an exception.)
When finished:
Encode to a distribution format.
Editing from a distribution format requires a transcode to an editing format, and will always look compromised.
posted by filmgeek at 8:35 AM on October 8, 2005
Where are the original videos?. That's what you want/need/need to learn to edit from, not distribution copies that have had most of the information thrown out and aren't mean't for editing.
Look, I get it. Somehow you ended up with a bunch of clips that you want to put together for others.
Divx, MPEG1, MPEG2, WMV are all Temporal compresssion codecs.
They all have their own unique JPEG flavor to throw out gobs of video quality to make the file smaller...but bonus:
Every frame does not have all the information. The use an "I" frame (Intra frame) that contains all the information...and the a series of frames that describe only the pixels that have changed.
You can "technically" only edit on an I frame.
This is why that trying to edit these distribution copies should not be done.
Why do you get such artifacts when you convert them to an I frame codec? Well, they were pretty lossy to begin with, and then you're converting them (often) to another lossy format, and bonus - you may be blowing up the picture.
Example:
An AVI (divx) encoded with two hours of infomation, crushed to 700mb. You decide you want to edit it. Well you blow up the 500 x 300.
Let's decide to go to DV. DV is a common, inexpensive I frame codec. It converts well later for DVD and requires minimal procession power to edit.
The original Video - 16x9 or 4x3 are the same amount of pixels in DV - 720x480. So those pixels are being uprezzed...but it's not a 2:1 increase, but some bizarro adjustment (a 100% size increase works better than a 37.5%. I'm pulling these numbers out of my ass...It's just the idea of a less than optimum scaler)
Now, DV is a wavlet I-frame compression...running at 5:1 (that is, around 5 min : gig.) So it'll be 2 hours? That's 24 gigs.
Oh, wait, a different compression. iMovie, FCP, (hell, everything) is geared towards video meant for the screen or DVD, meant for the screen.
The "other" compression types you're familiar with (divx, h264, mpeg 2) were never meant to be edited.
joeblough writes "divx/mpeg4 is pretty much state of the art and is used almost universally on the 'net. only h.264 is more advanced than mpeg4 at this point, and you're right, its just too new. divx will probably get you better looking video at lower bitrates than the lavc codec, but its probably not worth the money."
Divx is okay. It's used on the net primarily for piracy - porn and stolen footage. I know of no video based client's website that's not WMV, MPEG, Real, or QT sorenson. h.264 is part of the MPEG-4 specification, and DivX is not Divx is a propritary format that is based on MPEG4, but not part of MPEG4 spec. Yes, I even own a phillips 642 DVD player for 'handling" video that may have come from the net on my TV.
But let's say you bought divx. It's < $50. you're wililng to spend time going over 10 hours of material, but not willing to shell out $50?br>
I encounter this all the time, btw.
Use the original video for editing, in an I frame format (HDV is an exception.)
When finished:
Encode to a distribution format.
Editing from a distribution format requires a transcode to an editing format, and will always look compromised.
posted by filmgeek at 8:35 AM on October 8, 2005
divx is a de-facto standard for encoding to mpeg4 part 2. you may not like it or have biases against it, but it is far superior to mpeg2, which is the other dominant lossy compression scheme. your clients may not use divx .avi, but almost every other human on the planet with a PC at home does. also, you should call up DirecTV to tell them that they shouldnt be using mpeg4 in their next set top boxes since it's used for piracy.
also i'm not quite sure about your assertion that divx is non-standard. i'm pretty sure it produces legal mpeg4 part 2 output. that standard has a lot of knobs and its possible for 2 codecs to produce legal output at the same bitrate that look very different quality-wise.
when i want to transcode a DVD i bought to watch on my HTPC or on my portable device (palm zodiac) i don't use QT sorensen (no stable open source codec), .wmv (no stable open source codec / x86 required to use microsoft DLLs) or real (no open source codec). someday i will use h.264 as support improves in the open source tools, as it is superior to mpeg4 part 2 at a given bitrate.
i understand that one needs I frame-based video to edit easily, but there are editors for mpeg2 and mpeg4 which decode the video around a chosen cutpoint to all I-frame so that it can be edited at arbitrary frame boundaries.
by the way, if you want to just merge all the clips together, the latest version of mencoder will take multiple source files and merge them all into the target file, albeit with no transitions or anything fancy like that.
posted by joeblough at 9:13 AM on October 8, 2005
also i'm not quite sure about your assertion that divx is non-standard. i'm pretty sure it produces legal mpeg4 part 2 output. that standard has a lot of knobs and its possible for 2 codecs to produce legal output at the same bitrate that look very different quality-wise.
when i want to transcode a DVD i bought to watch on my HTPC or on my portable device (palm zodiac) i don't use QT sorensen (no stable open source codec), .wmv (no stable open source codec / x86 required to use microsoft DLLs) or real (no open source codec). someday i will use h.264 as support improves in the open source tools, as it is superior to mpeg4 part 2 at a given bitrate.
i understand that one needs I frame-based video to edit easily, but there are editors for mpeg2 and mpeg4 which decode the video around a chosen cutpoint to all I-frame so that it can be edited at arbitrary frame boundaries.
by the way, if you want to just merge all the clips together, the latest version of mencoder will take multiple source files and merge them all into the target file, albeit with no transitions or anything fancy like that.
posted by joeblough at 9:13 AM on October 8, 2005
You can "technically" only edit on an I frame.
Yeah I have run into the same conclusion. But I can't figure out why any given frame can't be "rendered" and made an I frame. Doesn't it "have all the information" if you render it, using the surrounding frames as input data? I guess the answer would involve learning the details, and would still be the same - "no" - blargh :)
posted by scarabic at 10:38 AM on October 8, 2005
Yeah I have run into the same conclusion. But I can't figure out why any given frame can't be "rendered" and made an I frame. Doesn't it "have all the information" if you render it, using the surrounding frames as input data? I guess the answer would involve learning the details, and would still be the same - "no" - blargh :)
posted by scarabic at 10:38 AM on October 8, 2005
i dont think the answer is no... for instance i'm pretty sure womble's products allow editing mpeg2 at non GOP boundaries by re-encoding the video around the cutpoints as i described above.
posted by joeblough at 2:32 PM on October 8, 2005
posted by joeblough at 2:32 PM on October 8, 2005
joeblough writes "divx is a de-facto standard for encoding to mpeg4 part 2. "
Technically DivX is similar to Mpeg4 part 2, but not identical. Otherwise they'd be using the MPEG logo.
Now here's where things get intersting.
DivX, Inc. has been granted patents on parts of the DivX codec, which is fully MPEG-4-Advanced Simple Profile compliant.
joeblough writes "also i'm not quite sure about your assertion that divx is non-standard"
While you can create DivX items that are compliant...parts of the patents are non-compliant as well, breaking the standard. Any mpeg1 can be played back by a computer that has the mpeg1 codec. If you have the mpeg4 codec, you cannot necessarily decode all divx streams.
Again, quoting from Wikiwpedia, which is not considered a perfect fact source:
"The current version of DivX is neither free software nor open source, but an open source version of the codec—called OpenDivX®—was released by DivX in early 2001, and this version served as the basis for the open source XviD codec, the specification of which is maintained by an independent group."
Standard Codecs (such as JPEG, MPEG and their ilk) are an open consortium. DivX is privately owned and operated, therefore is proprietary, not a standard.
Getting back to editing:
Software that permits editing of non-I frame material, converts the frames to I frames on the fly, makes the edit and reconverts back to the correct temporal cadence. Actually, FCP does this for HDV (HDV is a whole bucket of stuff I don't think has anything to do with this.) There have been some MPEG-2 Editors...but it never looks the same, since the MPEG-2 original is heavily compressed.
I think Liquid Blue does this with some IMX streams.
But, It still looks...well...shitty. It's better to edit from the raw D-5 HD tape...than the letterboxed VHS. This is a difference between a raw tape format and a distribution format. Could you edit VHS and then upconvert to put on HD? Sure. But it looks like crap because like DiVX most of the information is thrown away.
filmgeek downloads & installs memcoder I totally see how memcoder would be great for the palm, Nintendo DS, converting DVDs etc....
But it seems mostly designed as a DVD recompressor, not an editor.
posted by filmgeek at 3:43 PM on October 8, 2005
Technically DivX is similar to Mpeg4 part 2, but not identical. Otherwise they'd be using the MPEG logo.
Now here's where things get intersting.
DivX, Inc. has been granted patents on parts of the DivX codec, which is fully MPEG-4-Advanced Simple Profile compliant.
joeblough writes "also i'm not quite sure about your assertion that divx is non-standard"
While you can create DivX items that are compliant...parts of the patents are non-compliant as well, breaking the standard. Any mpeg1 can be played back by a computer that has the mpeg1 codec. If you have the mpeg4 codec, you cannot necessarily decode all divx streams.
Again, quoting from Wikiwpedia, which is not considered a perfect fact source:
"The current version of DivX is neither free software nor open source, but an open source version of the codec—called OpenDivX®—was released by DivX in early 2001, and this version served as the basis for the open source XviD codec, the specification of which is maintained by an independent group."
Standard Codecs (such as JPEG, MPEG and their ilk) are an open consortium. DivX is privately owned and operated, therefore is proprietary, not a standard.
Getting back to editing:
Software that permits editing of non-I frame material, converts the frames to I frames on the fly, makes the edit and reconverts back to the correct temporal cadence. Actually, FCP does this for HDV (HDV is a whole bucket of stuff I don't think has anything to do with this.) There have been some MPEG-2 Editors...but it never looks the same, since the MPEG-2 original is heavily compressed.
I think Liquid Blue does this with some IMX streams.
But, It still looks...well...shitty. It's better to edit from the raw D-5 HD tape...than the letterboxed VHS. This is a difference between a raw tape format and a distribution format. Could you edit VHS and then upconvert to put on HD? Sure. But it looks like crap because like DiVX most of the information is thrown away.
filmgeek downloads & installs memcoder I totally see how memcoder would be great for the palm, Nintendo DS, converting DVDs etc....
But it seems mostly designed as a DVD recompressor, not an editor.
posted by filmgeek at 3:43 PM on October 8, 2005
Response by poster: not that the history of Divx isn't interesting -- but i'm still looking for advice -- i've got some Divx files, and i want to cut some parts together, crossfaded if at all possible. i've since ruled out imovie as i can't seem to exactly specify the resolution, and final cut seems bound and determined to make everything set for tv or dvd. i downloaded mencoder, but don't see how it will let me cut different files together. any suggestions?
posted by jruckman at 4:26 PM on October 8, 2005
posted by jruckman at 4:26 PM on October 8, 2005
Response by poster: modification: i am using imovie now, i've gotten over the res. issue
posted by jruckman at 6:21 PM on October 8, 2005
posted by jruckman at 6:21 PM on October 8, 2005
i think we are in violent agreement about divx. i dont actually even use it because of the proprietaryness you describe. i suspected that it could produce legal mpeg4 but when applied using defaults, it doesnt.
yeah, mencoder is not an editor. i do use it to recompress DVDs and (mostly) transcode ATSC to files that look just as good but are way, way smaller. yes there is some loss in the transcode, but frankly its not perceptible to me.
anyway.. back to the question at hand. mencoder can merge files, but you wont get any transitions or anything:
mencoder -oac copy -ovc lavc -lavcopts vcodec=mpeg4:vhq:trell:mbd=2:v4mv:autoaspect:vqscale=2:vpass=1 file1.avi file2.avi file3.avi -o /dev/null
(do pass 1 of mpeg4 encode; high quality; trellis encoding, 4 motion vectors, automatically set the ODML aspect ratio and don't bother finding optimal quantization.)
mencoder -oac copy -ovc lavc -lavcopts vcodec=mpeg4:vhq:trell:mbd=2:v4mv:autoaspect:vbitrate=1500:vpass=2 file1.avi file2.avi file3.avi -o merged_file.avi
(do pass 2 of the encoding, all of the above stuff but now limit the bitrate to 1500kbits/sec)
one might be able to replace -ovc lavc -lavcopts ... with -ovc copy, but i'm not sure that most players are going to with one file that has a bunch of different bitrates, resolutions and codecs all merged together. in fact, one should probably reencode the audio so that doesnt cause a similar problem.
before i got into all of this, i used iMovie/FCP express and DVD studio pro to make DVDs of stuff. if you managed to get your .avis into iMovie and can edit them, then you've gotten over the initial hurdle.
in the end you'll have a "by reference" .mov file (.mov container with DV inside) in the iMovie project directory that can be converted to mpeg2, sorensen or whatever with quicktime. however, if you want to go the mpeg4 route, mencoder does not understand the "by reference" .mov container, so you'll have to do a full export of the file from iMovie (in DV format) to get a flat file. i have done exactly this with videos of my kids; i didnt do any editing, but i used iMovie to capture the video from my miniDV cam, then:
DV Clip -> export audio with quicktime
.aiff -> .ac3 using A.Pack (which comes with DVDSP)
DV Clip -> progressive lavc mpeg4 (-vf lavcdeint arg added to mencoder) with mencoder, since the DV from the camera is interlaced.
muxed video and audio with transcode's "avimerge" program.
for the audio step, you could convert to mp3, or maybe leave it as raw PCM (aiff), its up to you. mencoder can convert to mp3 with one of its -oac codecs, or if you used -oac copy you'd just get whatever audio is in the DV clip, which i'm assuming would have been converted to aiff by iMovie. there are open source tools for ac3 or aac encoding... i know that the aac programs are called faac and faad, but for the life of me i cant remember now if there is an open source tool that can create ac3 audio. perhaps "sox" can do it... i cant remember. google yields some results but at least one of the programs is win32 only.
phew.
posted by joeblough at 7:21 PM on October 8, 2005
yeah, mencoder is not an editor. i do use it to recompress DVDs and (mostly) transcode ATSC to files that look just as good but are way, way smaller. yes there is some loss in the transcode, but frankly its not perceptible to me.
anyway.. back to the question at hand. mencoder can merge files, but you wont get any transitions or anything:
mencoder -oac copy -ovc lavc -lavcopts vcodec=mpeg4:vhq:trell:mbd=2:v4mv:autoaspect:vqscale=2:vpass=1 file1.avi file2.avi file3.avi -o /dev/null
(do pass 1 of mpeg4 encode; high quality; trellis encoding, 4 motion vectors, automatically set the ODML aspect ratio and don't bother finding optimal quantization.)
mencoder -oac copy -ovc lavc -lavcopts vcodec=mpeg4:vhq:trell:mbd=2:v4mv:autoaspect:vbitrate=1500:vpass=2 file1.avi file2.avi file3.avi -o merged_file.avi
(do pass 2 of the encoding, all of the above stuff but now limit the bitrate to 1500kbits/sec)
one might be able to replace -ovc lavc -lavcopts ... with -ovc copy, but i'm not sure that most players are going to with one file that has a bunch of different bitrates, resolutions and codecs all merged together. in fact, one should probably reencode the audio so that doesnt cause a similar problem.
before i got into all of this, i used iMovie/FCP express and DVD studio pro to make DVDs of stuff. if you managed to get your .avis into iMovie and can edit them, then you've gotten over the initial hurdle.
in the end you'll have a "by reference" .mov file (.mov container with DV inside) in the iMovie project directory that can be converted to mpeg2, sorensen or whatever with quicktime. however, if you want to go the mpeg4 route, mencoder does not understand the "by reference" .mov container, so you'll have to do a full export of the file from iMovie (in DV format) to get a flat file. i have done exactly this with videos of my kids; i didnt do any editing, but i used iMovie to capture the video from my miniDV cam, then:
DV Clip -> export audio with quicktime
.aiff -> .ac3 using A.Pack (which comes with DVDSP)
DV Clip -> progressive lavc mpeg4 (-vf lavcdeint arg added to mencoder) with mencoder, since the DV from the camera is interlaced.
muxed video and audio with transcode's "avimerge" program.
for the audio step, you could convert to mp3, or maybe leave it as raw PCM (aiff), its up to you. mencoder can convert to mp3 with one of its -oac codecs, or if you used -oac copy you'd just get whatever audio is in the DV clip, which i'm assuming would have been converted to aiff by iMovie. there are open source tools for ac3 or aac encoding... i know that the aac programs are called faac and faad, but for the life of me i cant remember now if there is an open source tool that can create ac3 audio. perhaps "sox" can do it... i cant remember. google yields some results but at least one of the programs is win32 only.
phew.
posted by joeblough at 7:21 PM on October 8, 2005
Response by poster: i think that's my plan:
divx-avi > quicktime pro > mov > imovie > DV > mencoder > mpeg4/divx
posted by jruckman at 8:02 PM on October 8, 2005
divx-avi > quicktime pro > mov > imovie > DV > mencoder > mpeg4/divx
posted by jruckman at 8:02 PM on October 8, 2005
Response by poster: i used quicktime pro to trim my clips down, BTW -- imove would have taken days to render all the original avi clips
posted by jruckman at 8:03 PM on October 8, 2005
posted by jruckman at 8:03 PM on October 8, 2005
This thread is closed to new comments.
divx/mpeg4 is pretty much state of the art and is used almost universally on the 'net. only h.264 is more advanced than mpeg4 at this point, and you're right, its just too new. divx will probably get you better looking video at lower bitrates than the lavc codec, but its probably not worth the money.
to edit with iMovie or FCPHD, i'm pretty certain you'll have to convert the pre-existing .avi files to DV first. this might be doable with mencoder, or perhaps even Quicktime Pro if you install the divx decoder (which i think is free - but i think its broken against quicktime 7/tiger, so beware). one problem i frequently have though is that quicktime doesnt seem to like mp3 audio in a divx/mpeg4 .avi and won't play/export the audio. either way you're talking about loads and loads of disk space for this video in DV format.
anyway, for that size video, lavc mpeg4 should look great even with only 1500kbits/sec or so.
posted by joeblough at 12:48 AM on October 8, 2005