DVD archiving to a media server - best practices?
May 22, 2007 7:03 AM Subscribe
I'm planning on ripping my DVD collection to a media server, with varying amounts of losslessness depending on the film. There are a lot of ways to do this, and I'm looking for the "best" way. After the jump I'll describe my current plan, please comment on and clarify the various technical considerations.
I own about 250 DVD movie titles, and a lot more non-movie DVDs. In trying to reduce clutter from my life and home, I intend to sell all of the movie DVDs except for a handful that have some special significance to me. However, I don't intend to lose the movies, but instead rip them to a fault-tolerant HD array (maybe Drobo, maybe ReadyNAS, maybe something else). Not completely legal, I know, but I'm going to do it anyway.
For proper archival, losslessness is very important. The obvious tricky thing that results from that is the amount of hard drive space required, especially since fault tolerance is a must. There's no way around the fact that a lot of money needs to be spent on hard drives.
To lower the HD costs somewhat, I'll divide the DVD's into two categories: 1) lossless ripping for important films with several discs, particularly good extras or otherwise high bitrate requirements, and 2) lossy ripping for less important films and films with unimportant extras. The lossless option is simple: I'll just decrypt the DVD and copy the VIDEO_TS folder onto the HD array. I have all the tools I need for this part, so the lossy part is the meat of this post.
I'm probably going to divide the lossy category into two subcategories: DVD9->DVD5 for movies where the retention of some extras and "navigable" DVD bits is necessary, and XviD/H.264 for movies where the main feature is enough.
This is where my knowledge ends. There are so many movies that I don't want to spend too much time tweaking individual rips for best possible quality, so I need some general guidelines. Actual questions:
1. Codec: Should I go with XviD or some H.264 variant for the "main feature only" movies? Relevant issues: quality/disk space, openness, quality of encoders, cross-platform compatibility, expected longevity of the format.
2. Continuing from the last question, what if the main feature fits on a DVD5 without recompression? Then it's really only a matter of HD space: 4+ gigs for the original MPEG2, or X gigs for a lossily encoded format. How big would an H.264 encoded movie file have to be before the loss of quality is near-imperceptible from the original? How about XviD? (Obviously I'm not going to scale the picture down from its original dimensions, i.e. 720x576 for PAL movies which most of my DVDs are.)
3. Container format: Matroska is very flexible with multiple audio and subtitle tracks, but sparsely supported by stand-alone players. Currently I'm using Xbox Media Center which works fine, but there's no telling what I'll use in the future. AVI seems too limited, and I don't really know much about MP4. The factors of question 1 apply. Which one should I pick?
4. Audio encoding: I'll want to preserve multi-channel audio: usually DD5.1, but preferably DTS when available. Can VBR MP3 handle these, or do I have to include the audio track as-is even if the video is XviD or H.264? AVI doesn't support multi-channel (IIRC), any caveats with other containers?
Any other relevant considerations are welcome, too. I have access to Mac, Windows and Linux computers, so I'll be able to use pretty much whatever tools you suggest.
I own about 250 DVD movie titles, and a lot more non-movie DVDs. In trying to reduce clutter from my life and home, I intend to sell all of the movie DVDs except for a handful that have some special significance to me. However, I don't intend to lose the movies, but instead rip them to a fault-tolerant HD array (maybe Drobo, maybe ReadyNAS, maybe something else). Not completely legal, I know, but I'm going to do it anyway.
For proper archival, losslessness is very important. The obvious tricky thing that results from that is the amount of hard drive space required, especially since fault tolerance is a must. There's no way around the fact that a lot of money needs to be spent on hard drives.
To lower the HD costs somewhat, I'll divide the DVD's into two categories: 1) lossless ripping for important films with several discs, particularly good extras or otherwise high bitrate requirements, and 2) lossy ripping for less important films and films with unimportant extras. The lossless option is simple: I'll just decrypt the DVD and copy the VIDEO_TS folder onto the HD array. I have all the tools I need for this part, so the lossy part is the meat of this post.
I'm probably going to divide the lossy category into two subcategories: DVD9->DVD5 for movies where the retention of some extras and "navigable" DVD bits is necessary, and XviD/H.264 for movies where the main feature is enough.
This is where my knowledge ends. There are so many movies that I don't want to spend too much time tweaking individual rips for best possible quality, so I need some general guidelines. Actual questions:
1. Codec: Should I go with XviD or some H.264 variant for the "main feature only" movies? Relevant issues: quality/disk space, openness, quality of encoders, cross-platform compatibility, expected longevity of the format.
2. Continuing from the last question, what if the main feature fits on a DVD5 without recompression? Then it's really only a matter of HD space: 4+ gigs for the original MPEG2, or X gigs for a lossily encoded format. How big would an H.264 encoded movie file have to be before the loss of quality is near-imperceptible from the original? How about XviD? (Obviously I'm not going to scale the picture down from its original dimensions, i.e. 720x576 for PAL movies which most of my DVDs are.)
3. Container format: Matroska is very flexible with multiple audio and subtitle tracks, but sparsely supported by stand-alone players. Currently I'm using Xbox Media Center which works fine, but there's no telling what I'll use in the future. AVI seems too limited, and I don't really know much about MP4. The factors of question 1 apply. Which one should I pick?
4. Audio encoding: I'll want to preserve multi-channel audio: usually DD5.1, but preferably DTS when available. Can VBR MP3 handle these, or do I have to include the audio track as-is even if the video is XviD or H.264? AVI doesn't support multi-channel (IIRC), any caveats with other containers?
Any other relevant considerations are welcome, too. I have access to Mac, Windows and Linux computers, so I'll be able to use pretty much whatever tools you suggest.
For proper archival, losslessness is very important
MPEG-2 (DVD format) is already lossy, as is most of the associated audio on DVDs (PCM a notable exception).
If you're planning on streaming wirelessly, you'll want to reduce bitrate to something your network is comfortable with. A good 802.11g can manage DVDs, but with a max bitrate of up to 10 mbps (6 typical), running multiple streams can become problematic.
Planning ahead, HD-XVID or H.264 can reduce 720p and 1080i/p to a wireless-comfortable bitrate.
VBR audio with video frames is a pain in the arse to synchronise if you do any cuts or have lag issues, and you just don't save that much space. Avoid.
posted by meehawl at 7:22 AM on May 22, 2007 [1 favorite]
MPEG-2 (DVD format) is already lossy, as is most of the associated audio on DVDs (PCM a notable exception).
If you're planning on streaming wirelessly, you'll want to reduce bitrate to something your network is comfortable with. A good 802.11g can manage DVDs, but with a max bitrate of up to 10 mbps (6 typical), running multiple streams can become problematic.
Planning ahead, HD-XVID or H.264 can reduce 720p and 1080i/p to a wireless-comfortable bitrate.
VBR audio with video frames is a pain in the arse to synchronise if you do any cuts or have lag issues, and you just don't save that much space. Avoid.
posted by meehawl at 7:22 AM on May 22, 2007 [1 favorite]
I've archived stacks of DVDs to my HTPC using many strategies. Sometimes I'll rip DVDs of TV episodes out to uncompressed standalone .mpg files, other times I'll opt for DIVX or h.264.
I've had excellent results using Auto Gordian Knot (Windows) to create DIVX files with AVI containers. I use 2-pass encoding, 640x386 (I'm not sure if you can do native 720x576), mp3 audio.
AutoGK allows you to create .avi files that are compatible with set-top machines like the Philips DVP642.
For h.264 encoding, I use Handbrake (Macintosh).
posted by porn in the woods at 7:27 AM on May 22, 2007
I've had excellent results using Auto Gordian Knot (Windows) to create DIVX files with AVI containers. I use 2-pass encoding, 640x386 (I'm not sure if you can do native 720x576), mp3 audio.
AutoGK allows you to create .avi files that are compatible with set-top machines like the Philips DVP642.
For h.264 encoding, I use Handbrake (Macintosh).
posted by porn in the woods at 7:27 AM on May 22, 2007
I'm not qualified to argue over the compression formats, except to note that you are forgetting how much time it takes to convert a DVD to some other format. At 20 minutes per disc (which is unbelievably fast), and assuming a total of 300 DVDs, this is an 100 man-hour project.
I just checked newegg, and you can get 400 GB for $100. If we assume every disc is 5 GB, you'll need 1,500 GB to copy all your discs without compression. That's only $400 worth of disks. (yes, it's a lot of money). $500 if you want to get a 5th disk but configure a RAID array.
The thing is that after spending all the time to do the compression, are you the type of person who is going to want to do a lossless copy at some point in the future?
Disks get cheaper every month, and larger disks are less expensive per GB than smaller ones. And straight disk copies are faster than encoding or conversions.
You can save space by only copying the main movie file rather than all the extras, roughly 1GB per hour.
This way you get DVD quality, full sound, and you don't spend 6 months doing a compression that you end up regretting later.
posted by Pastabagel at 8:02 AM on May 22, 2007
I just checked newegg, and you can get 400 GB for $100. If we assume every disc is 5 GB, you'll need 1,500 GB to copy all your discs without compression. That's only $400 worth of disks. (yes, it's a lot of money). $500 if you want to get a 5th disk but configure a RAID array.
The thing is that after spending all the time to do the compression, are you the type of person who is going to want to do a lossless copy at some point in the future?
Disks get cheaper every month, and larger disks are less expensive per GB than smaller ones. And straight disk copies are faster than encoding or conversions.
You can save space by only copying the main movie file rather than all the extras, roughly 1GB per hour.
This way you get DVD quality, full sound, and you don't spend 6 months doing a compression that you end up regretting later.
posted by Pastabagel at 8:02 AM on May 22, 2007
You will not be able to encode multi-channel audio in MP3 unless you're willing to matrix it into Dolby Pro-Logic II. I'd just include the original DD or DTS. Dolby Digital is usually 384kbps on DVDs, so it's not much bitrate. AVI can hold those formats just fine. The MP4 format will allow you to include multi-track AAC audio, but I'd just keep it simple and include the original audio, even in your lossy encodes.
That being said, I like Pastabagel's idea. Just copy everything; no re-encoding.
posted by marionnette en chaussette at 8:53 AM on May 22, 2007 [1 favorite]
That being said, I like Pastabagel's idea. Just copy everything; no re-encoding.
posted by marionnette en chaussette at 8:53 AM on May 22, 2007 [1 favorite]
Since you're interested in 1. preserving the DVD extras and 2. keeping all the video lossless..., have you considered simply keeping your DVD collection in a DVD jukebox? My brother went this route and has been very happy with it. (I can find out the exact model he got if you like.)
Ripping DVDs to hard drive makes a lot of sense when you're dumping the extras and compressing the movies to 1 - 1.5GB apiece, but those 8.5GB ISOs are gonna eat you alive -- a TERABYTE of HD space will only store 117 discs! And considering all the time you're going to put into ripping them, you'd better damn sure have some data redundancy (e.g., RAID-5) for when, not "if", a hard drive fails.
Just thinking about sinking all those hours into ripping 250 DVDs in the first place makes me ill, but you may have more patience than me. Still, since you have the discs, just dump 'em in a jukebox and be done with it.
posted by LordSludge at 2:24 PM on May 22, 2007
Ripping DVDs to hard drive makes a lot of sense when you're dumping the extras and compressing the movies to 1 - 1.5GB apiece, but those 8.5GB ISOs are gonna eat you alive -- a TERABYTE of HD space will only store 117 discs! And considering all the time you're going to put into ripping them, you'd better damn sure have some data redundancy (e.g., RAID-5) for when, not "if", a hard drive fails.
Just thinking about sinking all those hours into ripping 250 DVDs in the first place makes me ill, but you may have more patience than me. Still, since you have the discs, just dump 'em in a jukebox and be done with it.
posted by LordSludge at 2:24 PM on May 22, 2007
A friend of mine did exactly what you want to do, created a huge data base of ripped dvds. He then convinced me to buy a big ol raid to copy them to. Now I have 300 ripped movies at home.
Sounds good eh?
Well here's the problem: at the same time I bought the raid, I decided to buy a new high-def 1080p TV. Now I can play the dvds he ripped on my tv, but they are in standard def, and many are compressed. After watching a few movies high-def uncompressed, I can't go back to standard. Its seems worse than black and white. And the compression makes it look like a truck ran over it.
So my point is, your DVD rips are already becoming obsolete, and if you compress them even more so.
Now what I did was get a hi-def recording card and huge raid and am building my collection of 1080 movies that way. Its not as quick and easy but its far better. Of course, from this angle LordSludge's complaint about 8.5gb isos is laughable...
posted by Osmanthus at 3:41 PM on May 22, 2007
Sounds good eh?
Well here's the problem: at the same time I bought the raid, I decided to buy a new high-def 1080p TV. Now I can play the dvds he ripped on my tv, but they are in standard def, and many are compressed. After watching a few movies high-def uncompressed, I can't go back to standard. Its seems worse than black and white. And the compression makes it look like a truck ran over it.
So my point is, your DVD rips are already becoming obsolete, and if you compress them even more so.
Now what I did was get a hi-def recording card and huge raid and am building my collection of 1080 movies that way. Its not as quick and easy but its far better. Of course, from this angle LordSludge's complaint about 8.5gb isos is laughable...
posted by Osmanthus at 3:41 PM on May 22, 2007
Osmanthus, as cool as that sounds, I weep for you.
posted by LordSludge at 9:03 PM on May 22, 2007
posted by LordSludge at 9:03 PM on May 22, 2007
Response by poster: Some really interesting viewpoints presented so far.
A jukebox won't be an option, since I want the discs out, gone.
HD is coming, definitely. This is why I'm doing this now, when the DVD format is still relevant. I won't get a lot of money per disc even at this stage, but at least I'll be able to unload the collection somewhere for some money without enormous difficulty.
As for quality, well, DVD quality is quite good enough. No, really :-). Judging from what I've read about decent SD-to-1080p upscaling hardware, it will make the viewing experience with HD equipment even quite a lot better in some cases.
I must admit I've thought about the "regretting having a further compressed rip" issue. If I just rip everything as-is, that's about 2TB of space required (most dual-layer DVDs tend to take about 6-8 gigs, and then there are the two-disc titles), which means three 1TB or four 750GB drives for a RAID5 setup (or a similar proprietary scheme, like ReadyNAS's X-RAID or Drobo's "RAID without the name" system). But this is something I'll just have to experiment with. If my test encodes (either DVD9->DVD5 or Xvid/H.264) don't produce good enough quality, then I'll just either a) copy the whole discs to RAID or b) copy just the main feature and discard the extras.
Drobo seems like the strongest storage option at this point, with its freely mixable drive sizes creating an easy, affordable expansion path later on.
Time of encoding isn't that big of a deal; I'll just background the task and be done eventually. I've got more than enough CPU's lying around at home and at work (fortunately I'm able to do stuff like this at work).
When one of the HD formats finally overtakes DVD for good, I don't intend to start upgrading my movie collection. My disc collecting days are definitely over. I'll just download the HD versions, either ripped or legit, whichever is more convenient in the future. I may buy HD discs, but I'll do the same rip/copy to hd/sell dance that I'm about to do with DVDs now. This will obviously have to wait for the necessary disk space to be relatively cheaply available. In the near future, HD will have little impact in my life - I might upgrade individual titles of my collection from standard to high def, but that'll be it. I don't even know when I'll buy HD-capable equipment.
posted by lifeless at 11:33 PM on May 22, 2007
A jukebox won't be an option, since I want the discs out, gone.
HD is coming, definitely. This is why I'm doing this now, when the DVD format is still relevant. I won't get a lot of money per disc even at this stage, but at least I'll be able to unload the collection somewhere for some money without enormous difficulty.
As for quality, well, DVD quality is quite good enough. No, really :-). Judging from what I've read about decent SD-to-1080p upscaling hardware, it will make the viewing experience with HD equipment even quite a lot better in some cases.
I must admit I've thought about the "regretting having a further compressed rip" issue. If I just rip everything as-is, that's about 2TB of space required (most dual-layer DVDs tend to take about 6-8 gigs, and then there are the two-disc titles), which means three 1TB or four 750GB drives for a RAID5 setup (or a similar proprietary scheme, like ReadyNAS's X-RAID or Drobo's "RAID without the name" system). But this is something I'll just have to experiment with. If my test encodes (either DVD9->DVD5 or Xvid/H.264) don't produce good enough quality, then I'll just either a) copy the whole discs to RAID or b) copy just the main feature and discard the extras.
Drobo seems like the strongest storage option at this point, with its freely mixable drive sizes creating an easy, affordable expansion path later on.
Time of encoding isn't that big of a deal; I'll just background the task and be done eventually. I've got more than enough CPU's lying around at home and at work (fortunately I'm able to do stuff like this at work).
When one of the HD formats finally overtakes DVD for good, I don't intend to start upgrading my movie collection. My disc collecting days are definitely over. I'll just download the HD versions, either ripped or legit, whichever is more convenient in the future. I may buy HD discs, but I'll do the same rip/copy to hd/sell dance that I'm about to do with DVDs now. This will obviously have to wait for the necessary disk space to be relatively cheaply available. In the near future, HD will have little impact in my life - I might upgrade individual titles of my collection from standard to high def, but that'll be it. I don't even know when I'll buy HD-capable equipment.
posted by lifeless at 11:33 PM on May 22, 2007
You know what I suggest? And this is a serious suggestion. Get a Netflix account. They have HDDVD and Bluray already.
Here's the thing. Let's say you drop $1000 on hardware, okay? You have to maintain it, etc. You spend money to buy new discs every so often, which increases the cost up until you have to add a new drive, and then it's a big increase. Plus all the time. And how often are you going to watch the same movie?
Don't think of Netflix as a movie rental. Think of it as off-site storage of your media. See, it's just $20 a month to store *your* 30,000 DVD and hi-def titles in their warehouse. You want to watch one of your movies, it's a slightly inconvenient 36-hour latency. (Netflix also streams xvid quality movies now to, as part of the $20/ month).
You free yourself of hardware headaches and spending all that money.
posted by Pastabagel at 7:32 AM on May 23, 2007
Here's the thing. Let's say you drop $1000 on hardware, okay? You have to maintain it, etc. You spend money to buy new discs every so often, which increases the cost up until you have to add a new drive, and then it's a big increase. Plus all the time. And how often are you going to watch the same movie?
Don't think of Netflix as a movie rental. Think of it as off-site storage of your media. See, it's just $20 a month to store *your* 30,000 DVD and hi-def titles in their warehouse. You want to watch one of your movies, it's a slightly inconvenient 36-hour latency. (Netflix also streams xvid quality movies now to, as part of the $20/ month).
You free yourself of hardware headaches and spending all that money.
posted by Pastabagel at 7:32 AM on May 23, 2007
Response by poster: Might have mentioned that I live in Finland, so no Netflix here... (or any equivalent service).
posted by lifeless at 12:24 AM on May 25, 2007
posted by lifeless at 12:24 AM on May 25, 2007
This thread is closed to new comments.
1. I use XviD. I didn't belabor it too much, not saying it's the best of all possible worlds, but it works fine for me. I actually encode the main feature and all of the special features, including audio commentary. I do the audio commentary as standalone mp3s. I use mplayer as my playback agent and it supports playing a movie file with a different audio track.
2. I have a few XviD settings but basically most of them center around hitting a target MB/hour of video. 700MB/hour seems pretty good to me.
3. I use AVI (libavcodec I think). I *thought* it could encode multi channel sound but I'm not sure. I encode most movies I don't care enough about to have the full disk in stereo.
I have all my original disks, so this is a convenience issue for me. As such I did not focus so much of future survivability. YMMV...
posted by RustyBrooks at 7:20 AM on May 22, 2007