The Invisible, Omnipresent Digital Backpack
July 16, 2012 10:53 PM Subscribe
I want to digitize everything, and I mean everything: DVDs, CDs, books, food… okay, not food, just the media. What should I keep in mind? How can I make this easy?
(I've looked at some previous questions, and this one is the closest I found.)
My main concern is having my media be reliably accessible without (much of) a hassle.
CDs I have a process for, since a good amount of the sort of thing I like is only available in that form, and I only listen to the stuff digitally. It can be a bit of an involved process, though, since I often have to get the metadata manually. Then I rip the CDs to FLAC for archival purposes.
DVDs I have no process for. Some of what I have isn't really necessary to keep on hand, what with Netflix streaming and all. Other things aren't available on Netflix (or aren't always available — ugh, don't get me started). And I'm a bit of a fan of extra features, which are entirely exclusive to the physical media (or so I believe). I've ripped a few DVDs using Handbrake, but it's also an involved process and I end up just grabbing the main feature. I'm sure I could rip something with alternate audio tracks, just like the subtitle tracks, but I'm not that familiar with the process. And I don't also want to go through the hassle of ripping the featurettes separately. I might not mind just making disk images of the DVDs and figuring out what to do about video files later.
Books are more of a problem. I have a Kindle Touch, and I've already downloaded a few books I own and love to re-read, but I can't easily get them all. Some are free, some are strangely expensive, some have not-so-good formatting, some have a confusing number of editions, some are not available. And books can't be ripped (uh, in the same sense as the optical media).
I have a home server and a setup involving multiple external drives and rsync, but I'm already kind of tired of dealing with it and getting more so. It would be great to have all of my media "in the cloud", accessible anywhere and anytime. That seems kind of like a pipe dream, though, considering my travels can take me to places where I can hardly check my e-mail. But everywhere I go I have my laptop (MBP) and phone (Galaxy Nexus) with me, and now my Kindle as well.
I'd also like to offload/outsource/whatever the digitizing effort to someone else if possible. Are there good services for this out there? Is there someone with reasonable rates I can trust?
(I've looked at some previous questions, and this one is the closest I found.)
My main concern is having my media be reliably accessible without (much of) a hassle.
CDs I have a process for, since a good amount of the sort of thing I like is only available in that form, and I only listen to the stuff digitally. It can be a bit of an involved process, though, since I often have to get the metadata manually. Then I rip the CDs to FLAC for archival purposes.
DVDs I have no process for. Some of what I have isn't really necessary to keep on hand, what with Netflix streaming and all. Other things aren't available on Netflix (or aren't always available — ugh, don't get me started). And I'm a bit of a fan of extra features, which are entirely exclusive to the physical media (or so I believe). I've ripped a few DVDs using Handbrake, but it's also an involved process and I end up just grabbing the main feature. I'm sure I could rip something with alternate audio tracks, just like the subtitle tracks, but I'm not that familiar with the process. And I don't also want to go through the hassle of ripping the featurettes separately. I might not mind just making disk images of the DVDs and figuring out what to do about video files later.
Books are more of a problem. I have a Kindle Touch, and I've already downloaded a few books I own and love to re-read, but I can't easily get them all. Some are free, some are strangely expensive, some have not-so-good formatting, some have a confusing number of editions, some are not available. And books can't be ripped (uh, in the same sense as the optical media).
I have a home server and a setup involving multiple external drives and rsync, but I'm already kind of tired of dealing with it and getting more so. It would be great to have all of my media "in the cloud", accessible anywhere and anytime. That seems kind of like a pipe dream, though, considering my travels can take me to places where I can hardly check my e-mail. But everywhere I go I have my laptop (MBP) and phone (Galaxy Nexus) with me, and now my Kindle as well.
I'd also like to offload/outsource/whatever the digitizing effort to someone else if possible. Are there good services for this out there? Is there someone with reasonable rates I can trust?
Most smartphones have applications available which help them to act as scanners. For example Turboscan for iPhones. These let you combine multiple images into a scan for higher fidelity, crop, tag and then store or mail as PDF (for example). This process is often faster than dragging whatever you want to capture to a scanner. It is also good for providing a digitization of things like pictures, maps and other awkwardly positioned stuff.
I would consider spending some money on a cloud based backup solution that gives you unlimited storage space. After tracking other askme threads I chose Backblaze to do this for me - have been happy with it.
posted by rongorongo at 1:32 AM on July 17, 2012
I would consider spending some money on a cloud based backup solution that gives you unlimited storage space. After tracking other askme threads I chose Backblaze to do this for me - have been happy with it.
posted by rongorongo at 1:32 AM on July 17, 2012
If you want to have all your media available, and you have a decently reliable home connection, I'd suggest you check out Plex. It's a very nice branch of XBMC; the media server component runs on all three major operating systems, and with the MyPlex component you have easy media portability.
For DVDs, if you want to keep all the extras and menus and such, ripping to ISO is definitely your best bet.
posted by jammer at 7:11 AM on July 17, 2012
For DVDs, if you want to keep all the extras and menus and such, ripping to ISO is definitely your best bet.
posted by jammer at 7:11 AM on July 17, 2012
Best answer: RipIt for the Mac makes automating DVD ripping very easy. If you go into System Preferences and tell your Mac to launch RipIt when you insert a DVD, you can then set RipIt to rip the disc and eject it when it's done. This can be hacked together with free utilities but I like RipIt's simplicity. It'll also convert the files into iTunes-playable formats, but you might want to fiddle with the options more than it lets you.
posted by davextreme at 7:11 AM on July 17, 2012
posted by davextreme at 7:11 AM on July 17, 2012
Best answer: Think like an archivist, and imagine a long horizon.
1) Come up with a master indexing system for your entire collection of analog materials. Everything has a number code and is listed in a spreadsheet with a brief description (or a database, with more metadata, like album cover scans, for example). Mark the index numbers on the analog materials -- both actual material and any case/binding they are in -- in a way that is reasonably permanent (except for rare or fragile items, then use high quality labels). Use descriptive codes -- HBB-001--> for hardback books, for example. You have no idea how confusing it will be if you undertake a massive digitizing project on multiple fronts at once. Or how much you will forget about the items you digitized in a few years. The point is to have full control over your workflow and leave a searchable record of what you converted (in the case of commercial audio, using online databases saves you this time and you might want to look into iTunes Match, for example, if you have a lot of old music).
2) Multiple fronts: audio and video digitizing is a real time deal. It requires a lot of computer processing power. You need to monitor it to some extent. Get a few machines in one room to do different real-time digitizing tasks at once so you do not dissipate this effort and time.
3) Quality: decide in advance what your future needs are, and err on the side of quality if unsure. You can run a line in from a cassette deck into the mic/audio in jack of any modern computer and digitize into audacity, or you can spend thousands on a dedicated audio interface, for example. Somewhere in the middle might be your sweet spot. Same for formats. Compression saves drive space, but with what it costs these days there is no point. Used the highest-quality uncompressed formats you can for all media types. TIFF for images; TIFF or very high quality PDF for documents, PCM/WAV for audio, etc. You can skimp on stuff you know is replicable crap -- mass market books or music you probably wouldn't keep in CD form, etc. -- and save drive space. But when in doubt, pull out the stops. Today's high quality is tomorrow's average quality.
4) Hard drives, not optical media. Tape still has its partisans but I think they are crazy. And now there's flash memory in the large-scale storage equation. ALL are an unknown long-term bet, but we are sure about the limits of optical media and the very quick decline thereof, which means getting your archive OFF optical media in the future could be a beast. Roughly estimate how much drive space you need, then double it; then double it again because you need to run a simultaneous onsite backup; then double it again to make an offsite backup. This is everything you own, and drives are so cheap now. I just bought a bunch of 3TB firewire/usb drives for $130 each!
5) Data AND format migration plan: use non-proprietary data formats if possible (an exception, for me, is PDF, which is so ubiquitous you cannot imagine a future where it would become unreadable). Count on needing to move the archive to new media regularly, and stay aware of changes in technology that might affect your archive. Don't forget that you will almost certainly need to migrate formats as well in some cases in the future (how far in the future, no one can say).
But I cannot emphasize enough how important the initial collection of metadata will do for you in the future. It's not something you can usually farm out to others if your materials are at all unique or unusual. I recommend seeing if you can find a young musician or college kid who's good with computers and/or audio-video stuff to man the digitizing stations at an hourly rate. You can do that while doing other things (studying, for example, or for that matter entering metadata) if you're a good multitasker. Some media you can just send off to services -- commercial audio and video on CD, DVD, VHS, etc. But I'm speaking of collections that have unique materials - all your band demos, the rare book or LP collection, the photos of your travels -- things where only you know the metadata facts, and where only you are superinvested in making sure they survive.
posted by spitbull at 7:15 AM on July 17, 2012 [3 favorites]
1) Come up with a master indexing system for your entire collection of analog materials. Everything has a number code and is listed in a spreadsheet with a brief description (or a database, with more metadata, like album cover scans, for example). Mark the index numbers on the analog materials -- both actual material and any case/binding they are in -- in a way that is reasonably permanent (except for rare or fragile items, then use high quality labels). Use descriptive codes -- HBB-001--> for hardback books, for example. You have no idea how confusing it will be if you undertake a massive digitizing project on multiple fronts at once. Or how much you will forget about the items you digitized in a few years. The point is to have full control over your workflow and leave a searchable record of what you converted (in the case of commercial audio, using online databases saves you this time and you might want to look into iTunes Match, for example, if you have a lot of old music).
2) Multiple fronts: audio and video digitizing is a real time deal. It requires a lot of computer processing power. You need to monitor it to some extent. Get a few machines in one room to do different real-time digitizing tasks at once so you do not dissipate this effort and time.
3) Quality: decide in advance what your future needs are, and err on the side of quality if unsure. You can run a line in from a cassette deck into the mic/audio in jack of any modern computer and digitize into audacity, or you can spend thousands on a dedicated audio interface, for example. Somewhere in the middle might be your sweet spot. Same for formats. Compression saves drive space, but with what it costs these days there is no point. Used the highest-quality uncompressed formats you can for all media types. TIFF for images; TIFF or very high quality PDF for documents, PCM/WAV for audio, etc. You can skimp on stuff you know is replicable crap -- mass market books or music you probably wouldn't keep in CD form, etc. -- and save drive space. But when in doubt, pull out the stops. Today's high quality is tomorrow's average quality.
4) Hard drives, not optical media. Tape still has its partisans but I think they are crazy. And now there's flash memory in the large-scale storage equation. ALL are an unknown long-term bet, but we are sure about the limits of optical media and the very quick decline thereof, which means getting your archive OFF optical media in the future could be a beast. Roughly estimate how much drive space you need, then double it; then double it again because you need to run a simultaneous onsite backup; then double it again to make an offsite backup. This is everything you own, and drives are so cheap now. I just bought a bunch of 3TB firewire/usb drives for $130 each!
5) Data AND format migration plan: use non-proprietary data formats if possible (an exception, for me, is PDF, which is so ubiquitous you cannot imagine a future where it would become unreadable). Count on needing to move the archive to new media regularly, and stay aware of changes in technology that might affect your archive. Don't forget that you will almost certainly need to migrate formats as well in some cases in the future (how far in the future, no one can say).
But I cannot emphasize enough how important the initial collection of metadata will do for you in the future. It's not something you can usually farm out to others if your materials are at all unique or unusual. I recommend seeing if you can find a young musician or college kid who's good with computers and/or audio-video stuff to man the digitizing stations at an hourly rate. You can do that while doing other things (studying, for example, or for that matter entering metadata) if you're a good multitasker. Some media you can just send off to services -- commercial audio and video on CD, DVD, VHS, etc. But I'm speaking of collections that have unique materials - all your band demos, the rare book or LP collection, the photos of your travels -- things where only you know the metadata facts, and where only you are superinvested in making sure they survive.
posted by spitbull at 7:15 AM on July 17, 2012 [3 favorites]
I also wanted to plug a very simple solution for analog (VHS, Beta, Hi8) video digitizing, assuming you aren't seeking to retain the highest quality DV resolution. The BlackMagic video recorder, for about $100, takes any analog video input into a mac via USB and records a quicktime movie at decent quality. It is so simple. That is its real virtue, so if you know (for example) relatively non-technical people who need to do this (I know many, which is why I recommend this all the time) it is a nearly foolproof solution.
posted by spitbull at 7:18 AM on July 17, 2012
posted by spitbull at 7:18 AM on July 17, 2012
Before I moved across the country 2 years ago, I went totally digital so as to unburden myself of all my DVDs and CDs. I couldn't see the sense in moving 200 DVDs when I could put them all on a drive that fit in my hand and weighed almost nothing compared to the weight of the discs.
Anyway, to the nitty gritty.
For streamlining the process of ripping all your CDs and DVDs, you could get on some nice torrent trackers and download all of the media you already own in digital format. (Of course, I'm only suggesting you downloading things you've already bought and were going to rip.) When you can click a couple of times to get the whole Buffy series instead of manually setting up and ripping 50 DVDs, you'll see the time savings immediately.
I want to echo spitbull in using something like XBMC, because this program will scrape metadata for you from thetvdb and themoviedb, so you don't have to spend any time entering metadata by hand. If you read up on what the filename formats that XBMC looks for when scraping metadata, you'll find that if you've been naming things consistently so far (like the tenth episode of season 3 of "Buffy" lives in the "Buffy" folder and is named 03x10.avi), you won't have to do anything.
XBMC also allows for off-site access, although I haven't been able to perfect that yet since I'm behind some weird firewalls I can't control, so I can't speak to the "cloud" nature of the XBMC program (i.e., being able to access stuff from anywhere). Ideally, you'd have some large tower sitting in your closet and a small, low-noise front end sitting in your living room connected to your television. You can access stuff on your phone and watch as long as you are on a network that can "see" your tower from the outside.
What you should do when you are done with this is put all the DVDs, CDs, and books into some boxes, and then bring them to your local library. You could sell them if you want, but that might be more hassle than it's worth, plus you'll be doing a good thing by donating them to a place you know would get some use out of them.
Also, I'm getting rambly so I will stop. Good luck.
posted by King Bee at 1:43 PM on July 17, 2012
Anyway, to the nitty gritty.
For streamlining the process of ripping all your CDs and DVDs, you could get on some nice torrent trackers and download all of the media you already own in digital format. (Of course, I'm only suggesting you downloading things you've already bought and were going to rip.) When you can click a couple of times to get the whole Buffy series instead of manually setting up and ripping 50 DVDs, you'll see the time savings immediately.
I want to echo spitbull in using something like XBMC, because this program will scrape metadata for you from thetvdb and themoviedb, so you don't have to spend any time entering metadata by hand. If you read up on what the filename formats that XBMC looks for when scraping metadata, you'll find that if you've been naming things consistently so far (like the tenth episode of season 3 of "Buffy" lives in the "Buffy" folder and is named 03x10.avi), you won't have to do anything.
XBMC also allows for off-site access, although I haven't been able to perfect that yet since I'm behind some weird firewalls I can't control, so I can't speak to the "cloud" nature of the XBMC program (i.e., being able to access stuff from anywhere). Ideally, you'd have some large tower sitting in your closet and a small, low-noise front end sitting in your living room connected to your television. You can access stuff on your phone and watch as long as you are on a network that can "see" your tower from the outside.
What you should do when you are done with this is put all the DVDs, CDs, and books into some boxes, and then bring them to your local library. You could sell them if you want, but that might be more hassle than it's worth, plus you'll be doing a good thing by donating them to a place you know would get some use out of them.
Also, I'm getting rambly so I will stop. Good luck.
posted by King Bee at 1:43 PM on July 17, 2012
Response by poster: Thanks for the responses so far, folks. I've recently played with Calibre (which rocks), and DIY Book Scanner looks amazing!
A couple points I'd like to respond to and/or clear up:
I mentioned I have a home server. I know a bit about what I'm doing, but I'd prefer not to deal with it anymore. This is partly because right now my ISP does some annoying firewall crap I've had to resort to SSH tunnel tricks to get around, and I don't have a choice (that I know of) of a different provider. I'm also going to be moving soon-ish and hence will have my home server at least temporarily off-line. And really, I'd rather not be a sysadmin even on a slice. Imagine I'm really, really, really lazy. If there's a reliable, accessible, reasonably-priced cloud service, I'd love to use it. Hell, I should be doing something like this for offsite backups, at least. I've been too lazy to look into it, but this is not the first time I've heard good things about Backblaze. I can also price out S3.
I'm totally with you on the torrent idea. Much of my trouble is that I'm buying CDs because they're not easily available in any digital format, legit or no. They're not as much of a problem, though. They come in seldom enough that my process handles them okay. I just have a bit of a backlog right now because of that big purchase I took a picture of. And I also have a stack of things I should get to and haven't (like Israeli ska and 90s rock, oh yeah). I'm not terribly worried about this, but I wouldn't mind someone else doing it for me. I'm also not terribly worried about most metadata, trusting on-line sources where available and maybe shrugging and moving on where not.
Thanks for verifying that ISOs are probably the best bet for my DVD extra feature wants. I'ma check out RipIt and see if that works for me. And maybe convince someone else to do the work. And I know that remote hosting/streaming of video is nasty. I'm all for transferring and storing some stuff I'm more likely to want on my travels.
Keep 'em coming, please. You people are the best.
posted by cardioid at 10:47 PM on July 17, 2012
A couple points I'd like to respond to and/or clear up:
I mentioned I have a home server. I know a bit about what I'm doing, but I'd prefer not to deal with it anymore. This is partly because right now my ISP does some annoying firewall crap I've had to resort to SSH tunnel tricks to get around, and I don't have a choice (that I know of) of a different provider. I'm also going to be moving soon-ish and hence will have my home server at least temporarily off-line. And really, I'd rather not be a sysadmin even on a slice. Imagine I'm really, really, really lazy. If there's a reliable, accessible, reasonably-priced cloud service, I'd love to use it. Hell, I should be doing something like this for offsite backups, at least. I've been too lazy to look into it, but this is not the first time I've heard good things about Backblaze. I can also price out S3.
I'm totally with you on the torrent idea. Much of my trouble is that I'm buying CDs because they're not easily available in any digital format, legit or no. They're not as much of a problem, though. They come in seldom enough that my process handles them okay. I just have a bit of a backlog right now because of that big purchase I took a picture of. And I also have a stack of things I should get to and haven't (like Israeli ska and 90s rock, oh yeah). I'm not terribly worried about this, but I wouldn't mind someone else doing it for me. I'm also not terribly worried about most metadata, trusting on-line sources where available and maybe shrugging and moving on where not.
Thanks for verifying that ISOs are probably the best bet for my DVD extra feature wants. I'ma check out RipIt and see if that works for me. And maybe convince someone else to do the work. And I know that remote hosting/streaming of video is nasty. I'm all for transferring and storing some stuff I'm more likely to want on my travels.
Keep 'em coming, please. You people are the best.
posted by cardioid at 10:47 PM on July 17, 2012
Response by poster: sebastienbailard (or others), how much should I worry about Amazon messing with what's on my Kindle if it comes from a torrent or other non-Amazon source? Especially if it's something that doesn't even legitimately exist in e-book form? Does it matter if I e-mail it to my Kindle, or should I transfer it directly over USB?
posted by cardioid at 10:18 PM on July 19, 2012
posted by cardioid at 10:18 PM on July 19, 2012
This thread is closed to new comments.
This is a first-principles analysis, I have no personal experience with itunes, Amazon Cloud Player, etc.
That said, let me google this for you.
Here's a fellow who just went through that stuff:
http://www.consideritfixed.com/2011/10/creating-my-private-cloud.html
Restricting the google search on "cd ripping service" to the last 12 months, I found a
WSJ article comparing 5 cd ripping services.
I'm not sure dvd ship-and-rip services exist. Hollywood spent a lot of money influencing politicians on the subject of transferring your dvds to hard disks and circumventing the copy protection on them:
http://en.wikipedia.org/wiki/Anti-circumvention
I have a home server and a setup involving multiple external drives and rsync, but I'm already kind of tired of dealing with it and getting more so. It would be great to have all of my media "in the cloud", accessible anywhere and anytime.
Regarding the multiple drives; you may want to switch to a tower case and maybe RAID?
You know you can access your home server from remote locations? The IP address you'd point your browser, file manager, vlc, ebook reader, etc. at changes once in a while, so you'd want dynamic dns:
http://en.wikipedia.org/wiki/Dynamic_dns#DDNS_for_ISP_users
You can even have a domain name associated with your home server this way.
Remotely hosting video can be kind of bandwidth intensive. Consider storing a few isos (or down-shifted formats) on your laptop when you travel.
That said, try googling remote dyndns htpc or "media server dyndns", to see some of the issues and solutions.
Books are more of a problem.
For organizing ebooks, there's calibre:
http://calibre-ebook.com/
And books can't be ripped (uh, in the same sense as the optical media).
http://www.diybookscanner.org/
also, if you have a book in dead tree, it's probably ethical to get a torrent of it ...
-------------------------------------------------------------
A quite dear geeky friend of mine is uncomfortable traveling with ripped or pirated media on his hard disks and devices. Immigration and border control folk can take your machine into the other room, scan the hard disk, and then nail you to the wall. A bit of cursory googling just now did not give any examples of people who had been abused this way, but it is a concern if you have ripped or pirated media on your devices and a government is looking for an excuse; i.e. dissidents, reporters, etc.
posted by sebastienbailard at 12:21 AM on July 17, 2012 [1 favorite]