iTunes is randomly deleting songs...which ones?

May 28, 2009 7:18 AM

Help me fix this absolute mess of my digital music collection which is in iTunes. Over 2300 CDs, but some are missing.... HELP!

Okay here's the details. In the 80s, 90s, and early 00s I amassed a fairly large CD collection, well over 2,000 CDs. many times in the past i had worked to rip all these CDs but never succeeded because of problems with CD data, etc. I also, being a Windows guy, waned a specific folder structure which added a lot of manual labor.

Giving up on my wish list, last year I decided I would just rip all my CDs using iTunes. I would let iTunes decide the folder structure, as I am an iPod user. I set iTunes to rip the CDs in Apple's lossless format, and I was not particular about the metadata figuring it's best to get the discs ripped, and I can go back and adjust the metadata to how I like it later (such as changing Phil Collins to Collins, Phil and so on). I set a computer to automatically rip CDs as they are inserted and did about 50 a week. Months later, I was done.

In that time I also have switched from being a Windows user to being a Mac user. I have also belatedly embraced digital music, no longer buying overpriced CDs and instead buying single songs or some albums on iTunes, Amazon, or some other online stores. My music collection is now an amalgam of my own ripped CDs in the Apple lossless codec, MP3s, M4As, etc.

In moving to being a Mac user, I put my music collection on an external drive hooked into a Mac Mini as the "server" and various other Macs in my house share the library. I set the Mac Mini to "Keep iTunes Music Organized". I figured since I had relinquished control of the folders to iTunes, I might as well leave it that way.

But here's the problem... I have since learned that checking the box to allow iTunes to keep the music folder organized can cause removal of data, so I turned it off about a month ago. But it was too late. I discovered this past week that at least 3 of the CDs I know I ripped are missing. Not just some songs, but entire CDs (my wife says iTunes is trying to give me taste as what I lost were the soundtracks to all 3 Beverly Hills Cop movies). Given that those 3 are gone I can only assume others are missing as well...but I have no idea which ones.

I'm now left with a problem. I want to validate all my CDs are there, and all songs on those CDs are there, but I don't want to have to manually pull 2300 CDs at an average of 10 songs per disc and then search iTunes to see if all the tracks are there or not. The amount of manual labor involved in doing that would be extraordinary.

I have thought about just saying "forget it" and reripping all 2300 CDs as that WOULD be quicker than the manual validation method, but I can't just start over as at least 100 songs I've paid for from the various sites are mixed in among the ripped tracks. (God only knows if I have lost tracks I bought from iTunes, etc. I have no real way to validate all those).

So what do I do? i need methods to either:
a) quickly and easily determine if my CDs are ripped in full

or

b) some method to extract the tracks bought from Amazon and iTunes, etc from the tracks I ripped myself so I can rerip all my CDs and reintegrate this music collection.

or

C) some other fix for this problem that will allow me to have a well-organized music library consisting of all the CDs I own plus all the tracks I have bought.

Where do I go from here??

posted by arniec to computers & internet (15 comments total)

How about:
d) Assume they are all there, then notice when they are not.

God knows how you found out a Beverly Hills Cop soundtrack was missing!

(I wouldn't worry about folder structure)
posted by devnull at 7:24 AM on May 28

Amazon songs have an Amazon tag in their comment field. You can make a smart playlist on that basis. You can download most Amazon songs again free from your digital locker, and if you buy MP3s from CD Baby, they will also download them again.

My collection is probably about 1/3 the size of yours CDwise. We're switching to digital for most things, but some you can only get on CD, like local bands that we buy CDs from at their shows. I maintain a spreadsheet on google docs telling me what I own and what's ripped (everything now). It was useful as I made sure what was ripped and what wasn't after a lightning strike fried my drive with my music and I had to start over from what was on my laptop, and since it's off site, it'll also be useful if I ever have to replace everything after a disaster.

(As far as I know, I haven't lost any data from my collection, and I do keep an eye on it.)

Hope this helps.
posted by immlass at 7:33 AM on May 28

I don't know if these: Ripped & Track Down Purchases will help you. I do recommend browsing around Doug's Scripts to find something that might help you.
posted by hariya at 7:51 AM on May 28

How are you checking that the albums are completely missing? Your examples are compilations and iTunes tends to stick compilations in their own folder, separate from normal albums.
posted by strangecargo at 9:54 AM on May 28

The "assume they're all there" bit doesn't work as it will end up making me lose money. With this many CDs, many of which are compilations, I have to look to see if I own a song before I buy it. I unfortunately can't name all 2300 of my CDs and every track on the.

The tags may work, but I've bought from Napster, Amazon, iTunes, etc. so I'll have to see if there's a pattern for all the tags.

Some of Doug's scripts do look useful in the long term but not this specific issue

And I know the albums are gone because I first looked in iTunes for the song then, when it wasn't found, I did a full hard drive scan for various album and track and artist names, and nothing came up. I also have browsed the Compilations folder and, nope, not there.

I'm thinking with the suggestions above I may use one of Doug's scripts to make a playlist of all my iTunes purchases, and then search the folder for MP3s since everything I ripped would be an M4A. Then just rerip everything (not looking forward to THAT time waster)
posted by arniec at 10:05 AM on May 28

Seconding stragecargo here. I have your set up: mac mini is the server, 2TB external hard drive in a RAID 1 configuration, Apple Lossless + Amazon purchased mp3, over 5000 CDs ripped manually. One day I thought I lost a CD. I had not - stupid iTunes classified it as "compilation" even though it was not, and so never showed up as a folder on the hard drive.

My recommendation for the future: get yourself a very large RAIDed external drive. That way you have some insurance against losing data. I surmise you never backed up your hard drive with all the music? Bad, bad arniec. I would go insane, if I had to re-rip all my CDs and again input/correct wrong metadata, get artwork, etc. Originally, I simply backed up my HDD occasionally, but that got tedious, plus I was constantly adding more music, so it wasn't practical either. RAID came to the rescue. This is not fool-proof, but for practical purposes, it'll do (really, I should also back up the RAIDed drive, but I'm too lazy).
posted by VikingSword at 10:32 AM on May 28

Oh, just to add - whenever I have that problem, I just spotlight search, then uncheck the "part of a compilation" box, and the damn thing counts as an album again. Tedious. If anyone has an automated/script way of doing it, I'm all ears.
posted by VikingSword at 10:34 AM on May 28

Actually it is on a Drobo, which uses a version of RAID. So if the hard drive CRASHED, I was fine. The problem is if a file is deleted, RAID is no protection there.

And yeah I've searched four different ways for these files. They aren't there.
posted by arniec at 11:09 AM on May 28

Good on you for backing up your data! Now having said that, I don't want to pull this thread OT, but Drobo is somewhat controversial. A google search turns up a lot of complaints about data loss. I did consider Drobo, and for these reasons (plus the proprietary nature of the beast) I opted for a conventional RAID external HDD.

Also, if this data loss is an ongoing issue, perhaps it's best to get to the bottom of this. Is it all possible that the loss is hardware related?
posted by VikingSword at 11:20 AM on May 28

To me, it sounds like your main problem is that you've moved from system to system and somehow things *may have* gotten lost along the way. Unless you're planning on pitching all of your CDs, I see no reason to panic. Whenever I've had a song turn up missing, it still showed up in iTunes with an exclamation point (or something) to let me know the original file was missing.

Here's my setup, which I have found to be foolproof.

I have a Mac Mini. Connected to it are a few USB drives, one of which is my iTunes HD. I also have a separate drive as a Backup iTunes HD. Finally, I have another drive set up as my Time Machine HD (and I told Time Machine not to back up my backup drives)

For backups, I use SuperDuper, which I've found to be absolutely superb (not to mention cheaper than it should be!). Every night, SuperDuper clones my drives to their appropriate backup drives (I have a few). The main Mac HD's clone is a bootable backup which turned out to be really handy when I had a HD die last year. And, like I said, I have Time Machine set up too...

Personally, I don't like a RAID setup because, when something goes wrong, it goes really really wrong... everywhere. I prefer my SuperDuper/Time Machine combo as a best of both worlds. Time Machine constantly keeps watch and SuperDuper clones daily.
posted by 2oh1 at 4:32 PM on May 28

Oops! I meant to pass along the link to SuperDuper!, and yes, the exclamation point really is part of the name. This thing is so good, they deserve the exclamation point.
posted by 2oh1 at 4:34 PM on May 28

Do you still have the original collection on the external drive? Here is how I would handle this (and have in the past):

1. Plug in that drive.

2. Open /Applications/Utilities/Terminal.app

3. Run:

$ cd 'Music/iTunes/iTunes Music'
$ find . -type f | sort > ~/Desktop/current-songlist.txt 

4. Now cd to the directory your original songs are at. This will be something like /Volumes/name-of-drive/path/to/directory. When you type "ls" in terminal, you should see the list of artists. Then run:

$ find . -type f | sort > ~/Desktop/original-songlist.diff
$ cd ~/Desktop/
$ diff original-songlist.txt current-songlist.diff > what-changed.txt

5. Take a look at the file "current-songlist.diff". This is a list of files that have changed between the two. Stuff that starts with '<> ' are songs that have been added.

If you did a lot of file renaming, you'll have to get fancier..
posted by cj_ at 5:56 PM on May 28 [1 favorite]

Oh. Another thing you can do is use rsync to copy just the missing files from the original collection to the new one. If you did a lot of renaming though, this will give you a ton of duplicate files.

One other thing you should know about iTunes: It doesn't actually care what's in those directories, it cares what's in iTunes Music Library.plist. This is like a big table of contents for every song it knows about and where it lives on disk. If you manually moved files to a different location instead of using iTunes, it won't know about that, so POOF your song goes away. Except it's still there on disk, the XML file doesn't know where. If you think this is what happened to you, the only thing you can do is just move the files out of the way, delete the plist file so you have a fresh iTunes, and re-import them.

I rather suspect this is your problem rather than mysteriously missing files, because I've been using iTunes for years now and have never 'lost' a single song. I have a python script I wrote to find these orphaned files, memail me if you're interested and I'll send you a link to it.
posted by cj_ at 6:01 PM on May 28

"I have since learned that checking the box to allow iTunes to keep the music folder organized can cause removal of data..."

What? I've never lost anything but if I am at risk for losing my music I want to know. Where did you hear this and what can be done to prevent it? (Other than turning off music folder organization, which I tend to like.)
posted by IndigoRain at 10:05 PM on May 28

Ok, perhaps this wasn't clear. I have checked....the FILES are gone. It's not just that the songs aren't in iTunes. The files are gone, their folders are gone. I have done hard drive scans from both Mac OS and Windows (through a linked drive).

Be it an iTunes issue, a Drobo issue, a "the files didn't move correctly from the Windows PC" issue...whatever, the files are not there. And what I'm trying to do is reconcile the files I do have with a physical CD collection to see what else, if anything, is missing.

I had assumed it was an iTunes issue due to this thread:
http://ask.metafilter.com/121973/is-there-a-song-limit-in-iTunes

where it is referenced that multiple sources have said iTunes with the "Keep music folder organized" option causes this problem. In keeping the music folder organized, it deletes things (probably an attempt to move that doesn't work causing in lost files/folders).

There is more about the iTunes issue here:

http://www.mac-forums.com/forums/os-x-applications-games/137543-keep-music-folder-organized-deleted-all-my-music-my-external-ntfs-hard-drive.html
posted by arniec at 7:03 AM on May 29