Merging folders in OS X

March 8, 2009 12:49 AM

Please help me merge two folders on my mac.

Two mp3 folders. One internal, one external. Many duplicates. Many unique files.
What's the simplest way of merging them on the external drive without losing anything?
Googling has led me to the Terminal's mv command, which seems to be the simplest solution. I was just wary because of the delicate nature of the operation. Can someone post an example of the exact syntax I would need to use?
Or perhaps another (free) solution?
Also, I'm sure some of my files have been updated on my mac. Can I also make it so newer files replace older ones?
Many thanks in advance.

posted by Silky Slim to Computers & Internet (22 answers total)

I would recommend copying using cp instead of moving, at first - this reduces the chance of losing data. Both the cp and mv commands support the '-i' flag, which will present you with a yes/no prompt before overwriting files. I would recommend using that as well. The following will suffice if your folders just contain mp3s without any subdirectories:

cp -i /path/to/source/folder/*.mp3 /path/to/destination/folder/ 

posted by jordanlewis at 1:03 AM on March 8, 2009

Response by poster: Alas, there are many, many sub-directories.
posted by Silky Slim at 1:06 AM on March 8, 2009

Hm. Are any files with the same name guaranteed to contain the same data, or newer data on one side only? That is, is it OK to simply copy things over by name, replacing files when their names collide? Or does the process need to be aware of the files' contents?

If the former, either tar or rsync would be my (unix-head) tools of choice.
posted by hattifattener at 1:17 AM on March 8, 2009

(Though, adding -r to the cp command might also do the trick.)
posted by hattifattener at 1:19 AM on March 8, 2009

Response by poster: It wouldn't be that bad to replace files with the same name, provided they're in the same directory on both sides, of course (I wouldn't want one file named "01.mp3" replacing an unrelated file of the same name). But I would prefer to have it replace with the newer version by default, or by directory of origin.
My problem seems so common and simple... Surely this can be done easily in Terminal... Right??
posted by Silky Slim at 1:27 AM on March 8, 2009

That's a little bit more tricky. If it's organized like Music/Artist/Album/song.mp3, I might suggest a multi-tiered approach: first run

mv -n /path/to/source/* /path/to/dest/

to take care of all of the artists that only the source has. The -n flag means "never overwrite an existing file". Next, we'll try to take care of unique albums:

for artist in /path/to/source/*; do 
    mv -n /path/to/source/$artist/* /path/to/dest/$artist/;
done

This will move all albums that only the source has to their respective artist directories in the destination.

Finally, we can move all mp3s that only the source has to the dest:

for artist in /path/to/source/*; do
    for album in /path/to/source/$artist/*; do
        mv -n /path/to/source/$artist/$album/* /path/to/dest/$artist/$album/;
    done
done

You might want to verify that this works before running it by replacing instances of 'mv -n' with 'echo' - echo will simply print out its arguments, allowing you to see what will happen before it actually does.
posted by jordanlewis at 1:28 AM on March 8, 2009

hattifattener is right. Use rsync.
posted by jordanlewis at 1:29 AM on March 8, 2009

Rsync is reasonably straightforward. But have a play around with some practice folders first. It is quite possible to seriously ruin your day with rsync!

Something like one of these should do the trick:
rsync -avx /src/foo /dest
rsync -avx /src/foo/ /dest/foo
posted by jonesor at 3:31 AM on March 8, 2009

Response by poster: Yikes
I'll try these tips later, after a stiff drink. Thanks.
posted by Silky Slim at 4:22 AM on March 8, 2009

Rsync without question.

n.b. the -r switch will do run it recursively so as to take subdirectories too.
posted by The Michael The at 6:19 AM on March 8, 2009

Throwing in another vote for rsync.

To be completely safe, do something like this:

rsync -avx /path/to/mp3dir/1 /path/to mp3dir/2 /path/to/NEWMP3Dir --progress

This will merge the two directories into a third, leaving your original ones intact. (And --progress is just a file by file progressmeter. It adds some overhead, fyi)
posted by Cat Pie Hurts at 7:25 AM on March 8, 2009

Best answer: I use a little app called SyncTwoFolders. It's easy as can be and works great. (Don't be put off by the French on the download page, the app works in English too. Just select your language at install).
posted by LuckySeven~ at 7:41 AM on March 8, 2009

You have two fundamentally different problems here : merging the directories and eliminating duplicates. iTunes itself will merge both your directories under it's own organizational scheme unless you've disabled "Copy Files to iTunes Library". Yes I also suggest rsync too if you've disabled iTunes organization.

I wrote a little perl script to find likely duplicates by md5 hash, but you must change #!/opt/local/bin/perl to #!/usr/bin/perl, and ignore the -h for sort by hamming distance which I never implemented.

If my script say two files are identical, then check if the names claim they are the same song. If so, you're safe deleting one. If not, I'd look into those mp3s more carefully, chances are you have miss-named mp3.
posted by jeffburdges at 8:02 AM on March 8, 2009

Holy crap.

cp -ri /path/to/source/folder/*.mp3 /path/to/destination/folder/


You just have to use the -r to recursive copy. Holy crap dudes, OSX is just BSD with a new dress on.
posted by judge.mentok.the.mindtaker at 8:26 AM on March 8, 2009 [1 favorite]

oh i had this thread open for *way* too long before i commented
posted by judge.mentok.the.mindtaker at 8:35 AM on March 8, 2009

Yes, rsync is your answer. If you are uncomfortable with the command-line, there are a number of GUI applications that can do this for you. Synk Pro. Chronosync.
posted by mrbarrett.com at 8:38 AM on March 8, 2009

Holy crap dudes, OSX is just BSD with a new dress on.

Sure, but I think a lot of modern Mac users still expect the seamless and nearly impenetrable bodysuit that was Classic.

Besides, rsync is nice because you can get deltas and in-copy status (especially good with large transfers).
posted by Cat Pie Hurts at 8:47 AM on March 8, 2009

I'm not an OS X user, but what happens if you just grab the music folder in the GUI and drop it onto the external drive? Won't it do what you want automatically, asking you questions along the way if there are conflicts?
posted by gjc at 10:29 AM on March 8, 2009

gcj - No. OSX is annoying and stupid in this regard. Let's say you drop a tree with empty subdirectories on top of a tree with the same structure, but filled with files, rather than doing the logical thing and merge the two, it will actually -replace- the filled structure with the empty set. Ridiculous. (To be fair, you will be asked when you want to replace the files, and it is undoable, but it's still a ridiculous default action, imo)
posted by Cat Pie Hurts at 11:00 AM on March 8, 2009

(sorry - gjc :)
posted by Cat Pie Hurts at 11:00 AM on March 8, 2009

gcj - No. OSX is annoying and stupid in this regard. Let's say you drop a tree with empty subdirectories on top of a tree with the same structure, but filled with files, rather than doing the logical thing and merge the two, it will actually -replace- the filled structure with the empty set. Ridiculous.

I'm pretty sure this has been argued to death here and all over the internet, but not everyone thinks it's ridiculous. Overwrite should overwrite by default, in every instance, and fancy merging should be possible but not assumed.
posted by jragon at 11:24 AM on March 8, 2009 [1 favorite]

Going a little bit off of what jeffburdges said, iTunes can potentially take care of this, depending on how you really want to do it. Just have iTunes organize the music automatically (so that all the music is stored in one location), and then go to File->Show Duplicates in iTunes. Then you will have a list of all duplicate songs and you can fix it up as you see fit.

Just throwing that idea into the mix.
posted by kosmonaut at 11:55 AM on March 8, 2009