Baby got too much back

March 2, 2009 8:53 AM

How can I consolidate all my pictures, music, pdf files, etc. from multiple drives? I have about 10 external drives containing photos and so forth. Different drives contain some duplicates, but the individual drives do not.

Say I want to copy the files from DRIVE1, DRIVE2, etc. to UBERDRIVE, which has room for everything. My first thought was

cd /Volumes/DRIVE1; find . -name "*.jpg" -exec cp {} /Volumes/UBERDRIVE/Pictures/ \;

and then repeating the process for DRIVE2 and so on.

I tried this for DRIVE1, and it seems to be really slow. (It's taken 10 hours thus far to find 100,000 jpgs on a new MacBook Pro.) Also, I think my current procedure will overwrite files with the same name, even if they're different files.

Does anyone have a better approach? Thanks!

I'm running Leopard, but I also have machines running Vista, XP, and Fedora 10.

posted by lukemeister to computers & internet (5 answers total)

1. Look at rsync (it comes with osx wrappers)
2. When dealing with files like this, quote the {} in case you have embedded spaces
3. File ops over 100,000 files will be slow regardless of how you do it.
4. Your process will also flatten any subdirectories (and it's an unusually bad idea to put 100,000 files into the same directory), and overwrite files with the same name. See point #1.
posted by devbrain at 9:05 AM on March 2, 2009

Rsync is a better solution. You can use it rather than cp to copy the files and it won't re-copy the same file twice. Unfortunately, you still have to copy the files. You could try (depending on free space and file size) putting the files into a tarball before copying them, this will reduce disk seek time. Also defragging your drive can speed up operations like this if fragmentation is increasing seek time.
posted by doteatop at 9:06 AM on March 2, 2009

No reason to get all tricky about it. Just option-drag the icon for each disk onto your new drive. You'll end up with a folder for each disk.

If, for some reason, you want to get put all your pictures or music in one flat directory, just press command-F while you have your big drive selected, search it for all files whose kind is "Images," and drag them to a new folder. The Finder will ask whether you want to replace any images with the same filename. You can click "Don't Replace" and the items with duplicate filenames will be left behind for you to manually sort through.

Alternatively, you can use a utility like Tidy Up! or File Buddy to find your duplicates.

Or you can drag everything onto iTunes and iPhoto, and let them consolidate your files.
posted by designbot at 12:03 PM on March 2, 2009

iTunes / iPhoto might be your best bet for a lot of this. At the very least they will handle duplicate files better than most other methods, are good at consolidating but still should organize things into logical directory trees.

After you're done, you can move the folder tree (music in iTunes, or original folder list from within iPhoto.app) to wherever you want, and delete the iTunes/iPhoto library files if you don't wish to use those apps for anything.

The caveat here is that iPhoto seems to freak and move incredibly slowly when I throw several dozen photos at it. I'm not sure how well it will handle pulling in 10k+ photos. Don't expect it to work very quickly, and unless newer versions have fixed this when it finds duplicates I believe it prompts for action every single time...
posted by caution live frogs at 3:26 PM on March 2, 2009

Thanks, everyone. I should clarify that I don't have room on the Uberdrive to copy every file on drives1 ... 10, just room for the images, pdfs, and so forth. The comments on putting huge numbers of files in one directory all well-taken. I need to rethink my plan.
posted by lukemeister at 9:32 AM on March 4, 2009