Tools for consolidating folders on Windows 10?

March 10, 2024 5:09 AM

After a few years of cloud syncing problems and short-sighted workarounds, I've got several mirrored versions of my documents folders + nested subfolders across multiple hard drives and thumb drives, where most of the files are identical but each folder may contain a few unique items or updated copies of the same file found elsewhere. Is there a more efficient way to merge these, purging exact dupes but keeping all unique or updated files?

I've tried to merge by brute-force copy-pasting individual folders into each other, which in Windows does bring up a little dialogue listing duplicate file names and asking which to keep. However, (a) that's complicated in instances where I was especially stupid and created new subfolders in just one of the libraries, or where I moved a file to a different location in just one of the libraries, and (b) that still requires me to work down the list of hundreds of same-name files to visually spot instances where I need to keep the updated version.

I'd really love a tool or workflow that just looks through a big set of folders, detects all the duplicate files located anywhere in the library (but not name-duplicates with a different date or file size), and one-click removes one of the copies, I don't even care which. Does this exist? Any other recommendations for accomplishing this task efficiently?

posted by Bardolph to Computers & Internet (8 answers total)

jdupes can do what you want, but it is command line only.

jdupes --recurse --delete c:\somefolder c:\someotherfolder

Will prompt you about every duplicate, or you can add

--noprompt

to delete 2nd copies automatically.

I would love a nice GUI that does similar but haven't found one yet.
posted by samj at 5:49 AM on March 10 [2 favorites]

@samj I built one to do this a long time ago, let me try to find it.
posted by SNACKeR at 6:14 AM on March 10 [1 favorite]

I did this recently, winnowing down multiple partially-duplicative backups of old laptops dating back to 1996.

I had good luck with DupeGuru, which has a GUI. It has a focus on safety (never deleting the only copy).

Alternatively, Beyond Compare is a venerable tool with a lot of configurability.

Both work on folders as well as files.

Two thoughts:

- are you sure you don't care which copy is deleted? If you have made lots of small changes in different branches of the tree, then unique files which logically belong together will end up scattered and hard to find. I would want to end up with a single cohesive folder hierarchy.

- if the data is important, consider whether you fully trust yourself (and any tools) to never make a mistake. I decided I didn't — so before doing any de-duplicating, I zipped the whole folder tree and uploaded it with rclone to AWS Glacier Deep Archive, where storage costs ~12 cents per 10GiB per year.
posted by Klipspringer at 6:17 AM on March 10 [1 favorite]

CloneSpy is what I use for this. It has options to delete the oldest or newest copy, move files instead of deleting, and one to ask you each time it finds a duplicate.
posted by soelo at 7:16 AM on March 10 [1 favorite]

The application I use for doing this is here: finddupes2

Note, it will only delete the files you select, so you can select one, both, or none.
posted by SNACKeR at 7:33 AM on March 10

I would use Czkawka, which is an open source GUI option. It has a lot of variables to tweak.

Linking to that, I see they even have a little table helpfully comparing other software options.
posted by Snijglau at 8:33 AM on March 10

Unison synchronizes two directories, with a UI that lets you choose what to do with conflicting files.

So not exactly what you’re looking for, but if you designate one directory the master and then sync the others in succession, it would work.

When comparing differing files, don’t trust the timestamps.
posted by qxntpqbbbqxl at 10:18 AM on March 10 [1 favorite]

Seconding Beyond Compare. For all it does it is inexpensive and irreplaceable.
posted by lhauser at 5:38 PM on March 10