How to get a grip on my data? File organisation. Photos. Etc

July 1, 2023 5:26 AM

My personal data is a complete mess. It's spread over multiple external hard drives and devices, a Windows PC, iCloud, various other cloud services, many manual backups over decades. There's a lot of confusing duplication and uncertainty about whether some things are backed up at all... The photos are particularly bad. How can I get started organising all this, removing duplicates, checking that cloud services are on my local machine too, and so on? I'm fairly techie (can use Linux tools if needed) but very disorganised.

iCloud is a big question-mark for me too. I'm not sure my desktop software is working right, the automatic download folder seems off. And might be duplicated in various places...

Argh. Help?

posted by d288478 to Technology (11 answers total)

Response by poster: Another aspect of this is that I'd like to get my data from these services and then stop using them altogether. But if I just download that data into another folder I'll only create even more duplication. At the same time, I can't assume I already have a local copy of each file because of... chaos.

- Dropbox
- Evernote
- Google Photos
- OneDrive
posted by d288478 at 5:45 AM on July 1, 2023

Best answer: Do you have the option to get some network attached storage? I'd start there, or at least get one specific place where you put your backups, like an existing hard drive with plenty of space. If it has some data on it right now, try to organize it in a basic way with fewer than ten folders. Photos, Videos, Music, Documents might be enough. Now of course you can have sub folders but just having one place that you copy every photo to can give you piece of mind. You also want to have a third copy of that somewhere.

Software like CloneSpy can help you find and remove duplicates. Software like Synctoy can help you copy things over from one folder to another by using the Contribute option. That way you can make sure all files that were spread across folders A and B are in B but not add any duplicates of files that were in both.

If I have a ton of one type of file, I tend to parcel them out alphabetically even more. Like I have an ABC folder and a DEF folder. It helps me work on a smaller chunk of files at a time when looking through for duplicates or renaming, for example. You can just dump all of your photos into one big folder and then work on subfolders over time. One trip can be a folder, but also Fall 2009 where you just pull everything from Sept-Nov 2009 together.
posted by soelo at 6:11 AM on July 1, 2023

Best answer: I did this by moving files directly to a large drive for backup, not to my current laptop. I consolidated from several old drives and laptops, etc. I wish I'd used a de-duplicater. There are a few, no idea which is best.

I organize personal data files by year. I try to organize photos by year. Music is separated by holiday, music, audio(podcasts, etc.), then by artist. It took ages; I watched (streaming) tv and movies while I did it. It's on a hard drive in a fireproof box as well as on cloud storage.
posted by theora55 at 8:47 AM on July 1, 2023 [1 favorite]

There is good advice here, but I would advise you in the long run to NOT abandon cloud storage. I tend to get preachy about this, but my philosophy is that nothing is safe until you have three copies of it: on one hard drive (usually the one in your machine, but that's not always possible, depending on quantity), at least one other copy locally (one or two external drives) and one remote (which usually means cloud storage). I haven't achieved perfection here, except for my most critical files. It's questionable whether all your data needs that much care, but you have to do the work to figure that out, which can indeed seem overwhelming.

I notice that you mention iCloud as one of your current cloud options, so I assume you are using a Mac. Some other options mentioned here are exclusive to Windows, but there are usually Mac options.

As far as cloud storage goes, perhaps decide on one and move on from there. In my experience, iCloud Drive is usable on Windows machines, but works best on Macs, where it can easily sync and provides backups for iDevices. I'm using iCloud exclusively right now. Dropbox works everywhere, but was a terrible resource hog on my Mac. OneDrive seems OK either way (and my wife pays for the family edition of Microsoft 365, so that helps). Your mileage, and that of other Mefites, will vary of course.

One thing I've never managed to do is de-duplicate everything, and decided long ago that perhaps it's not worth the effort, unless files are large or there are a LOT of copies of something. I have used actual duplicate-detecting software (part of the MacCleaner Pro suite on Mac, Duplicate Cleaner Pro when I was on Windows) and had great satisfaction finding complete folders that were dupes on the same device and simply removing all but one of them, then working down by size for everything else.

I also intentionally duplicate some things on the same device. I use Apple Photos, which of course is copies to iCloud, but I also keep exported copies of my pictures nearby (on the same drive) because I don't entirely trust Photos to protect them.
posted by lhauser at 1:20 PM on July 1, 2023 [2 favorites]

I'm fairly techie (can use Linux tools if needed)

git-annex
posted by Bangaioh at 3:54 PM on July 1, 2023 [1 favorite]

Best answer: A couple of months ago a Metafilter comment mentioned Johnny Decimal and it has been AMAZING. I use 3 versions on my personal laptop, another on my work shared files (files on my Onedrive, outlook folders and hard drive all have the same structure) and in my actual house to inventory/tidy rooms.

I had a hideously messy laptop with data from multiple computers and cloud services. I went down to just one main cloud service and a backup that mirrors. I can find EVERYTHING. Now once a week, I clear out my inboxes and download folders to the matching folders and it’s so easy. Life changing.

Re: photos, I used an app to search for duplicates and then to separate out scans which I went through as paperwork and mass converted to PDFs, and for the rest, google photos does a decent job auto creating albums on face recognition and time, so I throw everything there and don’t aim for 100%, just 90% organised.
posted by dorothyisunderwood at 7:01 PM on July 1, 2023

Also I know I have duplicates still but I’m fine having duplicates in the same folder, e.g. 23.04 Plans for renovation 2021 has multiple folders from that project which probably have duplicates in them, but more importantly I know where my renovation files are.
posted by dorothyisunderwood at 7:03 PM on July 1, 2023

+1 for getting a NAS. The following isn't cheap but it covers all the bases. I'm gonna be saying NAS a lot.
Multiple external drives --> put it all on the NAS, stop using drives (can be used for extra backup, maybe)
Mac/PC/Linux --> move files to NAS. Also run the NAS'es agent locally on each device to backup the whole thing
Cloud Data -> use NAS backup agent to pull down an extra copy for safekeeping
Photos --> Dedupe first as others suggested, I had great success with DigiKam (mac opensource), then ... NAS
Backup --> Extra cloud which your Network Attached Storage is then backed up to
posted by yoHighness at 1:39 PM on July 3, 2023

PS I don't use any more than RAID 0 on my NAS, that's what the cloud backup is for - save that money.
posted by yoHighness at 1:44 PM on July 3, 2023

Figure out *why* you want it to be organized first, and when you've got it "good enough" to give you enough of that goal, stop. It is possible to organize damn well forever, but that's not a great way to spend the time. ;-)
posted by talldean at 6:43 PM on July 12, 2023

Response by poster: Thank you for all these helpful answers. I’ve now consolidated my data onto a single large drive (of which I have backups) and have been using Clonespy and Winmerge to reduce some duplication.
posted by d288478 at 4:00 PM on August 22, 2023