Tips and techniques for organizing my digital life
April 8, 2014 5:24 AM Subscribe
My desktop PC (running Windows 7) contains four hard drives, with information dating back to 1984. How can I organize that information, de-duplicate it, and archive it in some kind of logical manner?
I'm a digital pack-rat. I've saved every term paper, application essay, and love letter I ever wrote. I also have lots and lots of digital photographs and scans of 35mm negatives. I'd like to take all these old files and archive them on Google Drive (which is pretty cheap now), so that I can search them and retrieve them easily. I have at least three problems I need to overcome:
1. Many of the documents are in outdated formats (e.g., Microsoft Word files from circa 1985), although it's probably possible to find utilities to convert them to PDF or plain text or whatever. I want to archive the files in some kind of format that will still be readable X years from now.
2. The files are scattered across many folders and four hard drives, with no rhyme or reason
3. There are many, many duplicates, usually stemming from past abortive attempts at either organizing the files or creating back-ups.
This task is so overwhelming in its complexity and scope that I don't even know where to start. Is there some kind of overall strategy, or battle plan, that I can follow here? I've already started looking at utilities that de-duplicate files, but they almost seem like they're more trouble than they're worth. They don't look like they're well-suited to identifying clusters of duplicated files that are present in multiple locations. Or maybe I just haven't delved into these programs enough.
I know it's a huge task, but my plan is to spend 15 minutes a day, for as long as it takes, to work on this project. This to-do item has been weighing on me for years, and I need to get it done.
I'm a digital pack-rat. I've saved every term paper, application essay, and love letter I ever wrote. I also have lots and lots of digital photographs and scans of 35mm negatives. I'd like to take all these old files and archive them on Google Drive (which is pretty cheap now), so that I can search them and retrieve them easily. I have at least three problems I need to overcome:
1. Many of the documents are in outdated formats (e.g., Microsoft Word files from circa 1985), although it's probably possible to find utilities to convert them to PDF or plain text or whatever. I want to archive the files in some kind of format that will still be readable X years from now.
2. The files are scattered across many folders and four hard drives, with no rhyme or reason
3. There are many, many duplicates, usually stemming from past abortive attempts at either organizing the files or creating back-ups.
This task is so overwhelming in its complexity and scope that I don't even know where to start. Is there some kind of overall strategy, or battle plan, that I can follow here? I've already started looking at utilities that de-duplicate files, but they almost seem like they're more trouble than they're worth. They don't look like they're well-suited to identifying clusters of duplicated files that are present in multiple locations. Or maybe I just haven't delved into these programs enough.
I know it's a huge task, but my plan is to spend 15 minutes a day, for as long as it takes, to work on this project. This to-do item has been weighing on me for years, and I need to get it done.
Do you have backup copies of your old files? If not, creating those is your first order of business. A big, modern, external drive will probably hold all of that.
posted by thelonius at 5:43 AM on April 8, 2014
posted by thelonius at 5:43 AM on April 8, 2014
Response by poster: @Thelonius: I use Backblaze (cloud app, similar to Carbonite) to back-up everything on all my drives.
posted by alex1965 at 5:48 AM on April 8, 2014
posted by alex1965 at 5:48 AM on April 8, 2014
ok great! I get nervous around old drives with important personal files
posted by thelonius at 5:57 AM on April 8, 2014
posted by thelonius at 5:57 AM on April 8, 2014
Best answer: I'd start by getting a duplicate file remover and running that on each drive (or across all the drives, if the program supports it). I've used Duplicate Cleaner in the past, but there are a million programs that do the same thing.
Run this first, and then you'll at least get rid of the backups of backups of backups that you've inevitably made, reducing the amount of crap you have to plow through. Sadly, I think the only solution for the rest of it is just deciding how you want to file things and then doing it, little by little.
posted by MeghanC at 6:02 AM on April 8, 2014 [1 favorite]
Run this first, and then you'll at least get rid of the backups of backups of backups that you've inevitably made, reducing the amount of crap you have to plow through. Sadly, I think the only solution for the rest of it is just deciding how you want to file things and then doing it, little by little.
posted by MeghanC at 6:02 AM on April 8, 2014 [1 favorite]
I just went through this and it took me a month (mostly because before deleting something I verified that it was stored offsite). I got a NAS and little by little dumped everything on there. The exception was photos that I keep on a USB3 hdd (and then are backed up onto the NAS + offsite storage).
I did all of the filing manually with the exception of photos. With photos, I imported whole old hard drives into a photo organization program that detected duplicates. (I used Iphoto).
I started with the photos (cause that was the impetus really) but that cleared up a lot of space. Then I dumped everything else (whole old harddrives) onto the NAS and started filing/deleting knowing the the original's where tucked away.
(some old HDDs I just left on the NAS and only sucked up the photos from them. They are there if I need them and if not, relatively speaking, they are not much room.)
Second using a treemap viewer to find and delete stuff...it'll make the off site backup part quicker.
posted by Spumante at 6:19 AM on April 8, 2014
I did all of the filing manually with the exception of photos. With photos, I imported whole old hard drives into a photo organization program that detected duplicates. (I used Iphoto).
I started with the photos (cause that was the impetus really) but that cleared up a lot of space. Then I dumped everything else (whole old harddrives) onto the NAS and started filing/deleting knowing the the original's where tucked away.
(some old HDDs I just left on the NAS and only sucked up the photos from them. They are there if I need them and if not, relatively speaking, they are not much room.)
Second using a treemap viewer to find and delete stuff...it'll make the off site backup part quicker.
posted by Spumante at 6:19 AM on April 8, 2014
Not answering the question, but I would do a basic organization and consolidation of the files and use desktop search to find what you want. Unless you're looking to make all these files mobile, the built-in desktop search and local folders will do a much better job than Google Drive at searching and retrieving your files.
You also have no guarantee that Google will continue support your Word 1.0 (or 2.0 or whatever) files. You could convert them all to Google Drive format, but that's going to ruin the document formatting, and it assumes that Google Drive will exist in 10 years. It probably will, but Google has been known to kill products for no good reason, and you can't download an installable copy of the software for later. Windows 8.1 will still work in 10 years, and you'll be able to install a version of Microsoft Office on it that will open those files.
posted by cnc at 9:36 AM on April 8, 2014
You also have no guarantee that Google will continue support your Word 1.0 (or 2.0 or whatever) files. You could convert them all to Google Drive format, but that's going to ruin the document formatting, and it assumes that Google Drive will exist in 10 years. It probably will, but Google has been known to kill products for no good reason, and you can't download an installable copy of the software for later. Windows 8.1 will still work in 10 years, and you'll be able to install a version of Microsoft Office on it that will open those files.
posted by cnc at 9:36 AM on April 8, 2014
Best answer: Buy a huge harddrive to consolidate it all, a 2 tb is like $100.
Work out how many different file types you're dealing with (probably 10-15?)
Each night extract all of the files of one type (.doc, .jpg etc) from your multiple hds with seach queries and slap them into a single folder for that doc type on your big drive. This is now a chronological record of all files of that type, which you should be able to de-dupe fairly easily and access by date range or search.
It also makes it easier to do batch conversions of outdated word processor files because they'll all be grouped by date, probably.
Also Windows search is pretty good these days, check out the document and picture library functions - by default they organise everything by date.
posted by Sebmojo at 8:30 PM on April 8, 2014 [1 favorite]
Work out how many different file types you're dealing with (probably 10-15?)
Each night extract all of the files of one type (.doc, .jpg etc) from your multiple hds with seach queries and slap them into a single folder for that doc type on your big drive. This is now a chronological record of all files of that type, which you should be able to de-dupe fairly easily and access by date range or search.
It also makes it easier to do batch conversions of outdated word processor files because they'll all be grouped by date, probably.
Also Windows search is pretty good these days, check out the document and picture library functions - by default they organise everything by date.
posted by Sebmojo at 8:30 PM on April 8, 2014 [1 favorite]
Best answer: Heh, this sounds like something me from a previous life would try to do. Not that there's anything wrong with organization or preserving your files in useful formats, but I would recommend that you take a step back and think about what your goals and needs are for this reorganization. How often do you need to refer back to these files? Do they really need to be organized a certain way, or can you use desktop search (possibly adding folders to the index if you're concerned about search speed) to find what you need? Would setting up libraries for your photos or using a photo organizer like Picasa (or an alternative) do what you need for indexing your photos?
Next, plan out what you want success to look like. If you don't know that from the beginning, you're going to rathole yourself into a neverending project like you seem to have tried before. Plan out whatever folder structure or system you want to use going forward and start using it as soon as possible for all of your new stuff. A system that you don't use is not going to be worth the time and effort to create. Don't discount the solution of just dumping all of your old stuff into a folder and pulling files and such out of it into your new system as you need them. If you haven't needed the file yet at all, what makes you think you'll definitely need the file to be organized correctly?
Break the problem into smaller tasks, and keep an eye on ways to automate them. Computers are great for boring, repetitive tasks, so make them do as many of them as possible. Others have suggested some tools for the specific problems of deduping and batch-converting files into newer formats already, so I won't go into those.
Anyway, good luck getting organized.
posted by Aleyn at 12:53 AM on April 9, 2014 [2 favorites]
Next, plan out what you want success to look like. If you don't know that from the beginning, you're going to rathole yourself into a neverending project like you seem to have tried before. Plan out whatever folder structure or system you want to use going forward and start using it as soon as possible for all of your new stuff. A system that you don't use is not going to be worth the time and effort to create. Don't discount the solution of just dumping all of your old stuff into a folder and pulling files and such out of it into your new system as you need them. If you haven't needed the file yet at all, what makes you think you'll definitely need the file to be organized correctly?
Break the problem into smaller tasks, and keep an eye on ways to automate them. Computers are great for boring, repetitive tasks, so make them do as many of them as possible. Others have suggested some tools for the specific problems of deduping and batch-converting files into newer formats already, so I won't go into those.
Anyway, good luck getting organized.
posted by Aleyn at 12:53 AM on April 9, 2014 [2 favorites]
« Older Supporting a grieving partner - what do you wish... | "Your server IP address is in the SpamCop database... Newer »
This thread is closed to new comments.
Use something like SpaceMonger (a treemap viewer) to get a general view of what your filing structure was like over previous years, because there probably was a rhyme and reason if you can see it, which will make it easier to sort existing folders into a larger archival structure (your new system). SpaceMonger will be initially best for larger files, like photos and videos, but you can zoom in.
Initial sorting: I prefer sorting folders by type, then by date.
So, all photos, with subfolders prefaced by the date & descriptor.
All documents, by date, possibly with subfolders divided by particular topic.
Reference material: by topic.
Moving onto google drive also gives you the option of tagging by topic, but just drag it into some kind of hierarchical structure first.
posted by Elysum at 5:42 AM on April 8, 2014