Need a system for organizing electronic files
July 19, 2006 2:41 PM
Need to clean out and organize 15 years of electronic files.
We are a small (8 person) design firm. We've been in business for about 15 years, and my boss would like me to go through our "Office Admin" folder on the server, get rid of useless documents and organize the stuff we're keeping. We currently have a folder for "Projects" and "Images" which seem well organized, but the "Office Admin" has been a grab bag of stuff, and I'd like to try to implement a system that would make sense to everyone and not just to me.
We are a small (8 person) design firm. We've been in business for about 15 years, and my boss would like me to go through our "Office Admin" folder on the server, get rid of useless documents and organize the stuff we're keeping. We currently have a folder for "Projects" and "Images" which seem well organized, but the "Office Admin" has been a grab bag of stuff, and I'd like to try to implement a system that would make sense to everyone and not just to me.
is there any way you could apply tags to each item, and let the users browse the system contents according to their own mental model?
'course the tagging involves some work, but it certainly sounds like you have plenty of work ahead of you anyway.
posted by diastematic at 2:51 PM on July 19, 2006
'course the tagging involves some work, but it certainly sounds like you have plenty of work ahead of you anyway.
posted by diastematic at 2:51 PM on July 19, 2006
I recently did something similiar when I had to clean up our file server. I did a search for all files that hadn't been accessed in over 2 years (atime), and copied those off onto a set of backup tapes, a set of CD's and to an external hard drive and then put them on a shelf.
It really wasn't worth it for me to sort through thousands of files and pick out the wheat from the chaff. If no one's looked at in years, what is the point of sorting it? Once you've moved off all the old/unused stuff, it might be easier to sort the remainder.
If you ever do need to find something in the old stuff, searching might be a better solution then categorization. Just make sure if you make archive copies that you preserve the file creation or modification dates - those really help narrow down searches.
If you have to sort everything because your boss wants you to, then never mind what I said.
posted by voidcontext at 2:57 PM on July 19, 2006
It really wasn't worth it for me to sort through thousands of files and pick out the wheat from the chaff. If no one's looked at in years, what is the point of sorting it? Once you've moved off all the old/unused stuff, it might be easier to sort the remainder.
If you ever do need to find something in the old stuff, searching might be a better solution then categorization. Just make sure if you make archive copies that you preserve the file creation or modification dates - those really help narrow down searches.
If you have to sort everything because your boss wants you to, then never mind what I said.
posted by voidcontext at 2:57 PM on July 19, 2006
Also, I'd advise against actually trashing anything. Storage is pretty cheap, and the time you spend debating whether or not to throw something out is probably worth more than an extra couple hard drives.
posted by voidcontext at 3:00 PM on July 19, 2006
posted by voidcontext at 3:00 PM on July 19, 2006
The smart thing to do is 'as little as possible'.
Only live and future projects *really* need to be organized.
The rest of the stuff can be put in the electronic attic. Create a folder called 'Old filing system' and put everything which isn't live in there.
Then look at the current stuff and work out the new system from there.
Trust me, time spent organizing old files which are seldom looked at is time wasted. It is better to accept that occasionally something will be hard to find than to waste time trying to organize everything.
I strongly suggest using something like Apple's Spotlight technology or whatever the Windoze equivalent is to find stuff in the attic when you have to.
posted by unSane at 3:01 PM on July 19, 2006
Only live and future projects *really* need to be organized.
The rest of the stuff can be put in the electronic attic. Create a folder called 'Old filing system' and put everything which isn't live in there.
Then look at the current stuff and work out the new system from there.
Trust me, time spent organizing old files which are seldom looked at is time wasted. It is better to accept that occasionally something will be hard to find than to waste time trying to organize everything.
I strongly suggest using something like Apple's Spotlight technology or whatever the Windoze equivalent is to find stuff in the attic when you have to.
posted by unSane at 3:01 PM on July 19, 2006
I disagree with advice to "keep everything, because storage is cheap and people are lazy." There is a good business case for deleting old files, and destroying old records. The first thing many lawyers do routinely these day is subpoena the full contents of all company servers, and there is no point in having to explain outdated and perhaps even erroneous material. Every company should have and stick to a records and file retention policy, as part of corporate governance. Ask anybody you meet who used to work for Arthur Andersen if this isn't good advice.
The first thing to do, if you really have files going back to 1990 or 1991, is to see if you have space on the server for inspecting and manipulating the files. I've seen old servers where there had been several attempts at "cleanup" which amounted to making compressed archives of everything in order to free space, and which were then allowed to be filled with additional material to the point they didn't have enough file system space to actually manipulate the compressed archives. Resolve those issues, make and fully verify backups (that is, do an actual test restore from backup media if you don't do this routinely as a maintenance check) and do a full set of file system optimizations, if they are not already part of your regular maintenance.
Next, characterize your files as to space, quantity, age, owner, date of last access, date of last revision, and what ever other attributes your file system metadata can provide. Depending on your file server OS, there may be administrative utilities that can make this process pretty simple, or you may have to find suitable utilities that are workstation based.
Next, examine your filesystem authorizations, and see what files may belong to persons or groups no longer authorized on the system. Files that belong to no one any more are pretty common, and I've seen some servers where previous administrators had locks on hundreds of megs of install files for old versions of software that they never bothered to delete or re-assign on the day they left the organization. If you have any appreciable amount of space tied up in these black holes, you need to develop a strategy for evaluation and deletion, if appropriate.
Next, figure out a short list of department heads who can help you figure out what to do with questionable items, and get their agreement to do so. Spreading the load, a little, can improve the utility of the cleaned up system quite a bit, if you don't let the project drop into a "save everything" mentality.
Next, figure out what kind of organization you'd like to have when cleanup is complete. Everything you leave has to fit into that schema, and you should document it, for the person who will be doing this again, in 5, 10, or 15 years. Generally, organization by department or function is likely to be the most relevant and long lasting way of organizing a general purpose file server, as it tends to line up directory structures with group rights and individual job responsibilities. Thus, in your top level Office Admin volume, you'd probably have Accounting, Sales, and Administration sub-directories, along with, perhaps, one or more company wide General sub-directories, for interdepartmental file sharing, and storage of company wide forms.
Next, produce a paper inventory of the system's pre-cleanup contents. There's nothing like having a master list of everything you started with, complete with path and metadata, when you're in the middle of deleting and re-organizing things, and a user comes in and swears you've nuked the revised-Eastern-sales-budget-approved-by-God, when it was never on the server or any back up tapes in the first place. The last cleanup of a medium sized file server I did, this inventory ran to 480 double sided pages, and over a half million files. But in the weeks following the clean up, it saved my job at least 16 times.
Next, with your preparations complete, produce a candidate deletion list, and get any approvals needed. Do a backup, or archive the files to be deleted to media, and perform the deletions. Create any new directories, groups, or other organizational elements, and move remaining files appropriately. Have users review results in a definite go/no go acceptance/rejection time window, during which they neither create or delete additional items (may be useful to restrict all logins to browse authority during this period). Restore any inadvertent deletions from backup, as needed, until acceptance is forthcoming. When the users have accepted the clean up results, perform a complete backup and return the server to normal operation.
posted by paulsc at 3:48 PM on July 19, 2006
The first thing to do, if you really have files going back to 1990 or 1991, is to see if you have space on the server for inspecting and manipulating the files. I've seen old servers where there had been several attempts at "cleanup" which amounted to making compressed archives of everything in order to free space, and which were then allowed to be filled with additional material to the point they didn't have enough file system space to actually manipulate the compressed archives. Resolve those issues, make and fully verify backups (that is, do an actual test restore from backup media if you don't do this routinely as a maintenance check) and do a full set of file system optimizations, if they are not already part of your regular maintenance.
Next, characterize your files as to space, quantity, age, owner, date of last access, date of last revision, and what ever other attributes your file system metadata can provide. Depending on your file server OS, there may be administrative utilities that can make this process pretty simple, or you may have to find suitable utilities that are workstation based.
Next, examine your filesystem authorizations, and see what files may belong to persons or groups no longer authorized on the system. Files that belong to no one any more are pretty common, and I've seen some servers where previous administrators had locks on hundreds of megs of install files for old versions of software that they never bothered to delete or re-assign on the day they left the organization. If you have any appreciable amount of space tied up in these black holes, you need to develop a strategy for evaluation and deletion, if appropriate.
Next, figure out a short list of department heads who can help you figure out what to do with questionable items, and get their agreement to do so. Spreading the load, a little, can improve the utility of the cleaned up system quite a bit, if you don't let the project drop into a "save everything" mentality.
Next, figure out what kind of organization you'd like to have when cleanup is complete. Everything you leave has to fit into that schema, and you should document it, for the person who will be doing this again, in 5, 10, or 15 years. Generally, organization by department or function is likely to be the most relevant and long lasting way of organizing a general purpose file server, as it tends to line up directory structures with group rights and individual job responsibilities. Thus, in your top level Office Admin volume, you'd probably have Accounting, Sales, and Administration sub-directories, along with, perhaps, one or more company wide General sub-directories, for interdepartmental file sharing, and storage of company wide forms.
Next, produce a paper inventory of the system's pre-cleanup contents. There's nothing like having a master list of everything you started with, complete with path and metadata, when you're in the middle of deleting and re-organizing things, and a user comes in and swears you've nuked the revised-Eastern-sales-budget-approved-by-God, when it was never on the server or any back up tapes in the first place. The last cleanup of a medium sized file server I did, this inventory ran to 480 double sided pages, and over a half million files. But in the weeks following the clean up, it saved my job at least 16 times.
Next, with your preparations complete, produce a candidate deletion list, and get any approvals needed. Do a backup, or archive the files to be deleted to media, and perform the deletions. Create any new directories, groups, or other organizational elements, and move remaining files appropriately. Have users review results in a definite go/no go acceptance/rejection time window, during which they neither create or delete additional items (may be useful to restrict all logins to browse authority during this period). Restore any inadvertent deletions from backup, as needed, until acceptance is forthcoming. When the users have accepted the clean up results, perform a complete backup and return the server to normal operation.
posted by paulsc at 3:48 PM on July 19, 2006
Paulsc's recommendations are great if you have a lot of time to spend on this project. For an approach that needs very little time:
Find a good location for the archived files, either on the network or on some suitable drive or device.
Separate out types of files. Make a folder called DOC and put all Word files in it. Do the same for XLS, PDF, etc.
For TXT, RTF, and DOC files, find a program - one that you like and can work with, such as TreePad, askSam, Zoot, etc. - to archive the multiple little files into single-file archives. Don't bother with any other organizing. You can search within the archive program for any information you need. (This step is optional in light of the next one.)
Buy a copy of dtSearch and use it to create an index of all of the files. The key here is that any later need for the files will be by use of the dtSearch desktop.
It is very likely that you will either never or very seldom have a need to refer to those old files. Using a powerful indexed search tool gives you quick access to the contents in the event that you need to get to them, but you don't have to spend a great deal of time at the outset.
posted by megatherium at 6:15 PM on July 19, 2006
Find a good location for the archived files, either on the network or on some suitable drive or device.
Separate out types of files. Make a folder called DOC and put all Word files in it. Do the same for XLS, PDF, etc.
For TXT, RTF, and DOC files, find a program - one that you like and can work with, such as TreePad, askSam, Zoot, etc. - to archive the multiple little files into single-file archives. Don't bother with any other organizing. You can search within the archive program for any information you need. (This step is optional in light of the next one.)
Buy a copy of dtSearch and use it to create an index of all of the files. The key here is that any later need for the files will be by use of the dtSearch desktop.
It is very likely that you will either never or very seldom have a need to refer to those old files. Using a powerful indexed search tool gives you quick access to the contents in the event that you need to get to them, but you don't have to spend a great deal of time at the outset.
posted by megatherium at 6:15 PM on July 19, 2006
I guess it'd be too simple-minded to suggest getting ahold of a good desktop search utility and letting it index all the files? "Pile, don't file," as they say.
posted by markcholden at 8:10 AM on July 20, 2006
posted by markcholden at 8:10 AM on July 20, 2006
I like what paulsc had to say. I agree that an organizational structure to any archive is very important and keeping everything can even be dangerous.
My advice on how to electronically store the documents differ from megatherium's above. At least, in one way. I don't see the benefit of storing things by file type. In the paper world, that's the equivalent of keeping things on notebook paper in one drawer, everything typed in another drawer, legal-sized in another, etc. That would leave related documents spread all over. The better idea, in my opinion, is to keep all related electronic files together - despite its file type. An example would be all things related to the "Hanson Project" would be archived under a folder name something like "Hanson Project 2002." Having that project spread all over the archive based on file type would be a mistake.
However, megatherium's advice on taking misc. bits of text and organizing them into an organized, searchable single-file database (I use Golden Section Notes) is excellent.
One other idea about archiving, that does involve file types, is to archive a copy of certain proprietary programs that you have created files with over the years. You may have created documents using a long-defunct program that is still installed on your hard drive. Remember how hard it might be to ever read that document again without that old program. Think about creating a folder called "Programs" that will have the setup files for some of those now-defunct applications. It wouldn't hurt, if space is not a concern, to put a copy of all major programs used to create your material. One day, what we all consider software that "everyone" uses, may one day be a quaint memory. Having a copy of that program, in order to read those files, could make the difference between being able to read them in the future - or not.
Good thoughts in this thread and it's very smart to be thinking of organizing all of this digital ephemera.
God Luck!
posted by Gerard Sorme at 1:01 PM on July 20, 2006
My advice on how to electronically store the documents differ from megatherium's above. At least, in one way. I don't see the benefit of storing things by file type. In the paper world, that's the equivalent of keeping things on notebook paper in one drawer, everything typed in another drawer, legal-sized in another, etc. That would leave related documents spread all over. The better idea, in my opinion, is to keep all related electronic files together - despite its file type. An example would be all things related to the "Hanson Project" would be archived under a folder name something like "Hanson Project 2002." Having that project spread all over the archive based on file type would be a mistake.
However, megatherium's advice on taking misc. bits of text and organizing them into an organized, searchable single-file database (I use Golden Section Notes) is excellent.
One other idea about archiving, that does involve file types, is to archive a copy of certain proprietary programs that you have created files with over the years. You may have created documents using a long-defunct program that is still installed on your hard drive. Remember how hard it might be to ever read that document again without that old program. Think about creating a folder called "Programs" that will have the setup files for some of those now-defunct applications. It wouldn't hurt, if space is not a concern, to put a copy of all major programs used to create your material. One day, what we all consider software that "everyone" uses, may one day be a quaint memory. Having a copy of that program, in order to read those files, could make the difference between being able to read them in the future - or not.
Good thoughts in this thread and it's very smart to be thinking of organizing all of this digital ephemera.
God Luck!
posted by Gerard Sorme at 1:01 PM on July 20, 2006
Thanks for the thoughtful responses. I'd hoped that there were "rules of order" for filing that I just wasn't clued in on, but perhaps it's an ad hoc process.
The main issue for me is that I'd like the structure to be a tool to help keep some organization, rather than a searchable grab bag. I've been using Copernic search for quite awhile and it can typically help me find anything with text.
Paulsc's comments resonate the most. I didn't think about making a back up list of all the directories.
Currently, I've decided to approach it by dumping it all on the table and see if I can discern an organic structure. Then I'll review the new system with boss and staff and go from there. What's making this a little easier is using the Freemind mind map software. I discovered you could copy and paste folders into it and it displays the list as part of the outline.
Here's a link to a picture...or at least I think I've linked it.
posted by donp17 at 6:36 PM on July 20, 2006
The main issue for me is that I'd like the structure to be a tool to help keep some organization, rather than a searchable grab bag. I've been using Copernic search for quite awhile and it can typically help me find anything with text.
Paulsc's comments resonate the most. I didn't think about making a back up list of all the directories.
Currently, I've decided to approach it by dumping it all on the table and see if I can discern an organic structure. Then I'll review the new system with boss and staff and go from there. What's making this a little easier is using the Freemind mind map software. I discovered you could copy and paste folders into it and it displays the list as part of the outline.
Here's a link to a picture...or at least I think I've linked it.
posted by donp17 at 6:36 PM on July 20, 2006
http://don.partlan.googlepages.com/officeadmin
there it is...I'm not much for html.
posted by donp17 at 6:37 PM on July 20, 2006
there it is...I'm not much for html.
posted by donp17 at 6:37 PM on July 20, 2006
One point paulsc made that I think may have gotten lost in the noise was "try your best to preserve the mod dates on the files; they can be important".
This precludes moving them with anything that isn't an archiver, on some filesystems and OS's.
posted by baylink at 7:40 PM on July 20, 2006
This precludes moving them with anything that isn't an archiver, on some filesystems and OS's.
posted by baylink at 7:40 PM on July 20, 2006
This thread is closed to new comments.
posted by brina at 2:49 PM on July 19, 2006