How to organise your research/ projects?

August 19, 2020 1:17 PM

I recently started working at a new lab and have now the opportunity to set up a structure for my research that hopefully will be scaleable, rational, efficient and understandable by others! Unfortunately I got somewhat stuck in the nitty-gritty of deciding on naming convention for files and folders, how to separate actual experiments from preliminary experiments, reports, analysis, documentation of data-analysis pipelines, and so on. I am already constantly hunting for files and trying to remember what was what, where I filed this ambiguous piece of info that is relevant to multiple projects, etc..

I think I already had two insights, namely that to rely solely on Windows Explorer to organize is not adequate, because you can only import meaning by folder name, hierarchy and file name. This tends to produce convoluted file trees with too many subfolders and too long file names. It is also not suited for ambiguity, because a piece of information can't belong to two projects (you can copy but then it isn't synchronized). So I plan to harness my coding powers. Every file with a filetype that supports it gets some info at the very beginning, like
# Description: Extract Information
# Details: From samples 1 and 2
# Type: Experiment
Then I run a Script which makes a list of every file, extracts relevant attributes like creation date, file name and path, and the above information.
This will be put in a Excel list along a hyperlink to the file. This list can then be filtered, sorted and every file is clickable, in essence a pimped file explorer.
The other thing I want to do is record as much data for experiments as possible in single Excle workbooks, to avoide file creep. Ideally every instance of an Experiment gets one excel file with protocol, raw data, analysis.
Did you do something similar, totally different, have any key insights in the matter? I’m already iterating over my scheme, but would be glad to hear from others how they managed similar problems.

posted by SweetLiesOfBokonon to Science & Nature (6 answers total)

I'm a researcher and this plan to create a map your file folder system in Excel strikes me as overly elaborate, though maybe I'm not appreciating how complex your project is. If files are named clearly and systematically an index usually shouldn't be necessary, I'd think. I have occasionally needed to create guides to my files when transferring a project I worked on alone to a successor at work. I usually create them in Word with file path links and notes of explanation.

Are you aware of the possibility of creating file and folder shortcuts in Windows Explorer? (In Windows, the option should appear when you right click on any folder or file in Explorer.) It would seem to me that this would address your desire to place a single bit of information in two places.
posted by reren at 2:52 PM on August 19, 2020 [5 favorites]

Just use search and have annoyingly long file names (date, project name, description of file). My research files are mostly organized by project with separate folders for admin and regulatory stuff that's simpler to keep together. Most of these are shared with multiple other people via Box. All your links and Excel spreadsheets of files are going to break and/or other people won't use or understand them and they won't be compatible with cloud storage. Keep it simple. Do not code a solution for this.

I use multiple Excel tabs to keep different data tables together and usually have one tab with a PivotTable for quick and dirty analysis - tabs get separated into .csv files when I do further analysis in statistical software, I keep all the files for use with that software together for various logistical reasons. I include a metadata tab, but a full protocol is going to be obnoxious to deal with in Excel. Protocols go in Word with dated file names, and old ones get tucked away in an "Archive" folder at the same level as needed.
posted by momus_window at 3:14 PM on August 19, 2020 [1 favorite]

Best answer: If you haven’t already, you might want to do some reading on “reproducible research” especially with regards to “file management” or “folder structure.”
posted by oceano at 4:08 PM on August 19, 2020 [2 favorites]

Best answer: If you like the pimped file explorer model, try a local wiki. I use Zim for something similar, and I've also had good luck with DokuWiki. You can link to (lists of) files from as many different pages you want, add whatever commentary you want, and interlink the pages. Various plugins for sorting etc.
posted by brianconn at 7:39 PM on August 19, 2020 [1 favorite]

Welcome back to a main frame world. Use some of the techniques that you would use with designing, say, a database. Sort out a naming protocol - I am at the stage where a MMM YY field is a standard part of any file that I create. Think about a hierarchy of files and using the hierarchy to create the naming protocols

Get used to creating the name of the file BEFORE you put any data into it.

Careful with data inputs - what you thought was 10.F can easily be interpreted as cell 10F - I am fairly strict about keeping numbers as ONLY numbers and letters as ONLY letters and if it is a number-letter construct - the number goes in one cell and the letter in another. Beats having to verify whether the entry is 1 or l by changing the font back and forth.
posted by Barbara Spitzer at 3:49 AM on August 20, 2020

If you work at a lab at a university, your library might have librarians who specialize in helping with research data management and would love to help out.
posted by kbuxton at 2:56 PM on August 20, 2020