"Now I know I’ve seen it around here somewhere": programming edition
May 25, 2021 2:43 PM Subscribe
At least once per week, I need to solve a tiny programming problem similar to one I have previously solved in some other project. (This most recently came up when I realized I might need to vectorize a function in R.) I waste so much time racking my brain for what I was working on when I solved it so that I can find it again. Is there a technical tool that can help with this?
Note: these aren’t necessarily things that would benefit from writing a new function and sticking it in a library somewhere. Most of the time what I need is a working example of how I got an already existing function to work with real data.
Note: these aren’t necessarily things that would benefit from writing a new function and sticking it in a library somewhere. Most of the time what I need is a working example of how I got an already existing function to work with real data.
If you're on a Mac, the command-line program
posted by theodolite at 2:51 PM on May 25, 2021 [6 favorites]
mdfind can search the contents of all of the text files on your computer almost instantaneously and has some useful options for narrowing things down. I do this all the time for the exact reason in your question, using whatever piece of the function I can remember as my search term. Using a recursive grep works similarly on Linux and Windows Powershell has Get-ChildItem, although those might be slower since I'm not sure they use an already-generated index like mdfind does.posted by theodolite at 2:51 PM on May 25, 2021 [6 favorites]
I think what I do with Joplin is what you are talking about. Joplin is a note taking app that uses Markdown, but presumably you could do the same thing with Microsoft Code with a Markdown plugin. I like Markdown in general, but it's particularly nice that you can easily flip into "fixed font code in shade" by using one or 3 grave accents.
For instance, I have two or three Markdown documents devoted entirely to ffmpeg, and I know I'm probably not the only one. That utility is fantastic.
But the same goes for Bash parameter reference specifications (is it one or two % or #)?
I have notes about the many infuriating Windows problems, notes on my nightly backup processes, etc. Oh, and a note about what are scripts, functions and aliases in my Linux setup (it can be hard to remember).
If you want me to send you an example MeFiMail me, but if it's not what you were thinking of never mind.
posted by forthright at 2:53 PM on May 25, 2021
For instance, I have two or three Markdown documents devoted entirely to ffmpeg, and I know I'm probably not the only one. That utility is fantastic.
But the same goes for Bash parameter reference specifications (is it one or two % or #)?
I have notes about the many infuriating Windows problems, notes on my nightly backup processes, etc. Oh, and a note about what are scripts, functions and aliases in my Linux setup (it can be hard to remember).
If you want me to send you an example MeFiMail me, but if it's not what you were thinking of never mind.
posted by forthright at 2:53 PM on May 25, 2021
I’d recommend more verbose comments in these projects so they come up more easily when you search your computer or your code repository.
I also think “a working example of how I got an already existing function to work with real data” is a great way to document. If you need to, copy whole files or multiple files into your notes, not just the function itself.
posted by michaelh at 3:05 PM on May 25, 2021 [3 favorites]
I also think “a working example of how I got an already existing function to work with real data” is a great way to document. If you need to, copy whole files or multiple files into your notes, not just the function itself.
posted by michaelh at 3:05 PM on May 25, 2021 [3 favorites]
Similar to mdfind, I use on mac
posted by cgg at 3:06 PM on May 25, 2021 [2 favorites]
find ./src -type f -name '*.js' -exec grep -il "search term" {} \; a ridiculous amount. Replace your source directory, file type and search term as needed.posted by cgg at 3:06 PM on May 25, 2021 [2 favorites]
I use emacs with org-mode for this, but it's just a glorified text file with some keyboard shortcuts for collapsing and expanding sections, etc. Nothing beats a single, flat file. That way you never have to guess where it is. Just open the file and search.
posted by number9dream at 3:10 PM on May 25, 2021 [6 favorites]
posted by number9dream at 3:10 PM on May 25, 2021 [6 favorites]
I keep a blog for documenting bioinformatics code snippets that I forget about and need to come back to a year or two later. Blog posts get indexed and made searchable via popular search engines, which makes it easier to find things. A domain-focused blog also helps others who may have similar technical questions and look online for answers, and it is useful as a public demonstration of knowledge, which can be useful for other purposes.
posted by They sucked his brains out! at 3:16 PM on May 25, 2021 [1 favorite]
posted by They sucked his brains out! at 3:16 PM on May 25, 2021 [1 favorite]
Seconding theodolite's approach of having a fast indexed search of your codebases; I use VSCode for this at work. Useful for everybody, but especially if you're the kind of person, like me, who's better at producing well-formed code than well-maintained notes.
posted by vincebowdren at 3:17 PM on May 25, 2021
posted by vincebowdren at 3:17 PM on May 25, 2021
I also use VSCode for this. I have a folder called "workbook" in my Google Drive. I just open new files every time I have something I want to hold onto longer than a transient copy-and-paste. I'm motivated to name them something intelligent so I get syntax highlighting. The search function in a single VSCode window works really nicely.
posted by supercres at 3:31 PM on May 25, 2021
posted by supercres at 3:31 PM on May 25, 2021
I use R, as well. I use something very simple: I have a Word document (currently 13 pages long) where I put code snippets, some explanatory text, and sometimes links to StackExchange posts and the like. When I need to find something, I just hit "ctrl-f" and search for it. This document has saved me countless hours. And I've trained myself to be really good about adding information to it, whenever I encounter something new and non-obvious.
posted by alex1965 at 3:33 PM on May 25, 2021 [1 favorite]
posted by alex1965 at 3:33 PM on May 25, 2021 [1 favorite]
I store code snippets in this app called Quiver (mac only)
posted by dhruva at 3:58 PM on May 25, 2021
posted by dhruva at 3:58 PM on May 25, 2021
I'm like this too! A few ideas:
* Have you looked into Code Snippets in RStudio? This is more for a "compose macros as you go" approach rather than making search easier, but, may work for ya!
* If you do want to build up some kind of document of examples as you go, I definitely recommend using reprex to compose reproducible examples. This will be especially helpful for showing examples using data that are ALSO reproducible - for future-you who really wants to remember how to clean messy address strings without relearning a data model from a particular API, say.
* In general, how consistent are your repos/scripts/apps in terms of folder structure, names for testing/helper/etc. scripts? (NO JUDGMENT here I am still working on this one, eep!) I find that getting more uniform in general helps me reduce cognitive load when I'm mucking about for whatever I'm seeking.
Curious to learn more about what you end up using!
posted by elephantsvanish at 3:59 PM on May 25, 2021
* Have you looked into Code Snippets in RStudio? This is more for a "compose macros as you go" approach rather than making search easier, but, may work for ya!
* If you do want to build up some kind of document of examples as you go, I definitely recommend using reprex to compose reproducible examples. This will be especially helpful for showing examples using data that are ALSO reproducible - for future-you who really wants to remember how to clean messy address strings without relearning a data model from a particular API, say.
* In general, how consistent are your repos/scripts/apps in terms of folder structure, names for testing/helper/etc. scripts? (NO JUDGMENT here I am still working on this one, eep!) I find that getting more uniform in general helps me reduce cognitive load when I'm mucking about for whatever I'm seeking.
Curious to learn more about what you end up using!
posted by elephantsvanish at 3:59 PM on May 25, 2021
I’ve used GitHub Gists to store code snippets before for this kind of quick reference.
posted by kellygrape at 4:57 PM on May 25, 2021 [2 favorites]
posted by kellygrape at 4:57 PM on May 25, 2021 [2 favorites]
On Linux I use a lot of 'ack' which is just a recursive grep thing that only looks at source code and text files by default. There are plenty of other tools like this that make that `find | grep` a whole lot simpler. Code is all generally under '~/proj' or '~/work' somewhere.
It also helps if you're used to using some sort of 'make a module or app out of everything' so that each project might have a file that lists all of the external dependencies, you can 'ack' those to find projects that used some module pretty easily. Also useful for CI and testing (like all dependencies are installed and right version?).
Every directory has a file called '0liners' to keep it at the top of the listing. That's where snippets and magic commands go. You can 'ack' those pretty easily.
Bigger stuff goes in either '~/Notes' or '~/Templates' depending on whether or not it's useful enough to eventually clean it up to be something you just read into you current program or not.
So yeah, mostly just grepping things until they get promoted into Notes or Templates. Once in a blue moon cat together all the 0liner files and pull out some good stuff.
posted by zengargoyle at 5:11 PM on May 25, 2021 [3 favorites]
It also helps if you're used to using some sort of 'make a module or app out of everything' so that each project might have a file that lists all of the external dependencies, you can 'ack' those to find projects that used some module pretty easily. Also useful for CI and testing (like all dependencies are installed and right version?).
Every directory has a file called '0liners' to keep it at the top of the listing. That's where snippets and magic commands go. You can 'ack' those pretty easily.
Bigger stuff goes in either '~/Notes' or '~/Templates' depending on whether or not it's useful enough to eventually clean it up to be something you just read into you current program or not.
So yeah, mostly just grepping things until they get promoted into Notes or Templates. Once in a blue moon cat together all the 0liner files and pull out some good stuff.
posted by zengargoyle at 5:11 PM on May 25, 2021 [3 favorites]
if you're an atlassian shop, I'd say log everything you do in confluence with great tagging and easy, sensible page titles. add a link to the repo location.
it's my mode of operating and it works pretty good so far.
posted by j_curiouser at 6:00 PM on May 25, 2021 [1 favorite]
it's my mode of operating and it works pretty good so far.
posted by j_curiouser at 6:00 PM on May 25, 2021 [1 favorite]
My reference library is more command-line stuff / one-liners, and I convert those into autocompletions in Alfred. For lumps o'code, I use Gists.
The refactored answer depends on whether you're a hierarchy / taxonomy person or a document-all-the-things person or a keyword-search person or... etc. -- that is, how do you typically retrieve information and how likely are you to start and stick to new habits if that helps?
A big ol' text file that's well-documented (per trig and others) might be sufficient for you. If you think in more structured ways you could always use a locally-hosted wiki.
posted by holgate at 10:08 PM on May 25, 2021
The refactored answer depends on whether you're a hierarchy / taxonomy person or a document-all-the-things person or a keyword-search person or... etc. -- that is, how do you typically retrieve information and how likely are you to start and stick to new habits if that helps?
A big ol' text file that's well-documented (per trig and others) might be sufficient for you. If you think in more structured ways you could always use a locally-hosted wiki.
posted by holgate at 10:08 PM on May 25, 2021
I used to rely heavily on keeping everything I did in a file for emacs C-s, so the current times have been interesting. But I've found, as long as it's conveniently searchable by month, and globally searchable ever, life finds a way.
posted by away for regrooving at 1:03 AM on May 26, 2021 [1 favorite]
posted by away for regrooving at 1:03 AM on May 26, 2021 [1 favorite]
Assuming this is for work — does your workplace use Slack?
I’ve found that Slacking these sorts of things to myself works great for me in a way that other solutions (e.g a giant text file) don’t. For me it’s the ease of access (don’t need to remember a file name), the formatting, and the searchability.
posted by mekily at 1:30 AM on May 26, 2021
I’ve found that Slacking these sorts of things to myself works great for me in a way that other solutions (e.g a giant text file) don’t. For me it’s the ease of access (don’t need to remember a file name), the formatting, and the searchability.
posted by mekily at 1:30 AM on May 26, 2021
I really like how Simon Willison keeps his Today I Learned site.
Every time he learns something new he adds a new file to the GitHub repository and it automatically rebuilds the site.
posted by fabius at 4:56 AM on May 26, 2021 [1 favorite]
Every time he learns something new he adds a new file to the GitHub repository and it automatically rebuilds the site.
posted by fabius at 4:56 AM on May 26, 2021 [1 favorite]
I use emacs org mode and its markdown format for notes. (Currently at 1644517 words for about 8y of notes) It has a capture mode where you hit a key combo and it opens a note taking window, then when you save it, that closes and returns you to where you started from. This is super fast. I can take and save a note before an atlasian web page even loads.
I save these notes in per year files on dropbox. I've tried wikis, confluence, docs, comments, email, google docs. None of those are ideal for personal notes. They are too slow, loose your pre-note context, have formatting that fights you, lose support, are difficult to take from job to job. Use something that will be yours forever and it will show you its value. Over time you will get better at knowing what to put in your notes.
The other things I do is put all code in one directory and use ripgrep. There is talk above about awk and find-grep but move into the 2020's and try ripgrep. It is so fast that it changed how I grep for things.
posted by bdc34 at 6:32 AM on May 26, 2021 [1 favorite]
I save these notes in per year files on dropbox. I've tried wikis, confluence, docs, comments, email, google docs. None of those are ideal for personal notes. They are too slow, loose your pre-note context, have formatting that fights you, lose support, are difficult to take from job to job. Use something that will be yours forever and it will show you its value. Over time you will get better at knowing what to put in your notes.
The other things I do is put all code in one directory and use ripgrep. There is talk above about awk and find-grep but move into the 2020's and try ripgrep. It is so fast that it changed how I grep for things.
posted by bdc34 at 6:32 AM on May 26, 2021 [1 favorite]
I have read people developing private tools that they build a big library of their own and call functions from there.
But to remember just syntax examples, I have a file in my home directory called one_liners.txt that just contains a one-line comment and then a one-line example -- for ffmpeg, youtube-dl, rsync, and other commands that have a ton of options.
posted by wenestvedt at 7:22 AM on May 26, 2021
But to remember just syntax examples, I have a file in my home directory called one_liners.txt that just contains a one-line comment and then a one-line example -- for ffmpeg, youtube-dl, rsync, and other commands that have a ton of options.
posted by wenestvedt at 7:22 AM on May 26, 2021
I've worked with sales engineers and consultants who rely on Evernote for this type of thing. It's not the best tool for code snippets, but it is searchable, taggable, accessible from any machine, etc.
posted by neushoorn at 9:02 AM on May 26, 2021
posted by neushoorn at 9:02 AM on May 26, 2021
Response by poster: Thanks, all, this is a nice variety of tools. A couple things I should clarify.
- I’m on Windows 10 and I don’t have admin privs, so solutions that use tools native to this environment are the least heavy lift. I can get stuff installed but there’s a process.
- We are not much of a “shop”; it’s academic science, which means both that I can invent my own idiosyncratic workflow, and also that I have to.
- I loved the clarifying questions about how I think/organize. Directory structure inside a project is consistent but higher order directory structure is super duper not. I love the idea of an organized set of things I might need later, but in practice, this hasn’t worked for me because I don’t always understand a snippet as useful until I encounter it again.
I think I will be happiest with a solution that allows me to search all files with a given extension in certain specified directories. This seems like it should exist even on Windows, but I don’t know it yet. I think a lot of the solutions upthread make most sense for other environments.
posted by eirias at 10:46 AM on May 26, 2021
- I’m on Windows 10 and I don’t have admin privs, so solutions that use tools native to this environment are the least heavy lift. I can get stuff installed but there’s a process.
- We are not much of a “shop”; it’s academic science, which means both that I can invent my own idiosyncratic workflow, and also that I have to.
- I loved the clarifying questions about how I think/organize. Directory structure inside a project is consistent but higher order directory structure is super duper not. I love the idea of an organized set of things I might need later, but in practice, this hasn’t worked for me because I don’t always understand a snippet as useful until I encounter it again.
I think I will be happiest with a solution that allows me to search all files with a given extension in certain specified directories. This seems like it should exist even on Windows, but I don’t know it yet. I think a lot of the solutions upthread make most sense for other environments.
posted by eirias at 10:46 AM on May 26, 2021
If you're using source control (like Git, Subversion, Perforce, Mercurial, etc.) then you should be able to quickly search through all the changes that you've personally made.
If you're not using source control, then you definitely should start because it helps give a time-based view of the code (among other things).
posted by meowzilla at 1:44 PM on May 26, 2021 [1 favorite]
If you're not using source control, then you definitely should start because it helps give a time-based view of the code (among other things).
posted by meowzilla at 1:44 PM on May 26, 2021 [1 favorite]
Agent Ransack / File Locator Pro (https://www.mythicsoft.com/agentransack/ ) is a great tool for searching files. You can search all files in a set of folders with a specific extension. It has a number of other useful options like filtering by file date, excluding some patterns, and wildcard / regex matching. It has a standalone version that works without admin permissions. I use it for locating phrases in code regularly.
posted by nalyd at 5:56 PM on May 26, 2021 [1 favorite]
posted by nalyd at 5:56 PM on May 26, 2021 [1 favorite]
Response by poster: Thanks again, everyone. After thinking it through I realized that the reason grep kept coming up in suggestions above was that it really is exactly the tool I want (even if I sort of hate and fear regexp). So I had my tech team install Cygwin and that’s that, or will be once I figure out which options actually do the thing efficiently.
I do also think the suggestions about note keeping are also quite worthwhile, they just intersect badly with my faults as a programmer. But it’s worth trying to develop a habit anyway because on the few instances I have done this right it has saved me buckets of time.
posted by eirias at 9:38 AM on May 27, 2021
I do also think the suggestions about note keeping are also quite worthwhile, they just intersect badly with my faults as a programmer. But it’s worth trying to develop a habit anyway because on the few instances I have done this right it has saved me buckets of time.
posted by eirias at 9:38 AM on May 27, 2021
You can kill two birds with one stone (kinda) by keeping notes about which regexes were useful to you (or, alternatively, make aliases/scripts for the useful ones).
posted by trig at 11:25 AM on May 27, 2021 [1 favorite]
posted by trig at 11:25 AM on May 27, 2021 [1 favorite]
This thread is closed to new comments.
posted by trig at 2:46 PM on May 25, 2021 [4 favorites]