keeping track of yourself in python

August 26, 2008 2:59 PM

How do you keep track of all your python libraries and modules and previous work?

It seems there is no end to new modules one can download for python. My problem is that I don't use python everyday, so i often forget if i installed some library, or exactly where are modules i have written myself, where they are, what modules they work with, what information they depend on etc etc.

Basically i am looking for any advice to keep myself organized in this regard; tools, conventions, tricks, and simply some fresh thought.

Maybe the only solution is to just keep on sluggin away until this python stuff is more naturally remembered, or maybe metafilter can shove me in a direction i haven't thought of.

posted by figTree to computers & internet (6 comments total)

well, as far as keeping track of previous work goes, its really up to personal preference, but i have a code directory, with subfolders for all major projects, plus a 'one-off' folder for hackery. i'm usually pretty good with documentation so i'll at least have a docstring at the top to remind me what the crap i was doing at that time.

for modules installed in site-packages, a quick scan of the directory can give you an overview of what you have. if you want more details, fire up pydoc and hit up the ol' browser, and you can see a nice index of all you got installed, plus the batteries that are, in fact, included.

IRT remembering what everything does, i'm always deep in the python chm (windows help, use xchm on mac), checking out the Library Reference, and anything i need to do, i always check to see if python has it builtin (probably 80% of the time yes). take the time to look through the top level of the library reference and see whats there for you.
posted by Mach5 at 4:43 PM on August 26, 2008

i often forget if i installed some library

Always install libraries with setup.py install, and they should end up in site-packages. Easy. Although to be honest, I don't understand the deal here. It's okay to forget what you have installed. Just use the libraries you need. If they're installed, great; if not, you'll get an error message and you can install them.

exactly where are modules i have written myself, where they are, what modules they work with, what information they depend on etc etc.

I keep all my source code in ~/src, making new directories for each package. I do not run them from here—I always make setup scripts and install them. It's easy enough to see which modules they work with; just grep *.py for "^import ". If you want to have more detailed dependency information that will allow auto-install of dependencies, look into setuptools and easy_install. Put a README in each of these directories with all the notes you need.

Also, keep your source in version control. Hopefully you can keep the repository on another computer so that you get automatic backup. I recommend Subversion.
posted by grouse at 5:58 PM on August 26, 2008

As grouse & Mach5 suggested, for third party libraries you can see what's been installed by looking in $PYTHON_INSTALLATION/lib/python-2.5/site-packages. I also keep any packages I install manually in a folder called "INSTALLED", which is useful if the package comes with documentation or other ancillary files.

If you'd feel more comfortable with a local (i.e. user account solution) rather than the global library one, there are a few solutions. You could designate a folder in your home directory for Python modules and append that to your PYTHONPATH. Note this will only work for modules you manually move into there - setup.py install will still install everything to the global site-packages. To get around this, you could create a virtual Python, that lets you treat Python and its libraries as if they were local, including for installation purposes.

As for controlling your own libraries, it would be very handy if you developed everything as a setuptools-enabled package. This takes a little bit of wrangling to setup (using paster & ZopeSkel) but after that you can install anything you're working on with "python setup.py develop" and Python will load your package in place, saving you the develop-install-change-install cycle. (This sicks a ".pth" file in Python's site-packages, making it treat your library as if it was physically in the package directory.)

Using setuptools and cheeseshop / pypi for as many packages as you can is handy, because setuptools handles pre-requisites and complains if it can't find any. So "easy_install formbuild" will download and install the formbuild package, and all the other packages that it depends on.
posted by outlier at 1:50 AM on August 27, 2008

using paster & ZopeSkel

Those seem mainly targeted at web development, and the latter particularly at Zope. Are they otherwise useful?
posted by grouse at 8:05 AM on August 27, 2008

Surprisingly, yes. Paster is the tool that generates project layour and boilerplate, ZopeSkel installs a series of templates, several of which are for Zope, but others of which are just general Python. It saves a lot of the fiddling around you are to do to get a setuptools tools enabled project, with a sane install script and a myriad of other features. (I'm fond of using "nose" to test my packages, which you can get to run off an appropriate setup script.)
posted by outlier at 3:49 PM on August 27, 2008

Definitely look into version control for your own code, but I'd recommend the current-generation distributed version control systems over Subversion. Subversion uses a server/client model, meaning you'll need to have two spots set aside for your code (one for the repository, and one for your checkout); DVCS collapses this into one.
posted by korpios at 7:35 PM on September 4, 2008