My python's breath smells like ImportError

October 6, 2006 8:14 AM

Referencing roll-your-own Python modules in other Python applications. What's my deal?

So at my newish job I am consolidating a bunch of redundant code into a core library which is referenced by various CGI scripts. The problem comes when I try importing these libraries. For sake of demonstration, my directory structure looks like this:

(a) /home/me/html/bin/
(b) /home/me/html/bin/edu/school/library/utilities/
(c) /home/me/html/bin/appname/

where (a) is the base CGI directory, (b) is the directory in which the library lives and (c) is where the application importing a module from the library lives.

When an application importing a module in (b) resides in (c) I get an ImportError. When an application importing a module in (b) resides in (a) the import comes off without a hitch. AFAIK, I've isolated every variable such that the only difference between the two cases is the directory in which the calling application lives.

While I can kludge things around by putting all my CGI apps in /bin, I'd prefer to organize things a bit better. Thus I invoke the MeFi hive mind to provide me the undoubtedly easy solution for importing modules from an arbitrary location in the file system.

My final salvos:
1. __init__.py lives in every directory from bin on down.
2. sys.path includes /home/me/html/bin/edu

posted by Fezboy! to computers & internet (6 comments total)

Perhaps you just need to set the PYTHONPATH environment variable in your apache config. When your app is in (a), Python can find it because it's a relative path I'm assuming. So if you add (b) to the PYTHONPATH (or maybe (a) depending on if (b) is inside a Python package or not) it should be able to find it no matter where the application lives.

By default Python is looking in "." and in /usr/share/lib/python/VERSION/site-packages (or something similar). If you put libraries somewhere else, you have to tell it where else to look.
posted by metajack at 8:21 AM on October 6, 2006

Sigh, somehow I missed the last three lines.

Are all the __init__.pys empty? Perhaps you just need an import X where X is the next dir down in each one.

ie, in edu/__init__.py:
import school
in edu/school/__init__.py:
import library

etc.
posted by metajack at 8:26 AM on October 6, 2006

I gave that a shot metajack and no dice. Boo for me...

In the interim, I did try a few other things.

if I do something like:
import edu.school.library then I get a reference to the library module.

if I do:
import edu.school.library.entities then it gives me an ImportError.

if I do:
after copying Person.py to this directory
import edu.school.Person then I get an ImportError

if I do:
from edu.school.Person import Person I also get an ImportError.

So, basically, it looks like I can get references to packages up to three levels deep but cannot get a reference to a module unless it is in a path that is directly descended from the calling script.
posted by Fezboy! at 9:35 AM on October 6, 2006

Ok. I did some quick tests.

First off, sys.path (or PYTHONPATH) should not have the package dir in it, it should be the directory the package dir is in. In your case: /home/me/html/bin

I set up a similar dir structure:

~/test/app (app dir)
~/test/app/test.py (test app, just contains an import statement)
~/test/edu (toplevel package)
~/test/edu/__init__.py (blank)
~/test/edu/school
~/test/edu/school/__init__.py (blank)
~/test/edu/school/library
~/test/edu/school/library/__init__.py (blank)
~/test/edu/school/library/utilities
~/test/edu/school/library/utilities/__init__.py (contains "import person")
~/test/edu/school/library/utilities/person.py (contains "class Person: pass")

I get no ImportError from test.py (running from it's dir with PYTHONPATH=~/test/) for any of these:
import edu
import edu.school
import edu.school.library
import edu.school.library.utilities
from edu.school.library.utilities.person import Person

I suspect you just had too much in your path.
posted by metajack at 10:29 AM on October 6, 2006

/me wipes tears of joy and shame from his eyes and gives a hearty thank you to metajack for the effort.

Turns out there was a symlink floating around in the path left by the previous web monkey. Pulling that and manually appending /home/.... to sys.path in site.py makes all things possible. Now I can continue writing doing my ad hoc work in Python while I finish setting up the actual environment all of this crap will have to be ported to eventually instead of pulling a series of all-nighters to get it done this weekend.
posted by Fezboy! at 12:13 PM on October 6, 2006

Now that your import woes are solved, do yourself a favor and use mod_python with the legacy CGI helper. Raw CGI is not a very good solution in most cases. Moving to mod_python or wsgi will help tons in cleaning up the Python code.

http://www.devx.com/webdev/Article/29357/0/page/3
posted by Invoke at 2:55 PM on October 6, 2006