Hi. I'm also an upstream developer on both Twisted and Nevow. Chris
just explained what's going on to me and I think it might be helpful in
resolving this issue if I explained the twisted plugin system's
motivation and design in a bit more depth.
Basically, the idea behind the Twisted plugin system is simple: allow
applications to inspect the importable modules in a particular package,
and load objects which provide a particular interface from those
modules. We tried as hard as possible not to change the rules of
regular Python importing.
However, a completely literal interpretation of Python imports made
development installations difficult to work with, because one needed to
copy plugin files from the development installation of a project (say,
Nevow) to the working Twisted installation every time they changed -
otherwise, you wouldn't be able to import the appropriate module. So we
added a tiny bit of extra logic to twisted.plugins' __init__.py to allow
dependent projects to have their own "twisted/plugin" directory, and
allow another mechanism (virtual-python, combinator, PYTHONPATH in
.bashrc) to control sys.path.
This was originally a very naive expression which would set __path__ on
twisted.plugins to allow importing from every "twisted/plugins"
directory on sys.path. However, this was a mistake, because it meant
that a developer who had specifically configured their PYTHONPATH to
pick up a Twisted installation in their home directory would suddenly
find modules coming from twisted.plugins in their system install. This
would mix together incompatible versions of Twisted, breaking otherwise
working code. The change that Chris described above fixes this bug, by
only including "twisted/plugin" directories on sys.path which
specifically are not packages, like those which are present in projects
which provide Twisted plugins (Axiom, Nevow, and MV3D, to name a few).
These projects specifically and intentionally omit __init__.py files,
because the intent is that when they are installed, they should not
overwrite the twisted/__init__.py or twisted/plugins/__init__.py
installed by Twisted. As it happens, neither of these files are empty
and overwriting them would break a Twisted installation.
Back to the original intent of this plugin system: it is to allow
applications to interrogate modules which can be loaded using Python's
normal 'import' semantics. Without the aforementioned __path__ magic
(which is provided for developers' convenience who are trying *not* to
touch the system install, and should not be relied upon by a system like
Debian, which controls the system install), if you have a
/usr/lib/python2.5/site-packages/mypackage/foo.py and a /var/lib/python-
support/python2.5/mypackage/bar.py (each next to its own __init__.py),
only mypackage.foo will be importable - mypackage.bar will be
effectively invisible to Python.
In my opinion, python-support should not be providing multiple
conflicting packages on the default sys.path regardless of this
particular issue. Obviously, this makes it more pressing, but multiple
things with the same name, disambiguated only by the ordering of
sys.path, can cause confusion for any code, especially naive code, which
attempts to interrogate sys.path. It causes confusion for users who are
looking at a directory listing in one window and an ImportError in
another. Consider this:
$ mkdir -p a/x
$ mkdir -p b/x
$ touch a/x/__init__.py
$ touch b/x/__init__.py
$ touch b/x/y.py
$ PYTHONPATH=a:b python -c 'import x.y'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named y
There's no information available for the user to determine that the "x"
package is coming from the "a" entry on sys.path here; the error is very
difficult to debug, even more so if it stems from a bug in a system-
installed package.
I hope this clearly illustrates why Twisted works this way. I believe
that the correct solution here is neither to remove the auto-generated
__init__.py nor to re-introduce the bug described above that we fixed
upstream in Twisted. Happily, there is a unit-test for this bug, so if
you do need to manipulate __path__ somehow, you can verify that it's a
supported way by running the twisted test suite.
To be honest I do not fully understand the purpose of python-support; in
particular, the addition of /var/lib/python-support/pythonX.X to
sys.path seems to create a lot of problems like this one and I don't see
what issues it solves. There are already tons of symlinks here; why not
just symlink everything in /var/lib into /usr/lib appropriately?
However, assuming there is a good reason for the addition of this path
entry, once a location has been selected for a given python package
(i.e. "/usr/lib/python2.5/site-packages" for "twisted.plugins"), every
other Debian package that intends to place files into that Python
package really needs to follow suit and put its own files there, or
resort to unpleasant and potentially confusing workarounds to makes the
available modules be importable.
