Hi Romain,
CC-ing cython-devel here, as this is not an entirely trivial question to
answer.
Romain Guillebert, 23.07.2011 06:52:
During this week, I've been trying to find a way to implement cimport, I
think that the cleanest way of implementing this is to compile pxds just
like pyxs are compiled.
You are aware that there are two types of .pxd files? One next to a .pyx or
.py file with the same name, which provides additional declarations for
that source file and can also serve as a cimportable .pxd for other
modules, thus providing a C-level API between modules, and a second type
that only contains external declarations (well, and potentially inline
functions etc.), usually for external C libraries or header files, but that
does not correspond to a .pyx/.py source file.
The idea of materialising the external .pxd files does sound appealing. I
imagine that this means that the resulting .py file would basically just
set up ctypes wrappers for external declarations and store them under the
corresponding name, right? Then a cimport from an external .pxd file would
turn into a regular import. Plus, you'd end up with a single place where
the external library (if any, or potentially many libraries) is loaded and
managed.
Cool.
Even the first kind of .pxd file would fit into this quite nicely. Here,
the corresponding .pyx/.py file would be enough to represent the module,
and the exported declarations in the .pxd file would just be ignored.
Consequently, the .pxd file would only be used to provide declarations for
the module itself and serve no other purpose. Thus, in both cases, there
would simply be an importable .py file for each cimported .pxd file, be it
user provided or generated.
I'm having trouble knowing where to implement that though.
I think that I will implement that in the method process_pxd of Context
in Cython/Compiler/Main.py (and it will probably call compile_single)
but I don't know if it's a good design.
What do you think ?
I think it could be done on the fly, when cimporting the .pxd file. Or
maybe it's better to just queue up the cimported files that need to get
compiled, and then compile them after compiling the main source file. For
the client side of the cimport mechanism, parsing the .pxd file is enough
to figure out what it will eventually export, so there is no actual need to
generate a .py file for it at the same time. I think the compilation will
be a separate operation anyway, because the .pxd file is currently parsed
into the context of the .pyx file being compiled. Hacking an additional
compilation onto that step may not be trivial.
I don't mind either way, i.e. if it's done while cimporting the .pxd or
afterwards.
In any case, this means that compiling a .pyx file can potentially generate
multiple .py files, which means that there should be a way to reuse already
generated files from other compiler runs (i.e. if you compile multiple .pyx
files which cimport the same .pxd files, then each .pxd file only needs to
get compiled once). Basically, this means that the compiler would check if
there is already a .py file for a .pxd, and not generate it in that case.
However, this is problematic, because we do not necessarily know if the .py
file was user provided or generated...
This makes me think that the ctypes backend may generally want to generate
its output files into a separate target *directory*, rather than just
providing a target file name.
Stefan
_______________________________________________
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
