mark florisson, 04.10.2011 23:19:
So I propose that after fused types gets merged we try to move as many utility codes as possible to their utility code files (unless they are used in pending pull requests or other branches). Preferably this will be done in one or a few commits. How should we split up the work
I would propose that new utility code gets moved out into utility files right away (if doable, given the current state of the infrastructure), and that existing utility code gets moves when it gets modified or when someone feels like it. Until we really get to the point of wanting to create a separate shared library etc., there's no need to hurry with the move.
We could actually move things before fused types get merged, as long as we don't touch binding_cfunc_utility_code.
Another reason not to hurry, right?
Before we go there, Stefan, do we still want to implement the header .ini style which can list dependencies and such?
I think we'll eventually need that, but that also depends a bit on the question whether we want to (or can) build a shared library or not. See below.
Another issue is that Cython compile time is increasing with the addition of control flow and cython utilities. If you use fused types you're also going to combinatorially add more compile time.
I don't see that locally - a compiled Cython is hugely fast for me. In comparison, the C compiler literally takes ages to compile the result. An external shared library may or may not help with both - in particular, it is not clear to me what makes the C compiler slow. If the compile time is dominated by the number of inlined functions (which is not unlikely), a shared library + header file will not make a difference.
I'm sure this came up earlier, but I really think we should have a libcython and a cython.h. libcython (a shared library) should contain any common Cython-specific code not meant to be inlined, and cython.h any types, macros and inline functions etc.
This has a couple of implications though. In order to support this on the user side, we have to build one shared library per installed package in order to avoid any Cython versioning issues. Just installing a versioned "libcython_x.y.z.so" globally isn't enough, especially during development, but also at deployment time. Different packages may use different CFLAGS or Cython options, which may have an impact on the result. Encoding all possible factors in the file name will be cumbersome and may mean that we still end up with a number of installed Cython libraries that correlates with the number of installed Cython based packages.
Next, we may not know at build time which set of Cython modules is in the package. This may be less of an issue if we rely on "cythonize()" in setup.py to compile all modules before hand (assuming that the user doesn't call it twice, once for *.pyx, once for *.py, for example), but even if we know all modules, we'd still have to figure out the complete set of utility code used by all modules in order to build an adapted library with only the necessary code used in the package. So we'd always end up with a complete library with all utility code, which is only really interesting for larger packages with several Cython modules.
I agree with Robert that a CEP would be needed for this, both for clearing up the implications and actual use cases (I know that Sage is a reasonable use case, but it's also a rather special case).
This will decrease Cython and C compile time, and will also make executables smaller.
I don't see how this actually impacts executables. However, a self-contained executable is a value in itself.
This could be enabled using a command line option to Cython, as well as with distutils, eventually we may decide to make it the default (lets figure that out later). Preferably libcython.so would be installed alongside libpython.so and cython.h inside the Python include directory.
I don't see this happening. It's easy for Python (there is only one Python running at a time, with one libpython loaded), but it's a lot less safe for different versions of a Cython library that are used by different modules inside of the running Python. For example, we'd have to version all visible symbols in operating systems with flat namespaces, in order to support loading multiple versions of the library.
Lastly, I think we also should figure out a way to serialize Entry objects from CythonUtilities, which could easily and swiftly be loaded when creating the cython scope. It's quite a pain to declare all entries for utilities you write manually
Why would you declare them manually? I thought everything would be moved out into the utility code files?
so what I mostly did was parse the utility up to and including AnalyseDeclarationsTransform, and then retrieve the entries from there.
Sounds like a drawback regarding the processing time, but may still be a reasonable way to do it. I would expect that it won't be hard to pickle the resulting dict of entries into a cache file and rebuild it only when one of the utility files changes.
Stefan _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
