[Cython] import-free setuptools integration

[Nils Werner]

I have implemented some very simple setuptools integration that circumvents
the requirement to `import cython` in `setup.py` entirely, doing away with
the
century-old problem of having to importing requirements during installation:



This implementation is inspired by CFFI's setuptools integration:
.

They do not require you to `import cffi` but simply define an additional
keyword
argument `cffi_modules` for `setup()`. This additional keyword argument
_does not
raise errors when cffi is not installed yet_ but can be used once cffi is
there
(as defined in `setup_requires`). Later setuptools will call cffi and have
it do
whatever it wants with the contents of the argument.

Cython can do the exact same. Below you have the usual `setup.py`, with the
chicken and egg `import` problem and a few `Extensions` that need to be
`cythonize`d:

import setuptools
from Cython.Build import cythonize  # this will fail if cython is not
present prior to running `pip install this`
from setuptools.extension import Extension

extensions = [
Extension("fib", ["fib.pyx"]),
Extension("fib2", ["fib2.pyx"]),
]

setuptools.setup(
name='example',
version="1.0.0",
packages=setuptools.find_packages(),
setup_requires=['cython'],
install_requires=['cython'],
ext_modules=cythonize(extensions),
)

With the changes I am proposing the usual

ext_modules=cythonize(extensions),

can be replaced with

cython_modules=extensions,

removing the need for `from Cython.Build import cythonize` and solving the
import problem:

import setuptools
from setuptools.extension import Extension

extensions = [
Extension("fib", ["fib.pyx"]),
Extension("fib2", ["fib2.pyx"]),
]

setuptools.setup(
name='example',
version="1.0.0",
setup_requires=['cython'],  # we must later require a Cython
version that has this kind of integration
install_requires=['cython'],
cython_modules=extensions,
)

This allows a nicer automated installation of tools that depend on cython
and
also allows end users to keep their `setup.py` cleaner and leaner.

Additionally, I have also extended `cythonize()` a bit and thus allow for
definitions like:

import setuptools
from setuptools.extension import Extension

extensions = [
Extension("fib", ["fib.c"]),
Extension("fib2", ["fib2.c"]),
]

setuptools.setup(
name='example',
version="1.0.0",
extras_require={'cython': ['cython']},
ext_modules=extensions,
cython_modules=extensions,
)

This will automatically compile pyx->c if cython is installed and a
re-compilation is
needed. Otherwise it will merely compile c->so.

This allows devs to ship the generated C-code (as you suggest per your
docs) but
takes the heuristic to find out whether to compile pyx->C->so or only C->so
completely out of `setup.py` and under control of cython.

The change is written to be backwards compatible:

 - Using `cython_modules` is entirely optional and not using it is
sideeffect-free
 - Having `cythonize()` internally automatically compile *.pyx files when
*.c files
   are provided is disabled by default and must be enabled using the
`replace_extension`
   keyword argument (`cython_modules` enables that option).
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] import-free setuptools integration

[Nils Werner]

I understand. Maybe it is then better to remove the extension-swapping and
ask the developer to specifically decide which extension is a cython_module
and which is an ext_module (and potentially have duplicate definitions of
the same extension):

setuptools.setup(
# ...
ext_modules=[
Extension("pure_c", ["pure_c.c"]),
Extension("mixed", ["mixed.c"]),
],
cython_modules=[
Extension("mixed", ["mixed.pyx"]),
Extension("pure_pyx", ["pure_pyx.pyx"]),
],
)

In this case

 - `pure_c` will never automatically be cythonized
 - `mixed` may be cythonized
 - `pure_pyx` may be cythonized

`pure_c` needs the developer to run `cythonize pure_c.pyx` before `python
setup.py build_sdist`. Shipping a package without the resulting .c file
will fail during installation.

`mixed` may come with both .c and .pyx files

 - If cython is not installed setuptools will simply compile the C code.
 - If cython is installed it is up to `cythonize()` to decide if it wants
to re-compile pyx->c first.

`pure_pyx` is similar to `mixed` but it may run into non-obvious problems
if cython is not installed and not defined as a `setup_requirement`: *It
will silently not be built* and be missing from the final installation. I
guess this won't be much of a problem though as it will most probably be
found out quickly and be fixed with a simple additional line in setup.py.



It might be a bit beyond the scope of what I am suggesting here but I have
also been experimenting with adding another keyword argument and another
command to setuptools:

setuptools.setup(
# ...
cython_manual_modules=[
Extension("pure_c", ["pure_c.pyx"]),
],
)

Combined with

python setup.py build_pyx

we could then automate the cythonization calls during development. This
removes the overhead of remembering each of your pyx files from the
developer and encourages them to *not write cython-related scripting logic*
in their setup.py.

In your case (you cythonize before sdist and dont want users to cythonize
during install) you would define your package as

setuptools.setup(
# ...
ext_modules=[
Extension("mixed", ["mixed.c"]),
],
cython_manual_modules=[
Extension("mixed", ["mixed.pyx"]),
],
)

And would run

python setup.py build_pyx sdist

to create a distribution file.

Again, all these changes are backwards compatible as none of the current
logic needs to be changed (the additional keyword argument and the command
class have been appended to the GitHub PR).

On Sun, Feb 7, 2016 at 9:38 PM, Greg Ewing 
wrote:

> Nils Werner wrote:
>
>>  - Having `cythonize()` internally automatically compile *.pyx files when
>> *.c files
>>are provided is disabled by default and must be enabled using the
>> `replace_extension`
>>keyword argument (`cython_modules` enables that option).
>>
>
> I'd be happier if it were still disabled by default even then.
> If I'm simply installing a package, I do *not* want it to
> try to recompile any .pyx files just because I happen to have
> cython installed.
>
> I know that isn't supposed to happen unless the .pyx is newer
> than the .c, but I wouldn't like to rely on that. There's a
> big difference between using a package and modifying it, and
> I'd rather be explicit about when I'm doing the latter.
>
> --
> Greg
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel