[Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Manuel Nuno Melo]

Hello devs,

I'm developing the setup.py for a scientific package, MDAnalysis
 (see PR #799
). We depend on
distutils and setuptool. Namely, we use setuptools.extension.Extension
class for our extensions.

Some older versions of setuptools (<18.0) do filename cythonization
themselves upon initialization of the Extension object.

Because we want to control name cythonization ourselves I try to directly
use distutils.extension.Extension, which has none of setuptools'
cythonization. However, this doesn't work because setuptools patches
distutils, so that distutils.extension.Extension effectively becomes
setuptools.extension.Extension.

setuptools does provide a copy of the unpatched Extension class, via
setuptools.extension._Extension. Yet, if I attempt to pass instances of
this class to Cython.cythonize it fails because my extension parent no
longer conforms to one of the expected classes:

  File "./setup.py", line 283, in cythonize
new_ext_modules = Cython.cythonize(self.ext_modules)
  File
"/scratch/virtualenv/lib/python2.7/site-packages/Cython-0.24-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py",
line 796, in cythonize
aliases=aliases)
  File
"/scratch/virtualenv/lib/python2.7/site-packages/Cython-0.24-py2.7-linux-x86_64.egg/Cython/Build/Dependencies.py",
line 686, in create_extension_list
raise TypeError(msg)
TypeError: pattern is not of type str nor subclass of Extension () but of type  and class distutils.extension.Extension

I believe cythonize should be more permissive with the extension parent
classes it accepts, or at least have the setuptools _Extension one in the
whitelist, since it's the only way to access the original distutils class.
What do you think?

In the meantime I'll solve this by subclassing
setuptools.extension.Extension and making sure it doesn't cythonize
anything.

Cheers,
Manuel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Erik Bray]

On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo
 wrote:
> Hello devs,
>
> I'm developing the setup.py for a scientific package, MDAnalysis (see PR
> #799). We depend on distutils and setuptool. Namely, we use
> setuptools.extension.Extension class for our extensions.
>
> Some older versions of setuptools (<18.0) do filename cythonization
> themselves upon initialization of the Extension object.
>
> Because we want to control name cythonization ourselves I try to directly
> use distutils.extension.Extension, which has none of setuptools'
> cythonization. However, this doesn't work because setuptools patches
> distutils, so that distutils.extension.Extension effectively becomes
> setuptools.extension.Extension.

I'm wondering what it is specifically you need to do in your
subclass--might it still be possible to do with a subclass of the
setuptools Extension?  Not saying I disagree with the overall idea,
but I also wonder if there isn't a better way.

Erik
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] PEP 509: detect dict modification by version tag

[Stefan Behnel]

Hi!

This new PEP seems interesting for Cython optimisations, too:

https://www.python.org/dev/peps/pep-0509/

Essentially, it adds a 64 bit modification counter to dicts that allows
detecting unmodified dicts, e.g. during lookups of methods or globals.

It's currently proposed for Py3.6.

Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Matthew Brett]

On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray  wrote:
> On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo
>  wrote:
>> Hello devs,
>>
>> I'm developing the setup.py for a scientific package, MDAnalysis (see PR
>> #799). We depend on distutils and setuptool. Namely, we use
>> setuptools.extension.Extension class for our extensions.
>>
>> Some older versions of setuptools (<18.0) do filename cythonization
>> themselves upon initialization of the Extension object.
>>
>> Because we want to control name cythonization ourselves I try to directly
>> use distutils.extension.Extension, which has none of setuptools'
>> cythonization. However, this doesn't work because setuptools patches
>> distutils, so that distutils.extension.Extension effectively becomes
>> setuptools.extension.Extension.
>
> I'm wondering what it is specifically you need to do in your
> subclass--might it still be possible to do with a subclass of the
> setuptools Extension?  Not saying I disagree with the overall idea,
> but I also wonder if there isn't a better way.

I know this is a terrible and ugly hack, but the projects I work in
have a 'fake_pyrex' directory, that fools setuptools into thinking
that 'pyrex' is installed, and therefore prevents it from doing the
.pyx -> .c filename conversions in the extension:

https://github.com/regreg/regreg/blob/master/setup.py#L33

Cheers,

Matthew
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Manuel Nuno Melo]

Our need to control cythonization comes from the fact that we implement
cython as a lazy and optional dependency. Lazy in the sense that we delay
as much as possible cythonization so that setuptools or pip have time to
install cython, if needed. Optional because we distribute both .pyx and
cythonized .c files, and decide on which to use based on user flags.

We, therefore, need an Extension class that only cythonizes if we decide to.

Thanks for the feedback,
Manel

On Apr 14, 2016 8:17 PM, "Matthew Brett"  wrote:
>
> On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray  wrote:
> > On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo
> >  wrote:
> >> Hello devs,
> >>
> >> I'm developing the setup.py for a scientific package, MDAnalysis (see
PR
> >> #799). We depend on distutils and setuptool. Namely, we use
> >> setuptools.extension.Extension class for our extensions.
> >>
> >> Some older versions of setuptools (<18.0) do filename cythonization
> >> themselves upon initialization of the Extension object.
> >>
> >> Because we want to control name cythonization ourselves I try to
directly
> >> use distutils.extension.Extension, which has none of setuptools'
> >> cythonization. However, this doesn't work because setuptools patches
> >> distutils, so that distutils.extension.Extension effectively becomes
> >> setuptools.extension.Extension.
> >
> > I'm wondering what it is specifically you need to do in your
> > subclass--might it still be possible to do with a subclass of the
> > setuptools Extension?  Not saying I disagree with the overall idea,
> > but I also wonder if there isn't a better way.
>
> I know this is a terrible and ugly hack, but the projects I work in
> have a 'fake_pyrex' directory, that fools setuptools into thinking
> that 'pyrex' is installed, and therefore prevents it from doing the
> .pyx -> .c filename conversions in the extension:
>
> https://github.com/regreg/regreg/blob/master/setup.py#L33
>
> Cheers,
>
> Matthew
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] PEP 509: detect dict modification by version tag

[Robert Bradshaw]

Cool! This was essentially my answer for "if you were to propose a Python
PEP, what would it be" on the podcast. We should get Cython listed as a
potential user as well.

On Thu, Apr 14, 2016 at 9:15 AM, Stefan Behnel  wrote:

> Hi!
>
> This new PEP seems interesting for Cython optimisations, too:
>
> https://www.python.org/dev/peps/pep-0509/
>
> Essentially, it adds a 64 bit modification counter to dicts that allows
> detecting unmodified dicts, e.g. during lookups of methods or globals.
>
> It's currently proposed for Py3.6.
>
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Robert Bradshaw]

I agree that we shouldn't disallow distutils' Extensions just because
setuptools was imported.

https://github.com/cython/cython/commit/d804bd2d8d9aac04b92d4bfb8dbc7f8a4c8079ac

On Thu, Apr 14, 2016 at 12:07 PM, Manuel Nuno Melo <
manuel.nuno.m...@gmail.com> wrote:

> Our need to control cythonization comes from the fact that we implement
> cython as a lazy and optional dependency. Lazy in the sense that we delay
> as much as possible cythonization so that setuptools or pip have time to
> install cython, if needed. Optional because we distribute both .pyx and
> cythonized .c files, and decide on which to use based on user flags.
>
> We, therefore, need an Extension class that only cythonizes if we decide
> to.
>
> Thanks for the feedback,
> Manel
>
> On Apr 14, 2016 8:17 PM, "Matthew Brett"  wrote:
> >
> > On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray 
> wrote:
> > > On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo
> > >  wrote:
> > >> Hello devs,
> > >>
> > >> I'm developing the setup.py for a scientific package, MDAnalysis (see
> PR
> > >> #799). We depend on distutils and setuptool. Namely, we use
> > >> setuptools.extension.Extension class for our extensions.
> > >>
> > >> Some older versions of setuptools (<18.0) do filename cythonization
> > >> themselves upon initialization of the Extension object.
> > >>
> > >> Because we want to control name cythonization ourselves I try to
> directly
> > >> use distutils.extension.Extension, which has none of setuptools'
> > >> cythonization. However, this doesn't work because setuptools patches
> > >> distutils, so that distutils.extension.Extension effectively becomes
> > >> setuptools.extension.Extension.
> > >
> > > I'm wondering what it is specifically you need to do in your
> > > subclass--might it still be possible to do with a subclass of the
> > > setuptools Extension?  Not saying I disagree with the overall idea,
> > > but I also wonder if there isn't a better way.
> >
> > I know this is a terrible and ugly hack, but the projects I work in
> > have a 'fake_pyrex' directory, that fools setuptools into thinking
> > that 'pyrex' is installed, and therefore prevents it from doing the
> > .pyx -> .c filename conversions in the extension:
> >
> > https://github.com/regreg/regreg/blob/master/setup.py#L33
> >
> > Cheers,
> >
> > Matthew
> > ___
> > cython-devel mailing list
> > cython-devel@python.org
> > https://mail.python.org/mailman/listinfo/cython-devel
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel