Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension
On Sat, Apr 16, 2016 at 1:29 PM, Manuel Nuno Melo wrote: > Hi Erik, > Please post your solution; I'm curious to see it. Will do in a bit. I need to see if I can distill it somewhat from its original context so that it can be better understood. > Currently, we're also using setup_requires but are moving away from it > because of two main issues: > > 1- there's no flexibility to hook into it via setuptools.Distribution > subclassing (unless you rewrite the entire __init__); I don't really think that's a big issue but maybe you've encountered some reason that it is. The worse thing about it is the general chicken-egg problem that you need to call setup() to get your setup_requires, but you may need some packages pulled in from setup_requires in order to do anything useful with setup(). d2to1 [1] solves this problem quite well, but unfortunately development on it is stalled for the time being. Another alternative that can work quite well is some version of Daniel Holth's "setup-requires" hack [2]. > 2- (and more serious) setuptools installs setup_requires dependencies in the > local directory, not as a system-wide install. That's a feature, not a bug. Build requirements for a package might be in conflict with the versions of those same requirements you have already installed. It also prevents things from being installed that are *not* required at runtime. It makes it easier for projects to pin a little more strictly to their known-working build requirements, without the constraints imposed by supporting a wider range of runtime requirements. > Even worse, it then puts a > path link to the current directory under the system-wide easy_install.pth. > This means that if you install from the source directory via 'sudo > ./setup.py install', you get a Cython egg downloaded into that directory, > and its path added to sys.path (!!). Needless to say this can break in many > nasty ways... Huh?? I'm not sure we're talking about the same thing now. The local .eggs cache directory created for setup_requires downloads most certainly does not get added to easy_install.pth. Are you sure you're not thinking of `setup.py develop`? [1] https://pypi.python.org/pypi/d2to1 [2] https://bitbucket.org/dholth/setup-requires/src > On Sat, Apr 16, 2016 at 1:10 PM, Erik Bray wrote: >> >> On Apr 14, 2016 21:07, "Manuel Nuno Melo" >> wrote: >> > >> > Our need to control cythonization comes from the fact that we implement >> > cython as a lazy and optional dependency. Lazy in the sense that we delay >> > as >> > much as possible cythonization so that setuptools or pip have time to >> > install cython, if needed. Optional because we distribute both .pyx and >> > cythonized .c files, and decide on which to use based on user flags. >> > >> > We, therefore, need an Extension class that only cythonizes if we decide >> > to. >> > >> > Thanks for the feedback, >> > Manel >> >> I have a solution for exactly this in astropy_helpers which makes it >> possible, for example, to pull in Cython via setup_requires, but only if >> it's needed. >> >> Remind me on Monday and I can point out how it works. >> >> > On Apr 14, 2016 8:17 PM, "Matthew Brett" >> > wrote: >> > > >> > > On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray >> > > wrote: >> > > > On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo >> > > > wrote: >> > > >> Hello devs, >> > > >> >> > > >> I'm developing the setup.py for a scientific package, MDAnalysis >> > > >> (see PR >> > > >> #799). We depend on distutils and setuptool. Namely, we use >> > > >> setuptools.extension.Extension class for our extensions. >> > > >> >> > > >> Some older versions of setuptools (<18.0) do filename cythonization >> > > >> themselves upon initialization of the Extension object. >> > > >> >> > > >> Because we want to control name cythonization ourselves I try to >> > > >> directly >> > > >> use distutils.extension.Extension, which has none of setuptools' >> > > >> cythonization. However, this doesn't work because setuptools >> > > >> patches >> > > >> distutils, so that distutils.extension.Extension effectively >> > > >> becomes >> > > >> setuptools.extension.Extension. >> > > > >> > > > I'm wondering what it is specifically you need to do in your >> > > > subclass--might it still be possible to do with a subclass of the >> > > > setuptools Extension? Not saying I disagree with the overall idea, >> > > > but I also wonder if there isn't a better way. >> > > >> > > I know this is a terrible and ugly hack, but the projects I work in >> > > have a 'fake_pyrex' directory, that fools setuptools into thinking >> > > that 'pyrex' is installed, and therefore prevents it from doing the >> > > .pyx -> .c filename conversions in the extension: >> > > >> > > https://github.com/regreg/regreg/blob/master/setup.py#L33 >> > > >> > > Cheers, >> > > >> > > Matthew >> > > ___ >> > > cython-devel mailing list >> > > cython-devel@python.org >> > > https://mail.python.org/mai
Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension
Ah, sorry Erik, you're absolutely right. I mixed my results a bit and must elaborate: 'setup_requires' on its own will indeed not generate the behavior I described. However, if you define the same dependency under 'setup_requires' AND 'install_requires', then you get the mess I mentioned. Essentially, when you reach 'install_requires' setuptools decides that the egg it has under .eggs suffices, and just links to it. This isn't an issue when depending on cython because it's really just a setup-time dependency. However, in our package we depend on numpy at both setup- and runtime, and therefore makes sense to have it under both 'requires' flags. This is why hooking in at the setup_requires step could be a potential workaround, if we could get it to do a full dependency install instead of local egg. I'm currently finishing up this approach. On Mon, Apr 18, 2016 at 11:16 AM, Erik Bray wrote: > On Sat, Apr 16, 2016 at 1:29 PM, Manuel Nuno Melo > wrote: > > Hi Erik, > > Please post your solution; I'm curious to see it. > > Will do in a bit. I need to see if I can distill it somewhat from its > original context so that it can be better understood. > > > Currently, we're also using setup_requires but are moving away from it > > because of two main issues: > > > > 1- there's no flexibility to hook into it via setuptools.Distribution > > subclassing (unless you rewrite the entire __init__); > > I don't really think that's a big issue but maybe you've encountered > some reason that it is. > The worse thing about it is the general chicken-egg problem that you > need to call setup() to get your setup_requires, but you may need some > packages pulled in from setup_requires in order to do anything useful > with setup(). > > d2to1 [1] solves this problem quite well, but unfortunately > development on it is stalled for the time being. Another alternative > that can work quite well is some version of Daniel Holth's > "setup-requires" hack [2]. > > > 2- (and more serious) setuptools installs setup_requires dependencies in > the > > local directory, not as a system-wide install. > > That's a feature, not a bug. Build requirements for a package might > be in conflict with the versions of those same requirements you have > already installed. It also prevents things from being installed that > are *not* required at runtime. It makes it easier for projects to pin > a little more strictly to their known-working build requirements, > without the constraints imposed by supporting a wider range of runtime > requirements. > > > Even worse, it then puts a > > path link to the current directory under the system-wide > easy_install.pth. > > This means that if you install from the source directory via 'sudo > > ./setup.py install', you get a Cython egg downloaded into that directory, > > and its path added to sys.path (!!). Needless to say this can break in > many > > nasty ways... > > Huh?? I'm not sure we're talking about the same thing now. The local > .eggs cache directory created for setup_requires downloads most > certainly does not get added to easy_install.pth. Are you sure you're > not thinking of `setup.py develop`? > > [1] https://pypi.python.org/pypi/d2to1 > [2] https://bitbucket.org/dholth/setup-requires/src > > > On Sat, Apr 16, 2016 at 1:10 PM, Erik Bray > wrote: > >> > >> On Apr 14, 2016 21:07, "Manuel Nuno Melo" > >> wrote: > >> > > >> > Our need to control cythonization comes from the fact that we > implement > >> > cython as a lazy and optional dependency. Lazy in the sense that we > delay as > >> > much as possible cythonization so that setuptools or pip have time to > >> > install cython, if needed. Optional because we distribute both .pyx > and > >> > cythonized .c files, and decide on which to use based on user flags. > >> > > >> > We, therefore, need an Extension class that only cythonizes if we > decide > >> > to. > >> > > >> > Thanks for the feedback, > >> > Manel > >> > >> I have a solution for exactly this in astropy_helpers which makes it > >> possible, for example, to pull in Cython via setup_requires, but only if > >> it's needed. > >> > >> Remind me on Monday and I can point out how it works. > >> > >> > On Apr 14, 2016 8:17 PM, "Matthew Brett" > >> > wrote: > >> > > > >> > > On Thu, Apr 14, 2016 at 6:08 AM, Erik Bray > >> > > wrote: > >> > > > On Wed, Apr 13, 2016 at 9:35 PM, Manuel Nuno Melo > >> > > > wrote: > >> > > >> Hello devs, > >> > > >> > >> > > >> I'm developing the setup.py for a scientific package, MDAnalysis > >> > > >> (see PR > >> > > >> #799). We depend on distutils and setuptool. Namely, we use > >> > > >> setuptools.extension.Extension class for our extensions. > >> > > >> > >> > > >> Some older versions of setuptools (<18.0) do filename > cythonization > >> > > >> themselves upon initialization of the Extension object. > >> > > >> > >> > > >> Because we want to control name cythonization ourselves I try to > >> > > >> directly > >> > > >> use distutils.extension.Exten
