Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Erik Bray]

On Mon, Apr 18, 2016 at 6:56 PM, Manuel Nuno Melo
 wrote:
> Ah, sorry Erik, you're absolutely right. I mixed my results a bit and must
> elaborate:
>
> 'setup_requires' on its own will indeed not generate the behavior I
> described.
>
> However, if you define the same dependency under 'setup_requires' AND
> 'install_requires', then you get the mess I mentioned. Essentially, when you
> reach 'install_requires' setuptools decides that the egg it has under .eggs
> suffices, and just links to it.
>
> This isn't an issue when depending on cython because it's really just a
> setup-time dependency. However, in our package we depend on numpy at both
> setup- and runtime, and therefore makes sense to have it under both
> 'requires' flags.
>
> This is why hooking in at the setup_requires step could be a potential
> workaround, if we could get it to do a full dependency install instead of
> local egg. I'm currently finishing up this approach.

Interesting.  This sounds to me like a regression, as I don't think it
used to do that.  This should probably be brought up on distutils-sig
or in the setuptools bug tracker, rather than spending a lot of time
hacking together a workaround or requiring changes specifically in
Cython (though I don't disagree that distutils.Extension should be
allowed in the first place).

IIRC it used to be that if the same package was specified in both
setup_requires and install_requires it would be built/installed twice.
This was recognized as inefficient, and that the copy installed in the
local .eggs directory satisfies an install_requires requirement it
should be copied from .eggs to the system site-packages (or whatever
the install target is).  I remember there at least being *discussion*
about that, but not sure what changes were made.

It sounds like in your case it's just adding Cython from your .eggs
into easy-install.pth which I agree is wrong.  If that's the case it
should be fixed in setuptools over all else.  I'll see if I can
reproduce it.  Jason Coombs who mantains setuptools is usually quick
to release bug fix versions if he gets a patch.

Erik

> On Mon, Apr 18, 2016 at 11:16 AM, Erik Bray  wrote:
>>
>> On Sat, Apr 16, 2016 at 1:29 PM, Manuel Nuno Melo
>>  wrote:
>> > Hi Erik,
>> > Please post your solution; I'm curious to see it.
>>
>> Will do in a bit. I need to see if I can distill it somewhat from its
>> original context so that it can be better understood.
>>
>> > Currently, we're also using setup_requires but are moving away from it
>> > because of two main issues:
>> >
>> > 1- there's no flexibility to hook into it via setuptools.Distribution
>> > subclassing (unless you rewrite the entire __init__);
>>
>> I don't really think that's a big issue but maybe you've encountered
>> some reason that it is.
>> The worse thing about it is the general chicken-egg problem that you
>> need to call setup() to get your setup_requires, but you may need some
>> packages pulled in from setup_requires in order to do anything useful
>> with setup().
>>
>> d2to1 [1] solves this problem quite well, but unfortunately
>> development on it is stalled for the time being.  Another alternative
>> that can work quite well is some version of Daniel Holth's
>> "setup-requires" hack [2].
>>
>> > 2- (and more serious) setuptools installs setup_requires dependencies in
>> > the
>> > local directory, not as a system-wide install.
>>
>> That's a feature, not a bug.  Build requirements for a package might
>> be in conflict with the versions of those same requirements you have
>> already installed.  It also prevents things from being installed that
>> are *not* required at runtime.  It makes it easier for projects to pin
>> a little more strictly to their known-working build requirements,
>> without the constraints imposed by supporting a wider range of runtime
>> requirements.
>>
>> > Even worse, it then puts a
>> > path link to the current directory under the system-wide
>> > easy_install.pth.
>> > This means that if you install from the source directory via 'sudo
>> > ./setup.py install', you get a Cython egg downloaded into that
>> > directory,
>> > and its path added to sys.path (!!). Needless to say this can break in
>> > many
>> > nasty ways...
>>
>> Huh??  I'm not sure we're talking about the same thing now.  The local
>> .eggs cache directory created for setup_requires downloads most
>> certainly does not get added to easy_install.pth.  Are you sure you're
>> not thinking of `setup.py develop`?
>>
>> [1] https://pypi.python.org/pypi/d2to1
>> [2] https://bitbucket.org/dholth/setup-requires/src
>>
>> > On Sat, Apr 16, 2016 at 1:10 PM, Erik Bray 
>> > wrote:
>> >>
>> >> On Apr 14, 2016 21:07, "Manuel Nuno Melo" 
>> >> wrote:
>> >> >
>> >> > Our need to control cythonization comes from the fact that we
>> >> > implement
>> >> > cython as a lazy and optional dependency. Lazy in the sense that we
>> >> > delay as
>> >> > much as possible cythonization so that setuptools or pip have time to
>> >> > insta

[Cython] unsupported meta-programming-related features

[Xuancong Wang]

Dear Cython developers,

Python supports meta-programming, in which a variable with name
specified in a string can be created at run-time. One built-in library
which make use of this is argparse.

For example:

parser.add_argument('-N', '--max_threads', help='maximum number of
concurrent decoding threads', type=int, default=16)

in this case, the variable max_threads is created from the string
argument. And then Cython will generate an incorrect C program with
the following error:

smt.py:78:88: undeclared name not builtin: headtail
smt.c:1:2: error: #error Do not use this file, it is the result of a
failed Cython compilation.

In comparison, I found that nuitka can convert this kind of Python
programs sucessfully. I hope Cython can be improved. Thanks!

Cheers,
Xuancong
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cannot cythonize subclasses of setuptools.extension._Extension

[Manuel Nuno Melo]

This strayed a bit from the original topic, but yes, I should bring this up
with setuptools project. Thanks for the feedback.
(My feeling was that this was a somewhat stalled problem, reported in two
issues and persisting after a couple of years:
#209

and #391

)

As you said, even if it is a hackish workaround here, Cython should still
allow setuptools.extension._Extension.

On Tue, Apr 19, 2016 at 11:48 AM, Erik Bray  wrote:

> On Mon, Apr 18, 2016 at 6:56 PM, Manuel Nuno Melo
>  wrote:
> > Ah, sorry Erik, you're absolutely right. I mixed my results a bit and
> must
> > elaborate:
> >
> > 'setup_requires' on its own will indeed not generate the behavior I
> > described.
> >
> > However, if you define the same dependency under 'setup_requires' AND
> > 'install_requires', then you get the mess I mentioned. Essentially, when
> you
> > reach 'install_requires' setuptools decides that the egg it has under
> .eggs
> > suffices, and just links to it.
> >
> > This isn't an issue when depending on cython because it's really just a
> > setup-time dependency. However, in our package we depend on numpy at both
> > setup- and runtime, and therefore makes sense to have it under both
> > 'requires' flags.
> >
> > This is why hooking in at the setup_requires step could be a potential
> > workaround, if we could get it to do a full dependency install instead of
> > local egg. I'm currently finishing up this approach.
>
> Interesting.  This sounds to me like a regression, as I don't think it
> used to do that.  This should probably be brought up on distutils-sig
> or in the setuptools bug tracker, rather than spending a lot of time
> hacking together a workaround or requiring changes specifically in
> Cython (though I don't disagree that distutils.Extension should be
> allowed in the first place).
>
> IIRC it used to be that if the same package was specified in both
> setup_requires and install_requires it would be built/installed twice.
> This was recognized as inefficient, and that the copy installed in the
> local .eggs directory satisfies an install_requires requirement it
> should be copied from .eggs to the system site-packages (or whatever
> the install target is).  I remember there at least being *discussion*
> about that, but not sure what changes were made.
>
> It sounds like in your case it's just adding Cython from your .eggs
> into easy-install.pth which I agree is wrong.  If that's the case it
> should be fixed in setuptools over all else.  I'll see if I can
> reproduce it.  Jason Coombs who mantains setuptools is usually quick
> to release bug fix versions if he gets a patch.
>
> Erik
>
> > On Mon, Apr 18, 2016 at 11:16 AM, Erik Bray 
> wrote:
> >>
> >> On Sat, Apr 16, 2016 at 1:29 PM, Manuel Nuno Melo
> >>  wrote:
> >> > Hi Erik,
> >> > Please post your solution; I'm curious to see it.
> >>
> >> Will do in a bit. I need to see if I can distill it somewhat from its
> >> original context so that it can be better understood.
> >>
> >> > Currently, we're also using setup_requires but are moving away from it
> >> > because of two main issues:
> >> >
> >> > 1- there's no flexibility to hook into it via setuptools.Distribution
> >> > subclassing (unless you rewrite the entire __init__);
> >>
> >> I don't really think that's a big issue but maybe you've encountered
> >> some reason that it is.
> >> The worse thing about it is the general chicken-egg problem that you
> >> need to call setup() to get your setup_requires, but you may need some
> >> packages pulled in from setup_requires in order to do anything useful
> >> with setup().
> >>
> >> d2to1 [1] solves this problem quite well, but unfortunately
> >> development on it is stalled for the time being.  Another alternative
> >> that can work quite well is some version of Daniel Holth's
> >> "setup-requires" hack [2].
> >>
> >> > 2- (and more serious) setuptools installs setup_requires dependencies
> in
> >> > the
> >> > local directory, not as a system-wide install.
> >>
> >> That's a feature, not a bug.  Build requirements for a package might
> >> be in conflict with the versions of those same requirements you have
> >> already installed.  It also prevents things from being installed that
> >> are *not* required at runtime.  It makes it easier for projects to pin
> >> a little more strictly to their known-working build requirements,
> >> without the constraints imposed by supporting a wider range of runtime
> >> requirements.
> >>
> >> > Even worse, it then puts a
> >> > path link to the current directory under the system-wide
> >> > easy_install.pth.
> >> > This means that if you install from the source directory via 'sudo
> >> > ./setup.py install', you get a Cython egg downloaded into that
> >> > directory,
> >> > and its path added to sys.path (!!). Needless to say this can b

Re: [Cython] unsupported meta-programming-related features

[Robert Bradshaw]

On Tue, Apr 19, 2016 at 2:13 AM, Xuancong Wang  wrote:

> Dear Cython developers,
>
> Python supports meta-programming, in which a variable with name
> specified in a string can be created at run-time. One built-in library
> which make use of this is argparse.
>
> For example:
>
> parser.add_argument('-N', '--max_threads', help='maximum number of
> concurrent decoding threads', type=int, default=16)
>
> in this case, the variable max_threads is created from the string
> argument. And then Cython will generate an incorrect C program with
> the following error:
>
> smt.py:78:88: undeclared name not builtin: headtail
> smt.c:1:2: error: #error Do not use this file, it is the result of a
> failed Cython compilation.


Argparse works just fine in Cython

import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-N', '--max_threads', help='maximum number of
concurrent decoding threads', type=int, default=16)
args = parser.parse_args()
print args.max_threads

Could you provide a full example that you think should work? Where is
headtail defined? Is it another argument?


> In comparison, I found that nuitka can convert this kind of Python
> programs sucessfully. I hope Cython can be improved. Thanks!
>

argsparse dynamically adds the attributes to the returned object, which
works fine in Cython. Unless you're doing something like

http://stackoverflow.com/questions/19299635/python-argparse-parse-args-into-global-namespace-or-a-reason-this-is-a-bad-idea

That's one of the few places we diverge, because people strongly preferred
compile-time errors to runtime errors in this case.
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] unsupported meta-programming-related features

[Stefan Behnel]

Xuancong Wang schrieb am 19.04.2016 um 11:13:
> Python supports meta-programming, in which a variable with name
> specified in a string can be created at run-time. One built-in library
> which make use of this is argparse.
> 
> For example:
> 
> parser.add_argument('-N', '--max_threads', help='maximum number of
> concurrent decoding threads', type=int, default=16)
> 
> in this case, the variable max_threads is created from the string
> argument.

Actually, no. What argparse returns on parsing is an object with
dynamically created attributes. It doesn't magically inject any global
variables into your module namespace.

Or were you trying to compile argparse itself? Then I don't see a reason
either why that shouldn't work.


> And then Cython will generate an incorrect C program with
> the following error:
> 
> smt.py:78:88: undeclared name not builtin: headtail
> smt.c:1:2: error: #error Do not use this file, it is the result of a
> failed Cython compilation.
> 
> In comparison, I found that nuitka can convert this kind of Python
> programs sucessfully. I hope Cython can be improved. Thanks!

Cython makes this an error because in almost all cases, with very few
exceptions, it is an actual bug in your program. For me personally, this
already helped me find several bugs in the past, both in my code as well as
other people's code, including at least one in Python's standard library.

If you think you know better, you can disable the check by setting

  Cython.Compiler.Options.error_on_unknown_names = False

Stefan

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] unsupported meta-programming-related features

[Xuancong Wang]

Sorry, I have a few extra steps which map all parser symbols into the
global space.

parser = argparse.ArgumentParser(...)
parser.add_argument('-ht', '--headtail', help='add headtail',
default=False, action='store_true')
opt=parser.parse_args()
globals().update(vars(opt))

xuancong

On Wed, Apr 20, 2016 at 1:41 AM, Robert Bradshaw  wrote:
> On Tue, Apr 19, 2016 at 2:13 AM, Xuancong Wang  wrote:
>>
>> Dear Cython developers,
>>
>> Python supports meta-programming, in which a variable with name
>> specified in a string can be created at run-time. One built-in library
>> which make use of this is argparse.
>>
>> For example:
>>
>> parser.add_argument('-N', '--max_threads', help='maximum number of
>> concurrent decoding threads', type=int, default=16)
>>
>> in this case, the variable max_threads is created from the string
>> argument. And then Cython will generate an incorrect C program with
>> the following error:
>>
>> smt.py:78:88: undeclared name not builtin: headtail
>> smt.c:1:2: error: #error Do not use this file, it is the result of a
>> failed Cython compilation.
>
>
> Argparse works just fine in Cython
>
> import argparse
> parser = argparse.ArgumentParser()
> parser.add_argument('-N', '--max_threads', help='maximum number of
> concurrent decoding threads', type=int, default=16)
> args = parser.parse_args()
> print args.max_threads
>
> Could you provide a full example that you think should work? Where is
> headtail defined? Is it another argument?
>
>>
>> In comparison, I found that nuitka can convert this kind of Python
>> programs sucessfully. I hope Cython can be improved. Thanks!
>
>
> argsparse dynamically adds the attributes to the returned object, which
> works fine in Cython. Unless you're doing something like
>
> http://stackoverflow.com/questions/19299635/python-argparse-parse-args-into-global-namespace-or-a-reason-this-is-a-bad-idea
>
> That's one of the few places we diverge, because people strongly preferred
> compile-time errors to runtime errors in this case.
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel