[Cython] Optimizing fastcall
Related to https://github.com/cython/cython/issues/3343
One issue with the current code generated for fastcall function is that
for a function like:
def f(a, *args, **kwds):
# something...
the wrapper function immediately generates a tuple for `args` and a dict
for `kwds`, which presumably misses most of the benefits of the more
efficient calling method. It therefore seems like a good idea to have an
optional optimization so that the more efficient representation can be
used (close to) directly.
I've created an initial attempt at it
https://github.com/cython/cython/pull/3346. This defines a complier
directive so that:
@cython.fastcall_args("*" or "**" or "both")
def function...
is converted to a relatively simple structure type. The tuple equivalent is
typedef struct {
PyObject *const *args;
Py_ssize_t nargs;
} __Pyx_FastcallTuple_obj;
and the dict equivalent a little more complicated is
typedef struct {
PyObject *const *args; // start of the keyword args values
PyObject *object; // either a dict, a tuple or NULL
} __Pyx_FastcallDict_obj;
// exists in one of three states:
// * args is NULL, "object" is NULL, meaning no keyword arguments
// * args is NULL, in which case "object" is actually a dict, and this
just defers to the dict methods
// * args is non-null, kwnames is a tuple
These then support simple operations (indexing, iterating, passing
directly to a fastcall function call etc). For most non-simple
operations they will just be coerced to a Python object with a warning
(but trying to put them in a closure will explicitly fail).
-
I suspect my proposed code is still a bit of a mess, so I'm not looking
for too much scrutiny of that right now. What I'm interested in is:
* is the syntax to use this sensible? (it's probably easily changed)
* is the level of warnings appropriate?
* any operations that I've missed that'd be sensible to support?
* anything else?
David
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Auto-generation of wrapper types
The process of wrapping a C struct or C++ class in an extension type often has the user doing a pretty mechanical duplication of attributes/functions that Cython already knows about. I'm looking at doing: cdef struct S: int a # etc. then `cython.autowrap[S]` would create an extension type that wraps S by value. All attributes convertible to/from a Python type gets a property (as well as any attribute that has an has an autowrap declared). For `cppclass` this would extend to member functions as well - this obviously gets more involved, but again the same basic rule applies of only including stuff with an obvious conversion. I wouldn't propose to deal with the C++ nightmare of "how to return an owned reference". The idea would be to copy by value or nothing. I'd also propose only minor customization via keyword arguments (for example the name of a cdef staticmethod constructor, the name of the "obj" field in the C++ class). The basic rule would be that if it doesn't do what you want then it's an extension type, so you can always inherit from it and define the missing bits. Obviously structs have already have an automatic conversion to/from dicts and some cppclasses have autoconversions too. This wouldn't aim to replace that - it'd just be an option that the user could explicitly ask for. I have a somewhat working prototype for the struct side. Obviously the real complications are on the C++ side, but I don't think it's hugely difficult providing you accept there's lots of stuff that can't be guessed and inheritance is the way round that. Does this sound reasonable/something that'd be accepted? Any other thoughts? David ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Auto-generation of wrapper types
(Sorry, originally replied just to Frank not to the list) If I understand you correctly it'd be something like: cdef class WrappingOfS(autowrap[S]): def new_method(self): # implementation i.e. if you want to do something the automation can't do then you just inherit. If I don't understand you correctly then please clarify... On 13/03/2020 09:56, Schlimbach, Frank wrote: I like automatism, this one looks useful. How would you allow extending the Extension type with a new attribute or method? I'd expect that in many cases you need to do more than just wrapping the C class/struct. -Original Message- From: cython-devel On Behalf Of da-woods Sent: Thursday, March 12, 2020 4:12 PM To: cython-devel@python.org Subject: [Cython] Auto-generation of wrapper types The process of wrapping a C struct or C++ class in an extension type often has the user doing a pretty mechanical duplication of attributes/functions that Cython already knows about. I'm looking at doing: cdef struct S: int a # etc. then `cython.autowrap[S]` would create an extension type that wraps S by value. All attributes convertible to/from a Python type gets a property (as well as any attribute that has an has an autowrap declared). For `cppclass` this would extend to member functions as well - this obviously gets more involved, but again the same basic rule applies of only including stuff with an obvious conversion. I wouldn't propose to deal with the C++ nightmare of "how to return an owned reference". The idea would be to copy by value or nothing. I'd also propose only minor customization via keyword arguments (for example the name of a cdef staticmethod constructor, the name of the "obj" field in the C++ class). The basic rule would be that if it doesn't do what you want then it's an extension type, so you can always inherit from it and define the missing bits. Obviously structs have already have an automatic conversion to/from dicts and some cppclasses have autoconversions too. This wouldn't aim to replace that - it'd just be an option that the user could explicitly ask for. I have a somewhat working prototype for the struct side. Obviously the real complications are on the C++ side, but I don't think it's hugely difficult providing you accept there's lots of stuff that can't be guessed and inheritance is the way round that. Does this sound reasonable/something that'd be accepted? Any other thoughts? David ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Auto-generation of wrapper types
The staticmethod constructor is useful I think - regular constructors can't take C/C++ types so the static method constructor provides a means of doing something like: cdef mycppclass inst = # some call to cpp return wrapperclass.create(inst) I definitely agree you want some means of naming the generated class. At the minute I'm my thoughts are to do two things: 1. Having a cache of the generated classes (so autowrap[S] always returns the exact same class if you've already created one). That'd mean there's no penalty for just always referring to it as autowrap[S], so you don't /really/ need to name it. 2. Allow something like `ctypedef autowrap[S] whatever_name` (which doesn't currently look to work...) With that said, adding an option to override the name would be pretty easy, and probably useful. As always, it's a fine balance is always between making sure you have the useful options and ending up in a huge option overload so no-one can work out where to start. On 13/03/2020 11:29, Schlimbach, Frank wrote: Right. Sorry, not sure what I was thinking. Instead of a staticmethod constructor, maybe it's more useful to have a keyword arg for the name of the generated extension class? -Original Message----- From: cython-devel On Behalf Of da-woods Sent: Friday, March 13, 2020 12:02 PM To: cython-devel@python.org Subject: Re: [Cython] Auto-generation of wrapper types (Sorry, originally replied just to Frank not to the list) If I understand you correctly it'd be something like: cdef class WrappingOfS(autowrap[S]): def new_method(self): # implementation i.e. if you want to do something the automation can't do then you just inherit. If I don't understand you correctly then please clarify... On 13/03/2020 09:56, Schlimbach, Frank wrote: I like automatism, this one looks useful. How would you allow extending the Extension type with a new attribute or method? I'd expect that in many cases you need to do more than just wrapping the C class/struct. -----Original Message- From: cython-devel On Behalf Of da-woods Sent: Thursday, March 12, 2020 4:12 PM To: cython-devel@python.org Subject: [Cython] Auto-generation of wrapper types The process of wrapping a C struct or C++ class in an extension type often has the user doing a pretty mechanical duplication of attributes/functions that Cython already knows about. I'm looking at doing: cdef struct S: int a # etc. then `cython.autowrap[S]` would create an extension type that wraps S by value. All attributes convertible to/from a Python type gets a property (as well as any attribute that has an has an autowrap declared). For `cppclass` this would extend to member functions as well - this obviously gets more involved, but again the same basic rule applies of only including stuff with an obvious conversion. I wouldn't propose to deal with the C++ nightmare of "how to return an owned reference". The idea would be to copy by value or nothing. I'd also propose only minor customization via keyword arguments (for example the name of a cdef staticmethod constructor, the name of the "obj" field in the C++ class). The basic rule would be that if it doesn't do what you want then it's an extension type, so you can always inherit from it and define the missing bits. Obviously structs have already have an automatic conversion to/from dicts and some cppclasses have autoconversions too. This wouldn't aim to replace that - it'd just be an option that the user could explicitly ask for. I have a somewhat working prototype for the struct side. Obviously the real complications are on the C++ side, but I don't think it's hugely difficult providing you accept there's lots of stuff that can't be guessed and inheritance is the way round that. Does this sound reasonable/something that'd be accepted? Any other thoughts? David ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisensch
Re: [Cython] Auto-generation of wrapper types
On 15/03/2020 08:14, Stefan Behnel wrote: What makes C structs and C++ classes special enough to make this a separate language feature? That should be part of a motivation somewhere (probably in a ticket Most of the other major things you could want to wrap are covered through other mechanisms - you can create a wrapper round a `cdef extern` function or enum with `cpdef`. The basic C types (int, float) etc largely have a direct Python equivalent that converts freely. This is mainly targetted at C++ classes which seem to be the last major feature that doesn't have this kind of wrapper. C structs just seemed easy to deal with by the same mechanism - although they obviously have the conversion to/from dict, which is fine but does involve a conversion, and so doesn't propagate modifications back. I'll create a ticket fairly shortly. How does this relate to the support for @dataclass in GH-2903? Is the only difference that you would write "x.member" instead of "x.structattr.member"? It's slightly difficult to answer that because we don't know exactly what @dataclass will look like in Cython. In Python it's mostly about generating `__init__`, `__repr__`, `__eq__` etc from a "struct-like" description. The Cython version I've submitted basically just follows the Python scheme, but it isn't obvious how Python-level access to attributes should work. "x.structattr" would probably end up doing the usual Cython conversion to/from a dict. This feature is mostly for creating a quick wrapper for some external C/C++ thing that already exists. For a struct it probably isn't too different from a dataclass. For a C++ class it should hopefully be able to fill in a lot of the functions too. Why "autowrap[S]" and not "autowrap(S)"? I'm asking also because there should probably be a way for this to work in Python files, for which a function call seems more natural. It also feels like a @decorator would somehow fit quite well here. Even when defined in a .pxd file, the struct would still only be exported in the corresponding module (with the same name). Something like this: @cclass cdef struct S: int x Not sure if reusing @cclass is a good idea, but it doesn't seem wrong. Definitely open to different syntax - now you point it out I think a function call might be better than an index (especially with optional keyword arguments to control it). One other option (for consistency with how functions/enums are already handled) would be `cpdef struct S/cpdef cppclass ClassName`. The difficulty here is that it overrides the name which I suspect could be a challenge (in most cases you'd probably want to access the original definition and the wrapper extension type separately in Cython). This might be an issue with a decorator too. Defining this in a .pxd file could then even allow cimporting modules to understand the extension type wrapper as well, by simply reconstructing the type representation on their side. I hadn't thought of this, but yes. This should be pretty simple - the type representation is really just: cdef class SomeName: cdef c_type* obj @staticmethod cdef factor_func(obj x) # possibly reference or r-value here reference (Almost) everything else it defines is a Python interface of def functions and properties so a cimporting module would have no special knowledge - for fast access the user would go through obj. David Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Preparing the first alpha release of Cython 3.0
(I've not got the hang of the difference between "reply" and "reply-to-list" so initially sent this to the wrong place...) I've pinged a couple that would be nice to have because they both target Python compatibility: https://github.com/cython/cython/pull/3323/ https://github.com/cython/cython/pull/3383 They're both fairly large changes though and haven't been through any real review yet so it's possible they're exactly the opposite of what you're looking for. Feel free to ignore if that's the case David On 25/03/2020 08:10, Stefan Behnel wrote: Hi all, I'm curently preparing a first alpha release for Cython 3.0 (alpha 1). Please ping me on any open pull requests that you think are ready and should be included. Note that this is not the end of the journey – anything that's not included right now can still come in later, before the first beta. (That's later this year, before you ask.) There will definitely be at least one more alpha release before we get even there. So, please concentrate on changes for this release that have a potentially disruptive user impact, so that our users can start preparing their code and/or complain early in order to keep us from making The One Big Mistake of breaking their code. Thanks, Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Auto-generation of wrapper types
1) I hadn't give too much thought to inheritance. I think trying to mirror it in the wrapped types might be more trouble than it's worth - duck typing will probably do for most cases. 2) I'm definitely aware of the ownership and common references issue. I was going to assume that arguments passed in by (maybe const?)-reference could be treated as "by-value", but I guess that'll have to be a heavily flagged caveat in the documentation. Agree that integration with other wrappers could be useful. A very basic version is probably surprisingly easy in a lot of cases - it'd just be a case of exposing where they store the underlying C++ object. On 21/03/2020 18:47, Robert Bradshaw wrote: I am in general of more automated wrapping. Two things that make C++ classes more difficulty are. (1) How to handle inheritance (including multiple inheritance)? Would the wrapped types mirror the inheritance? (2) We can do auto-conversion of most types because they are passed by value. This breaks down a bit with char*, but generally we assume we can take a copy. Similarly with structs. However C++ classes are (in practice) much more stateful and this gets into all sorts of thorny issues with what the conventions should be about ownership and common references. There's also a fair number of existing libraries out there that specifically target wrapping C++ libraries (often directly from the source files). As well as making it easier to wrap from C++ classes from Cython, it'd be good to have more seamless (and efficient) integration with those. On Sun, Mar 15, 2020 at 1:34 PM da-woods wrote: On 15/03/2020 08:14, Stefan Behnel wrote: What makes C structs and C++ classes special enough to make this a separate language feature? That should be part of a motivation somewhere (probably in a ticket Most of the other major things you could want to wrap are covered through other mechanisms - you can create a wrapper round a `cdef extern` function or enum with `cpdef`. The basic C types (int, float) etc largely have a direct Python equivalent that converts freely. This is mainly targetted at C++ classes which seem to be the last major feature that doesn't have this kind of wrapper. C structs just seemed easy to deal with by the same mechanism - although they obviously have the conversion to/from dict, which is fine but does involve a conversion, and so doesn't propagate modifications back. I'll create a ticket fairly shortly. How does this relate to the support for @dataclass in GH-2903? Is the only difference that you would write "x.member" instead of "x.structattr.member"? It's slightly difficult to answer that because we don't know exactly what @dataclass will look like in Cython. In Python it's mostly about generating `__init__`, `__repr__`, `__eq__` etc from a "struct-like" description. The Cython version I've submitted basically just follows the Python scheme, but it isn't obvious how Python-level access to attributes should work. "x.structattr" would probably end up doing the usual Cython conversion to/from a dict. This feature is mostly for creating a quick wrapper for some external C/C++ thing that already exists. For a struct it probably isn't too different from a dataclass. For a C++ class it should hopefully be able to fill in a lot of the functions too. Why "autowrap[S]" and not "autowrap(S)"? I'm asking also because there should probably be a way for this to work in Python files, for which a function call seems more natural. It also feels like a @decorator would somehow fit quite well here. Even when defined in a .pxd file, the struct would still only be exported in the corresponding module (with the same name). Something like this: @cclass cdef struct S: int x Not sure if reusing @cclass is a good idea, but it doesn't seem wrong. Definitely open to different syntax - now you point it out I think a function call might be better than an index (especially with optional keyword arguments to control it). One other option (for consistency with how functions/enums are already handled) would be `cpdef struct S/cpdef cppclass ClassName`. The difficulty here is that it overrides the name which I suspect could be a challenge (in most cases you'd probably want to access the original definition and the wrapper extension type separately in Cython). This might be an issue with a decorator too. Defining this in a .pxd file could then even allow cimporting modules to understand the extension type wrapper as well, by simply reconstructing the type representation on their side. I hadn't thought of this, but yes. This should be pretty simple - the type representation is really just: cdef class SomeName: cdef c_type* obj @staticmethod cdef factor_func(obj x) # possibly reference or r-value here referen
Re: [Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2
Hi,
(Sent to both cython-devel and cython-users. However, I'm not subscribed
to users I think, so please forward if it doesn't reach there).
On closer inspection I believe both are following Python behaviour (and
thus behaving correctly). In Python any
`__name_preceded_by_double_underscores` used within a class ends up
looking up `_Classname__name_preceded_by_double_underscores`. Therefore
in case 1 it's looking up `_Bar__foo` and not finding it.
Therefore, I think it is the "correct" thing to do.
This proved a pain with the `__pyx_unpickle` functions that Cython
generates. In the end I just disabled it for `__pyx_` names, which I
guess could be called hypocrisy .
David
On 23/04/2020 18:47, Stefan Behnel wrote:
Hi,
thanks for the very detailed report.
I'm CC-ing cython-devel, but let's keep the discussion for this on the
cython-users list.
The background here is Github issue 1382 and PR 3123
https://github.com/cython/cython/issues/1382
https://github.com/cython/cython/pull/3123
This was newly merged for alpha 2, and I consider it somewhat provisional.
It makes "private" Python names follow Python semantics, but if that poses
excessive breakage for existing code, then we'll have to find a pragmatic
way to deal with it.
'Toni Barth' via cython-users schrieb am 23.04.20 um 19:07:
I just updated Cython from 3.0a1 to 3.0a2 in one of my projects and
noticed that it seems to introduce issues with private
functions/methods, means methods with leading __.
I noticed two cases which now fail in my project. I'm however unsure if
thats supposed to happen or if its a problem with the new Cython build.
Case 1: private function not declared
crash.pxd:
cpdef __foo()
cdef class Bar:
cpdef run(Bar self)
crash.pyx:
cpdef __foo():
print("foo")
cdef class Bar:
cpdef run(Bar self):
__foo()
print("bar")
Running this with the following command:
py -3 -c "import pyximport;pyximport.install();import
crash;crash.Bar().run()"
Result:
crash.c
C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
FutureWarning: Cython directive 'language_level' not set, using '3str'
for now (Py3). This has changed from earlier releases! File:
D:\crash\crash.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
Error compiling Cython file:
...
cpdef __foo():
print("foo")
cdef class Bar:
cpdef run(Bar self):
__foo()
^
crash.pyx:6:4: undeclared name not builtin: __foo
At least this looks like a bug to me. Global module names should not be
impacted by this change.
Error compiling Cython file:
...
cpdef __foo():
print("foo")
cdef class Bar:
cpdef run(Bar self):
__foo()
^
crash.pyx:6:4: '__foo' redeclared
Error compiling Cython file:
...
cpdef __foo()
^
crash.pxd:1:12: Previous declaration is here
Lovely :)
Another problem, probably related, occurs when trying to override a
private method in a derived class:
Case 2: overriding private class methods
base.pxd:
cdef class Base:
cdef void __foo(Base self)
cpdef run(Base self)
base.pyx:
cdef class Base:
cdef void __foo(Base self):
print("foo!")
cpdef run(Base self):
self.__foo()
extend.pxd:
from base cimport Base
cdef class Extend(Base):
pass
extend.pyx:
cdef class Extend(Base):
cdef void __foo(Extend self):
print("bar")
Running with the following command:
py -3 -c "import pyximport;pyximport.install();import
extend;extend.Extend().run()"
Results in the following output:
extend.c
C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python36_64\lib\site-packages\Cython\Compiler\Main.py:344:
FutureWarning: Cython directive 'language_level' not set, using '3str'
for now (Py3). This has changed from earlier releases! File:
D:\crash\extend.pxd
tree = Parsing.p_module(s, pxd, full_module_name)
Error compiling Cython file:
...
cdef class Extend(Base):
cdef void __foo(Extend self):
^
extend.pyx:3:7: C method '_Extend__foo' not previously declared in
definition part of extension type 'Extend'
This, OTOH, is intended. Private methods are not meant to be inherited and
overridden. That's what makes them "private" :) That's consistent with
Python semantics.
I think it's somewhat up for debate if this should apply in the same way to
cdef classes as for Python classes. But from a language design point of
view, having this kind of difference between the two seems a bad idea.
Do you fee
Re: [Cython] [cython-users] Problems with "private" functions/methods in Cython 3.0a2
Hopefully I've succeeded in subscribing correctly so I can now post to both groups... On 23/04/2020 21:32, Stefan Behnel wrote: The question is: how bad is it for existing code, and how can we either keep things working or make it easy for users to adapt? Should we implement some kind of fallback lookup that still finds unmangled global names if they exist? I mean, double-underscore names are not entirely unheard of in C code and C libraries. The Python behaviour redners their usage from within classes more difficult. I think this could actually be a real problem for people trying to interact with C libraries. I'd favour a fallback that finds unmangled cdef/c names only (and maybe gives a helpful warning but no match if a Python name is found). That gives 100% compatibility on the Python side, but a sensible fallback on the C side. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Infer types and array.pxd
I'm not convinced that compiler directives like "infer_types" should propagate into the cimported files at all. It seems reasonable to me that these should be controlled by the author of the pxd file, not by the eventual cimporter. On 02/08/2020 14:18, Stefan Behnel wrote: Oliver Haas schrieb am 02.08.20 um 13:44: The array.pxd ( https://github.com/cython/cython/blob/master/Cython/Includes/cpython/array.pxd) requires infer_types = True when cimported. Is there a reason not to explicitly type the variable op as array in the lines 134 and 141? The return type of the function that creates the object is known, so the normal (safe) type inference should work here without setting "infer_types=True". Do you want to set "infer_types=False", or why is this failing for you? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Infer types and array.pxd
https://github.com/cython/cython/issues/1071 On 02/08/2020 18:23, Stefan Behnel wrote: Am 2. August 2020 17:02:39 MESZ schrieb da-woods: I'm not convinced that compiler directives like "infer_types" should propagate into the cimported files at all. It seems reasonable to me that these should be controlled by the author of the pxd file, not by the eventual cimporter. I agree. It might not be as strikt as that, there could still be directives that should be inherited, but definitely not all of them. Can't look right now, but there might already be a ticket about this. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
A few things to point out: 1) Because Cython is mainly a is code translation tool it doesn't currently have an overview of how things would eventually be evaluated - it leaves a lot for the C compiler to sort out. Your suggestion would presumably add a dependency on a C compiler at the .pyx -> .c stage (currently the only dependencies are Python+Cython itself). 2) Presumably `twice` and `make_ptr_class` would also have to work at run-time? They are regular `def` functions after all? That seems like a big change to now have to dynamically create `cdef` functions and classes at both Cythonization-time and runtime. And also a Python-runtime representation of every C type (so that you can call `make_ptr_class` at runtime) I totally get the appeal of being able to wrap template-types quickly and easily. However, to me this idea actually seems a lot harder than something which creates new syntax, because you don't distinguish these special functions and so they have to work universally as Python functions too. In summary: if it worked it'd be very nice, but I personally don't have any idea how you'd implement it within how Cython current works. David On 24/02/2021 14:29, Celelibi wrote: Le Tue, Feb 23, 2021 at 11:24:42AM -0500, Prakhar Goel a écrit : I had similar ideas in the past (assuming I understand your proposal). I'd love it if a decorator could link to some python function. This python function would get something that represents the cdef function (with the ast). It would then return a transformed ast that could be spliced into the original program. As I said, I'm not sure allowing AST manipulation would be that useful. I mean, it's rarely ever needed in python. Why would it be needed with cython? However, it would surely make the code much less readable, since the code that's executed is not the code that's written. The use-cases I have in mind are more like these. Which are common patterns in python. Compile-time implementation choice: if os == "Windows": cdef thread_something(): # Handle Windows-specific behavior else: cdef thread_something(): # Assume POSIX behavior Decorators on cdef functions: def twice(f): cdef fun(x): return f(2*x) return fun @twice cdef foo(x): # Do something math-y Creating cdef classes on demand: def make_ptr_class(T): cdef class C: cdef T *ptr return C IntPtr = make_ptr_class(int) FloatPtr = make_ptr_class(float) My current use-case is actually exactly this one. Creating wrappers classes for pointer types. Which I currently cannot make both generic and type safe. The python code run at compile-time would basically manipulate python objects representing cdef functions, classes and C types. Internally, the way it could be implemented is that cython could generate a python program that would produce the AST. But it doesn't mean the AST itself has to be visible. Best regards, Celelibi ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [ENH] Compile-time code execution (macros / templates)
1. I'm not sure why there would be any dependency on a C compiler in either mine or Celelibi's proposal. Could you please elaborate? At least in my proposal, we only ever go from Cython AST to Cython AST and use Python to do it. None of these things require a C compiler. To me it looked like Celelibi's proposal was basically asking for "some subset" of regular Cython code to be evaluated at Cythonization time. Since evaluation of Cython code requires compiling the C code first, it seems to me it'd need a C compiler. It depends on the subset of course. I agree that your suggestion of modifying the AST can/should be done in Python. 2. Again I can't speak for Celelibi but my proposal would be compile-time only. I'm happy to add restrictions (decorators on cdef functions only or maybe even special syntax) to ensure that this is the case. The D-language which Celelibi linked as an example claims "All functions that execute in CTFE must also be executable at run time. The compile time evaluation of a function does the equivalent of running the function at run time" and Celelibi didn't seem to be proposing any special syntax. So on that basis I don't see where any compile-time only restrictions can come from. It's quite possible that I've missed the point of the proposal in both cases though. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] How best to contribute
I'm tactically ignoring the bits of this I don't know the answer to: > there are issues marked as "blockers", are all of them truly critical I think the main criteria for being a "blocker" was to try to keep significant breaks in compatibility to the Cython 3 release (so that users might been to fix their code a little after Cython 3, but probably not after that). Thus https://github.com/cython/cython/issues/1159 is a "blocker" because people might be relying on the old behaviour. Personally I think most of the remaining blockers are probably small enough that maybe they shouldn't really block anything (but opinions may vary on this). Of those blockers https://github.com/cython/cython/issues/2912 looks like something that could be fixed pretty quickly if you were inclined just by making a new PR based on the old one. > And, in the long term, is there a way to start designing a new backend for HPy I think the first question is: do you add HPy code to the existing .c file with extra include-guards, or do you want to make it a compiler option to either generate a PyObject* .c file or a HPy .c file? My opinion would be to make it a compiler option - I think doing it with include-guards would end up mixing in too much dissimilar code. If you do want to generate a separate backend then we don't currently have anything similar. Maybe have a separate shadow sets of nodes and adding a transform just before the code generation step (so `ExprNodes.AttributeNode` is transformed into `ExprNodes.HPy.AttributeNode` which then handles the code generation, but doesn't need to include any of the analyse_types steps). My impression is that you can mix PyObject with HPy so you don't necessarily have to do everything in one go. Having some "switchable backend" mechanism might turn out to be useful generally: there was also a request for "micropython" support a while back, which is realistically too different to be forced into the current PyObject code, but might be possible to support with switchable backends (if someone were inclined to do it). David On 26/02/2021 08:03, Matti Picus wrote: Hi. I would like to start contributing in a meaningful way to Cython on the order of ~1 day a week, within the framework of the time allocated to me from my employer (Quansight Labs) toward open source contributions. Over time, my goal is push for an HPy[0] backend for Cython, but I also want to help the project move forward toward an official 3.0 release and in general to help out where needed. So working backwards: What are the immediate pain points that I could help out with in general? How can I help with the 3.0 release (there are issues marked as "blockers", are all of them truly critical)? And, in the long term, is there a way to start designing a new backend for HPy? I would also like to suggest that Cython hold monthly open "developer calls" where people can meet face-to-face (on zoom) to work out priorities and triage particularly difficult issues or PRs. I would be happy to try to set this up (mainly to negotiate a time that would work) if you-all think it is a good idea and would contribute more than it would distract. Thanks, Matti [0] HPy [https://hpy.readthedocs.io/en/latest] ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Overhead in MemoryviewSlice Conversion?
They're generated using a string-substitution tool (and thus hard to
search for). But the code is
https://github.com/cython/cython/blob/8609e0fa7d361f1392823ff6e1a618720cd62df3/Cython/Utility/MemoryView_C.c#L130.
I found it by searching for "ObjectToMemviewSlice ", which appears just
above the prototype in the generated C code.
My impression is that creating memoryview slices isn't always hugely
quick - I think the old np.ndarray syntax can be a little quicker to
create (although it's worse in other ways)
On 21/03/2021 18:23, Golden Rockefeller wrote:
I know there has been some work last year at improving the overhead in
creating cython.view.arrays (cvarray). But I have noticed a large
overhead in converting that cvarray to a memoryview (e.g double[:]).
I would like some advice in inspecting cython's source code.
For example, where can I find
"__Pyx_PyObject_to_MemoryviewSlice_ds_double"?
Through testing, my guess is that this function introduces most of the
overhead:
"""
@timefunc("cvarray_to_memoryview")
def _(Py_ssize_t L):
cdef Py_ssize_t i
cdef double* arrptr
cdef cvarray cvarr
cdef double[:] arr
for i in range(loops):
arrptr = malloc(sizeof(double) * L)
cvarr = cvarray((L,),sizeof(double),'d', mode="c", allocate_buffer=False)
cvarr.callback_free_data = free
cvarr.data = arrptr
arr = cvarr # Only difference is this conversion
# Prevents dead code elimination
str(arr[0])
@timefunc("cvarray")
def _(Py_ssize_t L):
cdef Py_ssize_t i
cdef cvarray cvarr = cvarray((L,),sizeof(double),'d')
for i in range(loops):
arrptr = malloc(sizeof(double) * L)
cvarr = cvarray((L,),sizeof(double),'d', mode="c", allocate_buffer=False)
cvarr.callback_free_data = free
cvarr.data = arrptr
# Prevents dead code elimination
str(cvarr[0])
"""
Timings (loops = 10; L in [1, 10, 100, 1000, 1])
Running cvarray_to_memoryview
1.555391 1.510217 1.518601 1.503522 1.733853 μs
Running cvarray
0.342308 0.353832 0.335155 0.361952 0.518414 μs
Golden Rockefeller
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Anything to include in the next 3.0 alpha release?
https://github.com/cython/cython/pull/3804 (__del__ in cdef classes) looks pretty complete to me. https://github.com/cython/cython/pull/3743 (duplicate utility code) is hopefully also pretty complete and does fix some bugs that have been recurred a few times. David On 24/03/2021 19:50, Stefan Behnel wrote: Hi, I'll prepare a new 3.0 alpha release during the next days. If you have or know anything that is ready for inclusion that I should take a look at, please send me a quick note. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Remove Py3.4 support in Cython 3.0?
None from me - nobody on Ubuntu 14.04 (or similar) is going to get Cython 3 from their distribution. And if they're installing Cython manually then they can probably install a newer Python manually. Looking at the stats on PyPI (https://pypistats.org/packages/cython) the numbers of downloads for 3.4 are pretty small (a few hundred). And that's for the stable 0.29.x branch mainly I imagine. I doubt if they're the people that are looking to use the latest thing anyway. David On 14/08/2021 09:21, Stefan Behnel wrote: Hi, I looked up where Python 3.4 is still being used. Ubuntu 14.04 – EOS 2019, EOL April 2022 Debian 8 – EOL Centos 6 – EOL Later distribution versions have newer CPythons: Ubuntu 16.04: Py3.5 (EOS April 2021, EOL 2024) Debian 9: Py3.5 (EOL June 2022) Centos 7: Py3.6 (EOL June 2024) I think we can safely drop Py3.4 support already for Cython 3.0. The advantage isn't big, since we still support Py2.7, but we can at least stop testing it, and start relying on the Py3.5 C-API for Py3. Given that Py3.5 is also going out of support in major Linux distributions within a year's time, I'd suggest we drop support for it in Cython 3.1, together with Py2 support, making the minimum version Py3.6. That would mean that we can start using f-strings, dict ordering, and what not. Any objections? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Annotations and Cython 3
One further option would be to revert to Cython 0.29 behaviour, but support the PEP-563 version as a "from __future__" import (essentially following the current behaviour of Python). When I wrong the original patch for the PEP-563 I tried to do that, but then removed the code in favour of keeping one simpler option. But the code to do this mostly already exists (somewhere) On 25/10/2021 19:11, da-woods wrote: > [...] ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Annotations and Cython 3
I suspect the answer to this is still "wait and see" but: do we have an opinion on what we want to do about annotations in Cython 3. Currently we're targetting PEP-563 behaviour. To me it looks fairly likely that this won't be what Python decides on. I guess the likely outcomes are "status quo" or something close to PEP-649. The issue with the PEP-563 (stringify everything) behaviour is that it's noticeably different, so releasing Cython 3 with that behaviour then changing to something else will be an obvious incompatible change. I'd be reluctant to implement PEP-649 until it's actually accepted (it looks hard...). Is it worth reverting to the Cython 0.29 behaviour (with some scoping fixes) on the basis that it's what most people are using now, and it's likely to be closer what what's picked? Or should we at least mark "decide what to do with annotations" as a blocker for Cython 3? David ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Annotations and Cython 3
On 26/10/2021 07:53, Stefan Behnel wrote: Am 26. Oktober 2021 07:54:10 MESZ schrieb Lisandro Dalcin: In any case, if you know how and can implement the __future__ import, that would be great. Otherwise, valid Python3 code using that import currently fails to compile in Cython. Probably just an oversight. A no-op future import is trivial to add and ignore. We do support the __future__ import as a no-op on the Cython 3 alphas: https://github.com/cython/cython/blob/e93c2c556c5c15148763b26c6e3eef35ee88a22f/tests/run/pep563_annotations.py#L4 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Open PRs
Hi Matus, I've had a look at your documentation PRs and merged 2/3 of them, so hopefully that's some progress. The other's are a little larger and a little outside my expertise so it'd need a much more detailed look before I was confident doing anything. I only got the power to merge things fairly recently so I'm still trying to judge what kind of things I can take unilateral decisions on. I am trying to get some of the simpler outstanding PRs dealt with where I can though. David On 24/06/2022 19:03, matus valo wrote: Hi All, I opened several PRs several weeks/months ago. I understand that a maintainer is an unpaid role so there are no expected deadlines, but still I would kindly ask for code review of the PRs. All PRs except one underwent several rounds of review, so they should be close to accepting (at least I hope that :-) ): * https://github.com/cython/cython/pull/4809 * https://github.com/cython/cython/pull/4629 * https://github.com/cython/cython/pull/4625 (there is another PR waiting for this PR to be merged: https://github.com/cython/cython/pull/4630) * https://github.com/cython/cython/pull/4552 Docs: * https://github.com/cython/cython/pull/4544 * https://github.com/cython/cython/pull/4537 * https://github.com/cython/cython/pull/4516 Thanks, Matus ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Welcome David Woods as a Cython core developer
Hi, I'm not sure if it's necessary to for me to reply to this, but: thanks. Hopefully some of the features will be (or already have been) useful to people. David On 31/07/2022 10:15, Stefan Behnel wrote: Hi everyone, with the release of the first 3.0 alpha that supports Python 3.11 (aptly named "alpha 11"), I'm happy to announce that David Woods has been promoted to a Cython core developer. David has shown an extraordinary commitment and dedication over the last years. His first merged commits were already back in 2015, mostly related to the C++ support. But within the last two years, he voluntarily took over more and more responsibility for bugs and issues and developed several major new features for the project. This includes the Walrus operator (PEP 572), cdef dataclasses (modelled after PEP 557), internal "std::move()" usage in C++ mode or support for Unicode identifiers and module names, all of which form a major part of the 3.0 feature set. David has more than deserved a place in the circle of present and prior core devs. David, thank you for your impressive work on Cython, and welcome to the core team! Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] fused type error changed in Cython 3
Hi Matus, I made a change to optimize dispatch of fused memoryview types. I suspect that's inadvertently caused the change. I have a reasonable idea what needs to be fixed, but would need to investigate properly. I don't think it's an intended change. David On 23/04/2023 12:14, matus valo wrote: Hi All, I am working on porting scipy to Cython3. The last missing part in order to have a test suite is 2 tests failing [1]. When I investigated the issue, I found out that Cython has changed the type of Exception. Consider following example: cimport numpy as cnp import cython ctypedef fused np_numeric_t: cnp.int8_t cnp.int16_t cnp.int32_t cnp.int64_t cnp.uint8_t cnp.uint16_t cnp.uint32_t cnp.uint64_t cnp.float32_t cnp.float64_t cnp.longdouble_t cnp.complex64_t cnp.complex128_t @cython.initializedcheck(False) def bandwidth_c(np_numeric_t[:, ::1]A): return In cython 0.29.X I got following: >>> import numpy as np >>> zz = np.zeros([5, 5], dtype='M') >>> import fused >>> fused.bandwidth_c(zz) Traceback (most recent call last): File "", line 1, in File "fused.pyx", line 32, in fused.__pyx_fused_cpdef def bandwidth_c(np_numeric_t[:, ::1]A): TypeError: No matching signature found And in Cython3 (master branch) I got following: >>> import numpy as np >>> zz = np.zeros([5, 5], dtype='M') >>> import fused >>> fused.bandwidth_c(zz) Traceback (most recent call last): File "", line 1, in File "fused.pyx", line 31, in fused.__pyx_fused_cpdef @cython.initializedcheck(False) ValueError: cannot include dtype 'M' in a buffer The question is does someone know why the exception changed in Cython3? I was not able to find out. Is this a feature or is it a bug? Interestingly, I was not able to replicate this behaviour in simple fused types containing basic types (int, float etc...). Thanks, Matus [1] https://github.com/scipy/scipy/pull/18242#issuecomment-1517459666 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cython 3 migration update and next releases
On 20/05/2023 10:59, Stefan Behnel wrote: It's probably a good time to have a final call for merges. Promoting and voting for PRs is welcome. I've flagged a couple of PRs for possible merging. They're both fairly minor Python-compatibility fixes so it probably isn't a disaster if they're missed though. Unfortunately, I doubt I can get the module state refactor into a usable state in time. David ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Appetite for working with upstream to extend the buffer protocol?
So my superficial thoughts: 1. The buffer protocol has two bits. The first says "given this predictable memory layout, you can look up an item in memory with these rules"; the second describes what the items in memory are. I think you're only proposing to change the second part of it. I'd encourage you not to change the first part - the nice thing about the first part is that it's relatively simple and doesn't try to do anything. For example I'd be sceptical about trying to support ragged arrays. 2. As you identify, for a more advanced memoryview to be useful in Cython, Cython really has to be able to know an underlying C type for your data at compile-time and be able to validate that the buffer it's passed matches that C type at runtime. The validation could have varying degrees of strictness (i.e. in the worst case we could just check the size matches and trust the user). We already support that to extent (packed structs with structured arrays) but that doesn't cover everything 3. For your variable length string example, the C struct to use is fairly obvious (just your `struct ss`). The difficult bit is likely to be memory management of that. I'd kind of encourage you not to expect Cython to handle the memory management for this type of thing (i.e. it can expose the struct to the user, but it becomes the user's own problem to work out if they need to allocate memory when they modify the struct). 5. Things like the datetime for Pandas, or a way of having a float16 type seems like the sort of thing we should definitely be able to do. 6. In terms of Apache Arrow - if there was demand we probably could add support for it. Their documentation says: "The Arrow C data interface defines a very small, stable set of C definitions that can be easily /copied/ in any project’s source code" - so that suggests it need not be a dependency. 7. One of the points of the "typed memoryview" vs the older "np.ndarray" interface is that it was supposed to be more generally compatible. While we could extend it to match any non-standard additions that Numpy tries to make, that does feel dodgy and likely to conflict when other projects do their own thing. I think it would be better if the Python standard could be extended (even if it was just something like a code to indicate "mystery structure of size X") Don't know if these thoughts are useful. They're a bit scattered. I guess the summary is "we could definitely do more with custom data types, but don't break the things that made the buffer protocol nice". David On 06/07/2023 17:43, Nathan wrote: Hi all, I'm working on a new data type for numpy to represent arrays of variable-width strings [1]. One limitation of using this data type right now is it's not possible to write idiomatic cython code operating on the array, instead one would need to use e.g. the NumPy iterator API. It turns out this is a papercut that's been around for a while and is most noticeable downstream because datetime arrays cannot be passed to Cython. Here's an example of a downstream library working around lack of support in Cython for datetimes using an iterator: [2]. Pandas works around this by passing int64 views of the arrays to Cython. I think this issue will become more problematic in the future when NumPy officially ships the NEP 42 custom dtype API, which will make it much easier to develop custom data types. It is also already an issue for the legacy custom data types numpy already supports [3], but those aren't very popular so it hasn't come up much. I'm curious if there's any appetite among the Cython developers to ultimately make it easier to write cython code that works with numpy arrays that have user-defined data types. Currently it's only possible to write code using the numpy or typed memoryview interfaces for arrays with datatypes that support the buffer protocol. See e.g. https://github.com/numpy/numpy/issues/4983. One approach to fix this would be to either officially or unofficially extend the buffer protocol to allow arbitrary typecodes to be sent in the format string. Officially Python only supports format codes used in the struct module, but in practice you can put any string in the format code and memoryview will accept it. Of course for it actually to be useful NumPy would need to create format codes that allow cython to correctly read and reconstruct the type. Sebastian Berg proposed this on the CPython discussion forum [3] and there hasn't been much response from upstream. I response to Sebastian, Michael Droettboom suggested [4] using the Arrow data format, which has rich support for various array memory layouts and has support for exchanging custom extension types [5]. The main problem with the buffer protocol approach is defining the protocol in such a way that Cython can appropriately reconstruct the memory layout for the data type (although only supporting strided arrays at first makes
Re: [Cython] uvloop issue in Cython3
Hi Matus,
This'll be related to
https://github.com/cython/cython/commit/abeb082098c13e243a2e2658f9eb45f1c151b091
Just an example (in Python, not Cython, but the Cython behaviour should
now be the same):
class C:
... def f(self, __kwd):
... print(locals())
...
>>> C().f(1)
{'self': <__main__.C object at 0x7f299ef878d0>, '_C__kwd': 1}
C().f(__kwd=1)
Traceback (most recent call last):
File "", line 1, in
TypeError: C.f() got an unexpected keyword argument '__kwd'
C().f(_C__kwd=1)
{'self': <__main__.C object at 0x7f299ef878d0>, '_C__kwd': 1}
I haven't looked in much detail at exactly what's happening in uvloop,
but I think this is likely the correct (i.e. Python) behaviour that
you're seeing.
Hopefully that's enough to track it down.
David
On 14/08/2023 18:25, matus valo wrote:
Hi All,
I was recently migrating uvloop to cython3 [1]. Unfortunately, after
switching to cython3, uvloop testsuite fails [2]. I am puzzled mainly
about following error:
==
ERROR: test_process_delayed_stdio__paused__no_stdin
(test_process.Test_UV_Process_Delayed)
--
Traceback (most recent call last):
File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line
938, in test_process_delayed_stdio__paused__no_stdin
__uvloop_sleep_after_fork=True))
File "uvloop/loop.pyx", line 1517, in
uvloop.loop.Loop.run_until_complete
return future.result()
File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line
911, in run_sub
**kwargs)
File "uvloop/loop.pyx", line 2808, in subprocess_shell
return await self.__subprocess_run(protocol_factory, args, shell=True,
File "uvloop/loop.pyx", line 2734, in
uvloop.loop.Loop._Loop__subprocess_run
@cython.iterable_coroutine
TypeError: _Loop__subprocess_run() got an unexpected keyword argument
'__uvloop_sleep_after_fork'
==
ERROR: test_process_delayed_stdio__paused__stdin_pipe
(test_process.Test_UV_Process_Delayed)
--
Traceback (most recent call last):
File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line
919, in test_process_delayed_stdio__paused__stdin_pipe
__uvloop_sleep_after_fork=True))
File "uvloop/loop.pyx", line 1517, in
uvloop.loop.Loop.run_until_complete
return future.result()
File "/home/runner/work/uvloop/uvloop/tests/test_process.py", line
911, in run_sub
**kwargs)
File "uvloop/loop.pyx", line 2808, in subprocess_shell
return await self.__subprocess_run(protocol_factory, args, shell=True,
File "uvloop/loop.pyx", line 2734, in
uvloop.loop.Loop._Loop__subprocess_run
@cython.iterable_coroutine
TypeError: _Loop__subprocess_run() got an unexpected keyword argument
'__uvloop_sleep_after_fork'
The issue is regarding calling this method:
https://github.com/MagicStack/uvloop/blob/1dd40f17f3b0d37e3779b6ad5041bab335142337/uvloop/loop.pyx#L2735-L2753
When the same code base is compiled in Cython 0.29.X and the test
suite is run, there is no such error. I tried to create a simple
reproducer but failed.
What do you think? Is it regression in Cython 3.0? Do you have any
hint where the root cause can be? If everything fails, I will try to
bisect the issue.
Matus
[1] https://github.com/MagicStack/uvloop/pull/534
[2] https://github.com/MagicStack/uvloop/pull/534/checks
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] uvloop issue in Cython3
On 14/08/2023 20:06, matus valo wrote: BTW, I think that the Cython documentation is lacking this change: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-semantics There's a segment further down on "Class-private name mangling". I think if we wanted to mention it then it'd go there. It's probably a detail we'd want to keep fairly brief I think. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Can we remove the FastGIL implementation?
I think the detail that was missing is you need to add the `#cython: fast_gil = True` to enable it. For me: Python 3.9 and 3.10 are basically identical (on master) **test_gil_already_held** with fast_gil Running the test... took 0.175062894821167 without Running the test... took 0.10976791381835938 **test_gil_released** with fast_gil Running the test... took 0.583066463470459 without Running the test... took 0.5824759006500244 test_gil_already_held is noticably faster with fast_gil. For Python 3.11: I get the crash in 0.29.x if I try to run using fast_gil. No defines are needed to get that... On master: **test_gil_already_held** with fast_gil Running the test... took 0.17254948616027832 without Running the test... took 0.10958600044250488 **test_gil_released** with fast_gil Running the test... took 0.5791811943054199 without Running the test... took 0.5597968101501465 Note that "without fastgil" is now as fast as "fastgil" used to be. As fastgil is now slower. This is reproducible. On Python 3.12 on master they're identical by default (which makes sense since I think we disable it). Defining -DCYTHON_FAST_GIL brings us back to roughly the same as 3.11 (i.e. now slower). So my conclusion is that from 3.11 onwards Python sped up their own GIL handling to about the same as we used to have, and fastgil has turned into a pessimization. David On 19/09/2023 11:58, Lisandro Dalcin wrote: Disclaimer: I may be doing something wrong, I did not put a lot of effort into it. With the microbenchmark that was offered in the GH issue, I see little difference. Use the attached zip file to reproduce yourself. Change tox.ini to "cython<3" to try 0.29.x. BTW, in the 0.29.x case, I see no compilation error as claimed in the GH issue. $ ./run.sh CFLAGS=-g0 -Ofast -DCYTHON_FAST_GIL=0 Running test_gil_already_held ... took 0.08735537528991699 Running test_gil_released ... took 0.6329536437988281 py37: OK ✔ in 3.57 seconds Running test_gil_already_held ... took 0.09007453918457031 Running test_gil_released ... took 0.4598276615142822 py38: OK ✔ in 3.19 seconds Running test_gil_already_held ... took 0.10935306549072266 Running test_gil_released ... took 0.4512367248535156 py39: OK ✔ in 3.25 seconds Running test_gil_already_held ... took 0.09970474243164062 Running test_gil_released ... took 0.46637773513793945 py310: OK ✔ in 3.21 seconds Running test_gil_already_held ... took 0.08569073677062988 Running test_gil_released ... took 0.46811795234680176 py311: OK ✔ in 3.22 seconds Running test_gil_already_held ... took 0.15221118927001953 Running test_gil_released ... took 0.2246694564819336 py37: OK (3.57 seconds) py38: OK (3.19 seconds) py39: OK (3.25 seconds) py310: OK (3.21 seconds) py311: OK (3.22 seconds) pypy3.9: OK (5.24 seconds) congratulations :) (21.71 seconds) CFLAGS=-g0 -Ofast -DCYTHON_FAST_GIL=1 Running test_gil_already_held ... took 0.08835673332214355 Running test_gil_released ... took 0.6265637874603271 py37: OK ✔ in 1.42 seconds Running test_gil_already_held ... took 0.09030938148498535 Running test_gil_released ... took 0.456279993057251 py38: OK ✔ in 1.17 seconds Running test_gil_already_held ... took 0.10986089706420898 Running test_gil_released ... took 0.45894527435302734 py39: OK ✔ in 1.2 seconds Running test_gil_already_held ... took 0.10107588768005371 Running test_gil_released ... took 0.5052204132080078 py310: OK ✔ in 1.21 seconds Running test_gil_already_held ... took 0.0855649414062 Running test_gil_released ... took 0.4581136703491211 py311: OK ✔ in 1.13 seconds Running test_gil_already_held ... took 0.15286779403686523 Running test_gil_released ... took 0.22533607482910156 py37: OK (1.42 seconds) py38: OK (1.17 seconds) py39: OK (1.20 seconds) py310: OK (1.21 seconds) py311: OK (1.13 seconds) pypy3.9: OK (1.64 seconds) congratulations :) (7.81 seconds) On Tue, 19 Sept 2023 at 10:09, Stefan Behnel wrote: Hi, I've seen reports that Cython's "FastGIL" implementation (which basically keeps the GIL state in a thread-local variable) is no longer faster than CPython's plain GIL implementation in recent Python 3.x versions. Potentially even slower. See the report in https://github.com/cython/cython/issues/5703 It would be helpful to get user feedback on this. If you have GIL-heavy Cython code, especially with nested with-nogil/with-gil sections across functions, and a benchmark that exercises it, could you please run the benchmark with and without the feature enabled and report the results? You can add "-DCYTHON_FAST_GIL=0" to your CFLAGS to disabled it (and "=1" to enable it explicitly). It's enabled by default in CPython 3.6-3.11 (but disabled in Cython 0.29.x on Python 3.11). Thanks, Stefan ___ cython-devel mailing list cython-devel@python.org https://
Re: [Cython] [cython-users] Re: Can we remove the FastGIL implementation?
One more detail - on 0.29.x it becomes a pessimization in Python 3.10 rather than Python 3.11. So in conclusion | Python <3.10 | Python 3.10 | Python 3.11 | Python 3.12b2 - 0.29.x | fast_gil is better | fast_gil is worse | fast_gil crashes | fast_gil crashes master | fast_gil is better | fast_gil is better | fast_gil is worse | fast_gil is worse (but off by default) On 19/09/2023 20:38, da-woods wrote: I think the detail that was missing is you need to add the `#cython: fast_gil = True` to enable it. For me: Python 3.9 and 3.10 are basically identical (on master) **test_gil_already_held** with fast_gil Running the test... took 0.175062894821167 without Running the test... took 0.10976791381835938 **test_gil_released** with fast_gil Running the test... took 0.583066463470459 without Running the test... took 0.5824759006500244 test_gil_already_held is noticably faster with fast_gil. For Python 3.11: I get the crash in 0.29.x if I try to run using fast_gil. No defines are needed to get that... On master: **test_gil_already_held** with fast_gil Running the test... took 0.17254948616027832 without Running the test... took 0.10958600044250488 **test_gil_released** with fast_gil Running the test... took 0.5791811943054199 without Running the test... took 0.5597968101501465 Note that "without fastgil" is now as fast as "fastgil" used to be. As fastgil is now slower. This is reproducible. On Python 3.12 on master they're identical by default (which makes sense since I think we disable it). Defining -DCYTHON_FAST_GIL brings us back to roughly the same as 3.11 (i.e. now slower). So my conclusion is that from 3.11 onwards Python sped up their own GIL handling to about the same as we used to have, and fastgil has turned into a pessimization. David On 19/09/2023 11:58, Lisandro Dalcin wrote: Disclaimer: I may be doing something wrong, I did not put a lot of effort into it. With the microbenchmark that was offered in the GH issue, I see little difference. Use the attached zip file to reproduce yourself. Change tox.ini to "cython<3" to try 0.29.x. BTW, in the 0.29.x case, I see no compilation error as claimed in the GH issue. $ ./run.sh CFLAGS=-g0 -Ofast -DCYTHON_FAST_GIL=0 Running test_gil_already_held ... took 0.08735537528991699 Running test_gil_released ... took 0.6329536437988281 py37: OK ✔ in 3.57 seconds Running test_gil_already_held ... took 0.09007453918457031 Running test_gil_released ... took 0.4598276615142822 py38: OK ✔ in 3.19 seconds Running test_gil_already_held ... took 0.10935306549072266 Running test_gil_released ... took 0.4512367248535156 py39: OK ✔ in 3.25 seconds Running test_gil_already_held ... took 0.09970474243164062 Running test_gil_released ... took 0.46637773513793945 py310: OK ✔ in 3.21 seconds Running test_gil_already_held ... took 0.08569073677062988 Running test_gil_released ... took 0.46811795234680176 py311: OK ✔ in 3.22 seconds Running test_gil_already_held ... took 0.15221118927001953 Running test_gil_released ... took 0.2246694564819336 py37: OK (3.57 seconds) py38: OK (3.19 seconds) py39: OK (3.25 seconds) py310: OK (3.21 seconds) py311: OK (3.22 seconds) pypy3.9: OK (5.24 seconds) congratulations :) (21.71 seconds) CFLAGS=-g0 -Ofast -DCYTHON_FAST_GIL=1 Running test_gil_already_held ... took 0.08835673332214355 Running test_gil_released ... took 0.6265637874603271 py37: OK ✔ in 1.42 seconds Running test_gil_already_held ... took 0.09030938148498535 Running test_gil_released ... took 0.456279993057251 py38: OK ✔ in 1.17 seconds Running test_gil_already_held ... took 0.10986089706420898 Running test_gil_released ... took 0.45894527435302734 py39: OK ✔ in 1.2 seconds Running test_gil_already_held ... took 0.10107588768005371 Running test_gil_released ... took 0.5052204132080078 py310: OK ✔ in 1.21 seconds Running test_gil_already_held ... took 0.0855649414062 Running test_gil_released ... took 0.4581136703491211 py311: OK ✔ in 1.13 seconds Running test_gil_already_held ... took 0.15286779403686523 Running test_gil_released ... took 0.22533607482910156 py37: OK (1.42 seconds) py38: OK (1.17 seconds) py39: OK (1.20 seconds) py310: OK (1.21 seconds) py311: OK (1.13 seconds) pypy3.9: OK (1.64 seconds) congratulations :) (7.81 seconds) On Tue, 19 Sept 2023 at 10:09, Stefan Behnel wrote: Hi, I've seen reports that Cython's "FastGIL" implementation (which basically keeps the GIL state in a thread-local variable) is no longer faster than CPython's plain GIL implementation in recent Python 3.x versions. Potentially even slower. See the report in https://github.com/cython/cython/issues/5703 It would be helpful to get user feedback on this.
Re: [Cython] Can we remove the FastGIL implementation?
On 20/09/2023 10:27, Stefan Behnel wrote: So – disable the feature in Python 3.11 and later? (Currently it's disabled in 3.12+.) That seems sensible. I think the other question is 0.29.x. On Python 3.11+ it silently produces code that crashes at runtime. We should probably disable it there (at least if there is another 0.29.x release). David ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start using the internal CPython APIs?
I'm a bit late in replying to this but here are some unordered thoughts. * I'm fairly relaxed about using `Py_BUILD_CORE` if useful - I think we mostly do have good fallback paths for most things so can adapt quickly when stuff changes. * CYTHON_USE_CPYTHON_CORE_DETAILS sounds reasonable, but it's yet another variation to test. * I wonder if fixing up the limited API implementation should be higher priority than creating a third level been "full" and "limited API". * I recall we were planning to ditch c89 as a strict requirement after 3.0? Incompatibility with C++ might be more of an issue though. * Even so, if there's a good way of turning it off then we could say: "if you want strict c89 support then you can't use CYTHON_USE_CPYTHON_CORE_DETAILS" and people would always have options. * Waiting and seeing may be a good option for now. David On 30/10/2023 20:42, Stefan Behnel wrote: Thank you for your comments so far. Stefan Behnel schrieb am 29.10.23 um 22:06: I seriously start wondering if we shouldn't just define "Py_BUILD_CORE" (or have our own "CYTHON_USE_CPYTHON_CORE_DETAILS" macro guard that triggers its #define) and include the internal "pycore_*.h" CPython header files from here: https://github.com/python/cpython/tree/main/Include/internal I just remembered that there's a one major technical issue with this. CPython now requires C99 for its own code base (Py3.13 actually uses "-std=c11" on my side). While they care about keeping public header files compatible with C89 and C++, their internal header files may not always have that quality, and won't be tested for it. So, governance is one argument, but technical reasons can also make this appear less appealing overall. I'll let things settle some more and see in what direction Py3.13 will eventually be moving. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Ready for Cython 3.1 ?
I agree with most of this. > I also consider Cython 3.1 a prime target for better Limited API support. Yes - but I wouldn't treat complete support as a blocker (I don't think this is what you meant though). There's a separate question about what we consider the minimum viable Limited API version we want to support. I imagine that'll ultimately be decided by "what we can make work", but I don't think it'll be less that 3.4 (when PyType_GetSlot) was added. It's probably something to decide later. David On 05/11/2023 22:06, Stefan Behnel wrote: Hi all, it looks like Cython 3.0.6 is going to be a "most things fixed" kind of release for the 3.0.x release series. Given the work that lies ahead of us for Cython 3.1, I think we're at a point to get started on that, making the future 3.0.x releases stable and "boring". As a reminder, Cython 3.1 will remove support for Python 2.7 and Python 3.[567], i.e. all Python versions that are now EOL. Python 3.8 will continue to receive security fixes for another year. Python 3.7 is EOL but still up for debate since it's probably not hard to support and still maintained in some Linux distributions for another couple of years. But I'm fine with considering it legacy. We'll probably notice if it gets in the way while preparing Cython 3.0, and can leave support in until there's a reason to remove it. https://github.com/cython/cython/issues/2800 I'd like to ease our feature development by using more modern Python features in our code base and by targeting less Python versions in Cython 3.1 compared to the "all things supported" Cython 3.0. I also consider Cython 3.1 a prime target for better Limited API support. Users probably won't care both for that and for outdated Python versions at the same time. Or, they can use Cython 3.0.x for continued legacy support. Since Cython 3.1 is mostly about ripping out old code, we can try to keep the development cycle short, so that new features don't have to wait that long. Certainly not as long as for Cython 3.0… Is everyone and everything ready to start working on Cython 3.1? Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Contributing to the Cython Project as a beginner
I don't think we've kept the "good first issue" tag on Github rigorously up-to-date, but see if there's anything tagged with that. Here are a few ideas: * The easiest thing to modify is probably the declarations in Cython/Includes to wrap the C and C++ standard libraries and the Python C API. They're missing reasonable chunks of the newer C and C++ standards, and Python C API that's been added in recent Python releases. So you could identify some of that and add it. * We're always keep to reduce warnings when building C code generated by Cython. There's definitely a few new ones that have appeared. Try building Cython itself, or running some tests and see what warnings appear and then see if you can fix them. * The compiler internals tend to be fairly complicated, but you're welcome to have a look. If you wanted to go that way then something like https://github.com/cython/cython/issues/4631 has parts that might be fairly achievable because they can largely be based on what we've done with typing.Optional. I'd suggest doing it in parts and starting with `typing.Union` instead of the newer `A | B` syntax. Hopefully that helps. David On 13/04/2024 02:20, So Min An wrote: Dear Cython Core Development Team, I hope this email finds you well. My name is Somin An and I am a junior studying at the University of Michigan. As part of my final project for a course, I was hoping to work on one or two beginner issues for the Cython project. This is my very first time attempting to contribute to an open-source project and would love to receive some guidance. I would appreciate any information on how I might be able to contribute. I am currently reading through the CONTRIBUTING.rst and the HackerGuide. If you could let me know at your earliest convenience, I would greatly appreciate it. Sincerely, Somin An ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] How to see why distutils-powered tests failed?
Hi Miro, Sorry for the slightly slow reply. When I tried to check this I can definitely see compile errors in the error output from the test. It looks like you're getting a link error though. I'm just going to do a PR to try to fix that. I'll ping you on GitHub when that's done. David On 16/06/2024 00:23, Miro Hrončok wrote: Hi. Apparently, I was not properly subscribed to the list when I sent this, trying again: Forwarded Message Subject: How to see why distutils-powered tests failed? Date: Mon, 13 May 2024 12:10:20 +0200 From: Miro Hrončok To: cython-devel@python.org Hello, I've recently encountered the following test failure when I wanted to run the upstream Cython testusite on Fedora CI. == ERROR: runTest (__main__.CythonRunTestCase.runTest) [-1] compiling (cpp/cy2/pythran) and running numpy_pythran -- Traceback (most recent call last): File "/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", line 266, in link self.spawn(linker + ld_args) File "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", line 1041, in spawn spawn(cmd, dry_run=self.dry_run, **kwargs) File "/usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py", line 70, in spawn raise DistutilsExecError( distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit code 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1550, in run ext_so_path = self.runCompileTest() ^ File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1163, in runCompileTest return self.compile( ^ File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1482, in compile so_path = self.run_distutils(test_directory, module, workdir, incdir) ^^^ File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 1378, in run_distutils build_extension.run() File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run self.build_extensions() File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions self._build_extensions_serial() File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial self.build_extension(ext) File "/var/ARTIFACTS/work-planwuditg_4/plan/discover/default-0/source/cython-3.0.9/runtests.py", line 637, in build_extension _build_ext.build_extension(self, ext) File "/usr/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 572, in build_extension self.compiler.link_shared_object( File "/usr/lib/python3.12/site-packages/setuptools/_distutils/ccompiler.py", line 752, in link_shared_object self.link( File "/usr/lib/python3.12/site-packages/setuptools/_distutils/unixccompiler.py", line 268, in link raise LinkError(msg) distutils.errors.LinkError: command '/usr/bin/g++' failed with exit code 1 There were 4 failures like this (all pythran_numpy related). I was clueless as to why is this failing. I was trying to make the tests tell me more. After long digging I ended up modifying /usr/lib/python3.12/site-packages/setuptools/_distutils/spawn.py in a test container directly to always make this conditional False: https://github.com/pypa/setuptools/blob/v69.1.0/setuptools/_distutils/spawn.py#L61 This turned: distutils.errors.DistutilsExecError: command '/usr/bin/g++' failed with exit code 1 Into: distutils.errors.DistutilsExecError: command ['/usr/bin/g++', ...] failed with exit code 1 Then, I was able to rerun the failed g++ command manually to see the actual linker error (missing flexiblas). However, the experience was far from ideal. Would it be possible (and desired) to modify the tests to actually print out the compiler/linker errors when they happen? Or perhaps this already is possible, but I am not doing it correctly? (I tried --no-capture, but no dice.) Thanks, ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CLI for using Shared utility module
Hi, I try not to think too hard about build systems because I don't really understand them. But: Most people will probably want to do this from setup.py instead of calling an executable, so that's probably the most important interface. I do agree we need something that can be run from the command-line, but the main users are build systems (e.g. scikit-build/ meson) and not people. People who are manually building a single Cython module from the command-line don't really need this. To generate the C code for a normal Cython module you usually just specify the source filename and it works out the output filename and module name from that. Here, we don't have a source filename (so we can't use that). Cython from setuptools usually puts the C files in the same place as the source files. From that, I think I agree with you - to make the shared C file the main argument should be the output path. (I think you should also be able to give the module name as a second argument, like for regular Cython builds). Taking one of Stefan's points > The question then is whether this can be combined with a normal module translation > (and should still generate the shared code file) or whether it's a separate step to do > the shared module generation. I think they should be separate steps. Because doing two things at once is more complicated, and it's for setuptools/meson/scikit-build rather than people. I agree with you that you should use the import path when building modules that use it. David On 11/02/2025 19:25, matus valo via cython-devel wrote: Hi All, I am writing to you to not get lost in https://github.com/cython/cython/pull/6531. I would like to get an agreement on how we would like to proceed with a CLI. The PR has two parts: 1. Utility for generating C file of shared utility. Here, I propose to pass the full path to the file e.g.: cython_shared_module scipy/_lib/cyshared.c The reason why not to use the import path is because the target package can be hidden inside a directory (e.g. src). This does not work well with the import path. Additional questions: * what we will call the utility for creation of the module? * should it be executable or we will use python -m '...'approach? * should utility create directories or to fail if directory does not exist? 2. cython parameter to instruct cython to compile module with shared utility Here, I propose to use import path and compilation directive: cython -X use_shared_utility=scipy._lib.cyshared my_module.pyx The reason to use the import path is because the file must be internally imported in any case. The import path is a full path including the shared module. Could you help me answer the mentioned questions? I think this is last piece of puzzle and after the PR should be finally complete (there are still PR review needed and rough edges to be polished). Thanks, Matus ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Shared utility Option vs. Compilation Directive
Personally: I don't think 3 is worthwhile. I weakly prefer 1 to 2 on the basis that we were trying to get away from Options where possible. However, this is probably a case where Options will usually be right (it forces a setting on every module that you're processing in a batch, which is what most people will want here. But maybe not everyone). David On 03/03/2025 19:47, matus valo wrote: Hi All, Based on the comment https://github.com/cython/cython/pull/6531#discussion_r1975502026 I would like to start a discussion about using Option vs Compilation Directive. Basically, we have 3 options: 1. Use PR https://github.com/cython/cython/pull/6531 as is. I will proceed with the rest of the comments. 2. Use `Option` directly with no further changes in the Cython codebase. This is the initial implementation. I created a branch with a version of the PR with rollback of all changes related to the Compilation Directive adjustments: https://github.com/cython/cython/compare/master...matusvalo:shared_library_option?expand=1 3. Use code changes/cleanup in https://github.com/cython/cython/pull/6646 but Shared Utility will be configured using `Option` instead of Compilation directive (Option 2.). I am not sure it this point makes sense, only in case that we want to keep changes by @da-woods <mailto:dw-...@d-woods.co.uk> improves the code internals. I would like to have an agreement on this topic before moving forward with other comments in the PR. Thanks, Matus___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
