[Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-04 Thread Jeroen Demeyer
Hello, I have a question about PEP 384: can undocumented functions ever be considered as part of the stable ABI? With undocumented, I mean not appearing in the "Python/C API Reference Manual". Whatever the answer to this question is, it would be good to make it explicit in PEP 384. I am in

Re: [Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-04 Thread Jeroen Demeyer
On 2018-04-04 17:56, Guido van Rossum wrote: It would be helpful if you explained the context of your request. The context is PEP 575. I guess my question is mostly about PyCFunction_Check(). I will not be able to keep it 100% backwards compatible simply because the goal of that PEP is precis

Re: [Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-09 Thread Jeroen Demeyer
On 2018-04-08 05:17, Nick Coghlan wrote: Changing macro definitions doesn't break the stable ABI, as long as the *old* macro expansions still do the right thing. To me, it looks like a bad idea to change macros. Imagine that the PyCFunction_Check macro changes in Python 3.8. Then an extension

Re: [Python-Dev] Are undocumented functions part of the stable ABI?

2018-04-11 Thread Jeroen Demeyer
On 2018-04-10 13:49, Nick Coghlan wrote: If it's only a semantic level change in the way the macro gets expanded, then whether or not it needs an ABI version guard gets judged on a case-by-case basis, and in this particular case, my view would be that developers should be able to write extensions

[Python-Dev] PEP 575: Unifying function/method classes

2018-04-12 Thread Jeroen Demeyer
Dear Python developers, I would like to request a review of PEP 575, which is about changing the classes used for built-in functions and Python functions and methods. The text of the PEP can be found at https://www.python.org/dev/peps/pep-0575/ No substantial changes to the contents of the P

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-13 Thread Jeroen Demeyer
To make it easier to test and try out PEP 575, I created a binder repo: https://mybinder.org/v2/gh/jdemeyer/pep575binder.git /master?filepath=index.ipynb ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-d

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-14 Thread Jeroen Demeyer
On 2018-04-13 21:30, Raymond Hettinger wrote: It would be nice to have a section that specifically discusses the implications with respect to other existing function-like tooling: classmethod, staticmethod, partial, itemgetter, attrgetter, methodgetter, etc. My hope is that there are no such

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-14 Thread Jeroen Demeyer
On 2018-04-13 15:23, Nick Coghlan wrote: There's also a section in the rationale which refers to METH_USRx flags, which I'm guessing from context are an idea you were considering proposing, but eventually dropped from the rest of the PEP. No, I actually still want to propose it. In my latest up

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-15 Thread Jeroen Demeyer
On 2018-04-14 23:14, Guido van Rossum wrote: That actually sounds like a pretty big problem. I'm sure there is lots of code that doesn't *just* duck-type nor calls inspect but uses isinstance() to decide how to extract the desired information. In the CPython standard library, the *only* fixes t

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-16 Thread Jeroen Demeyer
On 2018-04-16 02:32, Raymond Hettinger wrote: I don't think that confidence is warranted. The world of Python is very large. When public APIs (such as that in the venerable types module) get changed, is virtually assured that some code will break. Yes, *some* code will break, I never denied

Re: [Python-Dev] PEP 572: Assignment Expressions

2018-04-17 Thread Jeroen Demeyer
On 2018-04-18 02:13, Chris Angelico wrote: I'm much happier promoting a full-featured assignment expression than something that can only be used in a limited set of situations. Is there reason to believe that extensions to the := operator might take it in a different direction? If not, there's ve

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-20 Thread Jeroen Demeyer
On 2018-04-14 23:14, Guido van Rossum wrote: That actually sounds like a pretty big problem. I'm sure there is lots of code that doesn't *just* duck-type nor calls inspect but uses isinstance() to decide how to extract the desired information. I have been thinking about this some more... One s

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-23 Thread Jeroen Demeyer
Hello, I just saw this PEP. There is a bit of overlap between PEP 573 and PEP 575 since these both change the calling convention for built-in methods. In particular, PEP 575 also proposes to add a "defining class" member (for different reasons). In PEP 575, this is added to the PyCFunction st

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-24 Thread Jeroen Demeyer
In PEP 573, instead of passing the defining class to the C function, why not pass the function object itself? That is far more general: once you have the function object, you can still access the defining class using your PyCMethod_CLASS. It's also more future-proof: if we ever decide to add ev

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-24 Thread Jeroen Demeyer
On 2018-04-24 14:53, Nick Coghlan wrote: In PEP 575, I'm already proposing a flag (METH_ARG0_FUNCTION) to pass the function *instead* of self. Unless PEP 573 is rejected, maybe that should change to passing the function *in addition* to self. That would definitely be an elegant way of addressin

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-24 Thread Jeroen Demeyer
On 2018-04-24 16:34, Jeroen Demeyer wrote: On the other hand, if you are passing the function object, then you can get __self__ from it (unless it's an unbound method: in that case __self__ is NULL and self is really args[0]). So there wouldn't be a need for passing "self".

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-24 Thread Jeroen Demeyer
On 2018-04-20 12:02, Jeroen Demeyer wrote: One solution to improve backwards compatibility would be to duplicate some classes. For example, make a separate class for bound methods in extension types, which would be literally a duplicate of the existing types.MethodType class (possibly with a

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-25 Thread Jeroen Demeyer
On 2018-04-25 20:33, Petr Viktorin wrote: Perhaps "m_objclass" could point to the module in this case That was exactly my idea also today. Instead of treating m_objclass as the defining class, we should generalize it to be the "parent" of the function: either the class or the module. __

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-26 Thread Jeroen Demeyer
- In Python code, __objclass__ should be the defining class, not the module. Indeed. My idea would be to add an accessor __parent__ returning the m_parent field (whatever it is) and then implement __objclass__ as something like: @property def __objclass__(self): parent = getattr(self, "_

Re: [Python-Dev] PEP 573 -- Module State Access from C Extension Methods

2018-04-26 Thread Jeroen Demeyer
On 2018-04-26 16:37, Nick Coghlan wrote: PEP 487 refers to this as the "owner" of a descriptor That's just copied from the Python docs: https://docs.python.org/3.8/reference/datamodel.html#object.__get__ Anyway, I never liked the name "owner" there, "cls" would have been much clearer. For

[Python-Dev] PEP 575 (Unifying function/method classes) update

2018-04-27 Thread Jeroen Demeyer
Hello all, I have updated PEP 575 and its reference implementation. See https://www.python.org/dev/peps/pep-0575/ The main differences with respect to the previous version are: * METH_PASS_FUNCTION now passes the function *in addition* to self (previously, it was passed *instead* of self). *

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-04-30 Thread Jeroen Demeyer
On 2018-04-30 15:38, Mark Shannon wrote: While a unified *interface* makes sense, a unified class hierarchy and implementation, IMO, do not. The main reason for the common base class is performance: in the bytecode interpreter, when we call an object, CPython currently has a special case for

Re: [Python-Dev] PEP 575: Unifying function/method classes

2018-05-03 Thread Jeroen Demeyer
On 2018-05-03 11:30, Victor Stinner wrote: Please don't queue backward incompatible changes for Python 4.0. You should use the regular deprecation process. I don't really see how that can be done here. As Stefan said The problem is that this change does not really fit into the deprecation cyc

[Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-05 Thread Jeroen Demeyer
Hello all, I have updated PEP 575 in response to some posts on this mailing list and to some discussions in person with the core Cython developers. See https://www.python.org/dev/peps/pep-0575/ The main differences with respect to the previous version are: * "builtin_function" was renamed to

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-08 Thread Jeroen Demeyer
On 2018-05-06 09:35, Nick Coghlan wrote: Thanks for this update Jeroen! If it doesn't come up otherwise, I'll try to claim one of the lightning talk slots at the Language Summit to discuss this with folks in person :) Sounds great! I'd love to hear what people think. As an example of how the n

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-14 22:38, Petr Viktorin wrote: Why are these flags added? They aren't free – the space of available flags is not infinite. If something (Cython?) needs eight of them, it would be nice to mention the use case, at least as an example. What should Python do with a m_methods entry that ha

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-14 19:56, Petr Viktorin wrote: It does quite a lot of things, and the changes are all intertwined, which will make it hard to get reviewed and accepted. The problem is that many things *are* already intertwined currently. You cannot deal with functions without involving methods for

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-14 22:38, Petr Viktorin wrote: Why are these flags added? I made a minor edit to the PEP to remove those flags: https://github.com/python/peps/pull/649 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listin

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-15 18:36, Petr Viktorin wrote: What is your ultimate use case? (I'll just answer this one question now and reply to the more technical comments in another thread) My ultimate use case is being able to implement functions and methods which are (A) equally fast as the existing bu

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-15 Thread Jeroen Demeyer
On 2018-05-15 18:36, Petr Viktorin wrote: Naturally, large-scale changes have less of a chance there. Does it really matter that much how large the change is? I think you are focusing too much on the change instead of the end result. As I said in my previous post, I could certainly make less

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-16 Thread Jeroen Demeyer
On 2018-05-16 17:31, Petr Viktorin wrote: The larger a change is, the harder it is to understand I already disagree here... I'm afraid that you are still confusing the largeness of the *change* with the complexity of the *result* after the change was implemented. A change that *removes* compl

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-17 Thread Jeroen Demeyer
On 2018-05-16 17:31, Petr Viktorin wrote: Less disruptive changes tend to have a better backwards compatibility story. A less intertwined change makes it easier to revert just a single part, in case that becomes necessary. I'll just repeat what I said in a different post on this thread: we can

Re: [Python-Dev] PEP: 576 Title: Rationalize Built-in function classes

2018-05-20 Thread Jeroen Demeyer
On 2018-05-19 11:15, mark wrote: PEP 576 aims to fulfill the same goals as PEP 575 (this is a copy of my comments on GitHub before this PEP was official) **Performance** Most importantly, changing bound methods of extension types from builtin_function_or_method to bound_method will yield a p

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-05-20 Thread Jeroen Demeyer
On 2018-05-19 15:29, Nick Coghlan wrote: That's not how code reviews work, as their complexity is governed by the number of lines changed (added/removed/modified), not just the number of lines that are left at the end. Of course, you are right. I didn't mean literally that only the end result

Re: [Python-Dev] PEP: 576 Title: Rationalize Built-in function classes

2018-05-22 Thread Jeroen Demeyer
For the record: the only reason that I replied on GitHub was because the proposal was not yet posted (as far as I know) to any mailing list. Typically, a post is made to a mailing list more or less at the same time as creating the PEP. In this case, there was a delay of a few days, maybe also

Re: [Python-Dev] Stable ABI

2018-06-01 Thread Jeroen Demeyer
On 2018-06-01 17:18, Nathaniel Smith wrote: Unfortunately, very few people use the stable ABI currently, so it's easy for things like this to get missed. So there are no tests for the stable ABI in Python? ___ Python-Dev mailing list Python-Dev@pytho

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-17 Thread Jeroen Demeyer
Hello, I have been working on a slightly different PEP to use a new type slot tp_ccalloffset instead the base_function base class. You can see the work in progress here: https://github.com/jdemeyer/PEP-ccall By creating a new protocol that each class can implement, there is a full decouplin

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-17 Thread Jeroen Demeyer
On 2018-06-17 14:50, Ronald Oussoren wrote: This looks interesting. Why did you add a tp_ccalloffset slot to the type with the actual information in instances instead of storing the information in a slot? Think of built-in functions. Every built-in function is a different callable and calls

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-17 Thread Jeroen Demeyer
On 2018-06-18 03:34, INADA Naoki wrote: Victor had tried to add `tp_fastcall` slot, but he suspended his effort because it's benefit is not enough for it's complexity. https://bugs.python.org/issue29259 I has a quick look at that patch and it's really orthogonal to what I'm proposing. I'm prop

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-18 Thread Jeroen Demeyer
On 2018-06-18 15:09, Victor Stinner wrote: 2) we implemented a lot of other optimizations which made calls faster without having to touch tp_call nor tp_fastcall. And that's a problem because these optimizations typically only work for specific classes. My PEP wants to replace those by somethi

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-18 Thread Jeroen Demeyer
On 2018-06-18 16:55, INADA Naoki wrote: Speeding up most python function and some bultin functions was very significant. But I doubt making some 3rd party call 20% faster can make real applications significant faster. These two sentences are almost contradictory. I find it strange to claim tha

Re: [Python-Dev] PEP 575 (Unifying function/method classes) update

2018-06-19 Thread Jeroen Demeyer
On 2018-06-18 15:09, Victor Stinner wrote: There are multiple issues with tp_fastcall: Personally, I think that you are exaggerating these issues. Below, I'm writing the word FASTCALL to refer to tp_fastcall in your patch as well as my C call protocol in the PEP-in-progress. * ABI issue: i

[Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-20 Thread Jeroen Demeyer
Hello, Let me present PEP 579 and PEP 580. PEP 579 is an informational meta-PEP, listing some of the issues with functions/methods implemented in C. The idea is to create several PEPs each fix some part of the issues mentioned in PEP 579. PEP 580 is a standards track PEP to introduce a new "

Re: [Python-Dev] C-level calling

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 08:00, Stefan Behnel wrote: Just to add another bit of background on top of the current discussion, there is an idea around, especially in the scipy/big-data community, (and I'm not giving any guarantees here that it will lead to a PEP + implementation, as it depends on people's wor

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 16:09, Antoine Pitrou wrote: But there seems to be some complication on top of that: - PyCCall_FastCall() accepts several types for the keywords, even a dict; That is actually a *simplification* instead of a *complication*. Currently, there is a huge amount of code duplicatio

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 16:42, Antoine Pitrou wrote: I'm wondering what amount of code and debugging is needed for, say, Cython or Numba to implement that protocol as a caller, without going through the C API's indirections (for performance). The goal is to have a really fast C API without a lot of indir

Re: [Python-Dev] Can we make METH_FASTCALL public, from Python 3.7? (ref: PEP 579

2018-06-20 Thread Jeroen Demeyer
On 2018-06-20 17:42, INADA Naoki wrote: I don't have any idea about changing METH_FASTCALL more. If Victor and Serhiy think so, and PyPy maintainers like it too, I want to make it public as soon as possible. There are two different things here: The first is documenting METH_FASTCALL such that

Re: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods

2018-06-21 Thread Jeroen Demeyer
On 2018-06-21 11:22, Victor Stinner wrote: https://www.python.org/dev/peps/pep-0580/#the-c-call-protocol CCALL_VARARGS: cc_func(PyObject *self, PyObject *args) If we add a new calling convention This is not a *new* calling convention, it's the *existing* calling convention for METH_VARARGS.

[Python-Dev] About [].append == [].append

2018-06-21 Thread Jeroen Demeyer
Currently, we have: >>> [].append == [].append False However, with a Python class: >>> class List(list): ... def append(self, x): super().append(x) >>> List().append == List().append True In the former case, __self__ is compared using "is" and in the latter case, it is compared using "=="

Re: [Python-Dev] About [].append == [].append

2018-06-21 Thread Jeroen Demeyer
On 2018-06-21 13:33, Ivan Pozdeev via Python-Dev wrote: First, tell us what problem you're solving. There is no specific problem I want to solve here. I just noticed an inconsistency and I wondered if it would be OK to change the implementation of comparisons of builtin_function_or_method ins

[Python-Dev] PEP 580 (C call protocol) draft implementation

2018-06-22 Thread Jeroen Demeyer
Hello all, I have a first draft implementation of PEP 580 (introducing the C call protocol): https://github.com/jdemeyer/cpython/tree/pep580 Almost all tests pass, only test_gdb and test_pydoc fail for me. I still have to fix those. Jeroen. ___

Re: [Python-Dev] About [].append == [].append

2018-06-23 Thread Jeroen Demeyer
On 2018-06-23 03:50, Steven D'Aprano wrote: I think it is more important that builtin methods and Python methods behave the same. +1 This inconsistency is the *real* problem here. It's one little extra complication to merging those classes (which was proposed in PEP 575, 576 and 579). _

[Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
Hello, On https://github.com/python/cpython/pull/7909 I encountered friction for a PR which I expected to be uncontroversial: it just moves some code without changing any functionality. So basically my question is: is there some CPython policy *against* refactoring code to make it easier to

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:11, Ivan Pozdeev via Python-Dev wrote: AFAICS, your PR is not a strict improvement What does "strict improvement" even mean? Many changes are not strict improvements, but still useful to have. Inada pointed me to YAGNI (https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:54, INADA Naoki wrote: ​No, YAGNI is posted by someone and they removed their comment. Sorry for that, I misunderstood the email that GitHub sent me. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/li

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:54, Ivan Pozdeev via Python-Dev wrote: This is exactly what that the YAGNI principle is about, and Inada was right to point to it. Until you have an immediate practical need for something, you don't really know the shape and form for it that you will be the most comfortable with.

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 13:54, INADA Naoki wrote: Real need is important than my preference. If it is needed PEP 580, I'm OK. Of course it's not needed. I never claimed that it was. I think it's *nice to have* right now and slightly more *nice to have* when changes/additions are made to call.c in the

Re: [Python-Dev] PEP 576

2018-06-26 Thread Jeroen Demeyer
On 2018-06-26 21:43, Mark Shannon wrote: https://github.com/markshannon/pep-576 Your version of PEP 576 looks very different from the "official" PEP 576 at https://www.python.org/dev/peps/pep-0576/ So can you please make a pull request to https://github.com/python/peps/pulls Also feel free to

Re: [Python-Dev] Policy on refactoring/clean up

2018-06-26 Thread Jeroen Demeyer
On 2018-06-27 00:02, Guido van Rossum wrote: And TBH a desire to refactor a lot of code is often a sign of a relatively new contributor who hasn't learned their way around the code yet, so they tend to want to make the code follow their understanding rather than letting their understanding follow

Re: [Python-Dev] PEP 576

2018-06-27 Thread Jeroen Demeyer
On 2018-06-26 21:43, Mark Shannon wrote: https://github.com/markshannon/pep-576 This actually looks close to Victor Stinner's bpo-29259. But instead of storing the function pointer in the class, you're storing it in the instance. One concern that I have is that this might lead to code dupli

[Python-Dev] Comparing PEP 576 and PEP 580

2018-07-03 Thread Jeroen Demeyer
Hello all, in order to make reviewing PEP 576/580 easier and possibly take some ideas from one PEP to the other, let me state the one fundamental difference between these PEPs. There are many details in both PEPs that can still change, so I'm focusing on what I think is the big structural dif

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-04 Thread Jeroen Demeyer
On 2018-07-04 03:31, INADA Naoki wrote: I think both PEPs are relying on FASTCALL calling convention, and can't be accepted until FASTCALL is stable & public. First of all, the fact that FASTCALL has not been made public should not prevent from discussing those PEPs and even making a (provision

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-05 Thread Jeroen Demeyer
On 2018-07-05 05:41, INADA Naoki wrote: And stabilizing calling convention is prerequirements of designing new calling APIs. I don't see why. I made my PEP with the assumption that the METH_FASTCALL calling convention won't change. As far as I know, nobody advocated for changing it. But even

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-05 Thread Jeroen Demeyer
On 2018-07-05 13:32, INADA Naoki wrote: Core devs interested in this area is limited resource. I know and unfortunately there is nothing that I can do about that. It would be a pity that PEP 580 (or a variant like PEP 576) is not accepted simply because no core developer cares enough. As f

[Python-Dev] On the METH_FASTCALL calling convention

2018-07-05 Thread Jeroen Demeyer
Hello all, As discussed in some other threads ([1], [2]), we should discuss the METH_FASTCALL calling convention. For passing only positional arguments, a C array of Python objects is used, which is as fast as it can get. When the Python interpreter calls a function, it builds that C array o

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-05 Thread Jeroen Demeyer
On 2018-07-05 14:20, INADA Naoki wrote: What you can do is "divide and conquer". Split PEP in small topics we can focus. The PEP is already small and focused, I really did my best to make it as minimal as possible. I don't see a meaningful way to split it up even further. __

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-06 Thread Jeroen Demeyer
On 2018-07-05 21:57, Guido van Rossum wrote: Would it be possible to get outside experts to help? I don't understand what you mean: to help with what? Designing the PEP? Discussing the PEP? Accepting the PEP? Lobbying Python core devs? The Cython developers (in particular Stefan Behnel) cert

Re: [Python-Dev] On the METH_FASTCALL calling convention

2018-07-06 Thread Jeroen Demeyer
On 2018-07-06 06:07, INADA Naoki wrote: Maybe, one way to improve METH_FASTCALL | METH_KEYWORDS can be this. kwds can be either tuple or dict. But that would be just pushing the complexity down to the callee. I'd rather have a simpler protocol at the expense of a slightly more complex support

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-06 Thread Jeroen Demeyer
On 2018-07-05 14:20, INADA Naoki wrote: like you ignored my advice about creating realistic benchmark for calling 3rd party callable before talking about performance... I didn't really want to ignore that, I just didn't know what to do. As far as I can tell, the official Python benchmark suite

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-06 Thread Jeroen Demeyer
On 2018-07-06 23:12, Guido van Rossum wrote: It's your PEP. And you seem to be struggling with something. But I can't tell quite what it is you're struggling with. To be perfectly honest (no hard feelings though!): what I'm struggling with is getting feedback (either positive or negative) from

Re: [Python-Dev] PEP 575, 576, 579 and 580

2018-07-07 Thread Jeroen Demeyer
On 2018-07-07 15:38, Mark Shannon wrote: Hi, We seem to have a plethora of PEPs where we really ought to have one (or none?). - PEP 575 has been withdrawn. - PEP 579 is an informational PEP with the bigger picture; it does contain some of the requirements that you want to discuss here. - PEP

Re: [Python-Dev] Comparing PEP 576 and PEP 580

2018-07-07 Thread Jeroen Demeyer
On 2018-07-07 14:54, Mark Shannon wrote: There is a minimal implementation and has been for a while. There is a link at the bottom of the PEP. Yes, I saw that but the implementation does not correspond to the PEP. In particular, this sentence from the PEP has not been implemented: When bindi

Re: [Python-Dev] On the METH_FASTCALL calling convention

2018-07-08 Thread Jeroen Demeyer
On 2018-07-07 10:55, Serhiy Storchaka wrote: The first part of handling arguments can be made outside of the C function, by the calling API. Sure, it could be done but I don't see the advantage. I don't think you will gain performance because you are just moving code from one place to another

Re: [Python-Dev] PEP 575, 576, 579 and 580

2018-07-09 Thread Jeroen Demeyer
On 2018-07-08 23:13, Mark Shannon wrote: I've added you suggestion, and everyone else's, to this github repo: https://github.com/markshannon/extended-calling-convention Feel free to comment on github, submit PRs or just email me directly if you have anything else you want to add. Do you agree

[Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580)

2018-07-09 Thread Jeroen Demeyer
Here is an initial version of a micro-benchmark for C function calling: https://github.com/jdemeyer/callbench I don't have results yet, since I'm struggling to find the right options to "perf timeit" to get a stable result. If somebody knows how to do this, help is welcome. Jeroen.

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-10 Thread Jeroen Demeyer
OK, I tried with --duplicate 200 and you can see the results at https://gist.github.com/jdemeyer/f0d63be8f30dc34cc989cd11d43df248 In short, the timings with and without PEP 580 are roughly the same (which is to be expected). Interestingly, a small but significant improvement can be seen when ca

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-10 Thread Jeroen Demeyer
On 2018-07-10 14:59, INADA Naoki wrote: Currently, we create temporary long object for passing argument. If there is protocol for exposeing format used by PyArg_Parse*, we can bypass temporal Python object and call myfunc_impl directly. Indeed, I already mentioned this at https://www.python.org

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-11 Thread Jeroen Demeyer
On 2018-07-11 00:48, Victor Stinner wrote: About your benchmark results: "FASTCALL unbound method(obj, 1, two=2): Mean +- std dev: 42.6 ns +- 29.6 ns" That's a very big standard deviation :-( Yes, I know. My CPU was overheating and was slowed down. But that seemed to have happened for a smal

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-11 Thread Jeroen Demeyer
On 2018-07-11 10:27, Antoine Pitrou wrote: I agree PEP 580 is extremely complicated and it's not obvious what the maintenance burden will be in the long term. But the status quo is also very complicated! If somebody would write a PEP describing the existing implementation of builtin_function_o

Re: [Python-Dev] Micro-benchmarks for PEP 580

2018-07-11 Thread Jeroen Demeyer
On 2018-07-11 10:50, Victor Stinner wrote: As you wrote, the cost of function costs is unlikely the bottleneck of application. With that idea, METH_FASTCALL is not needed either. I still find it very strange that nobody seems to question all the crazy existing optimizations for function calls

[Python-Dev] PEP 580 (C call protocol) minor update

2018-07-16 Thread Jeroen Demeyer
I made some minor updates to PEP 580 (PEP editors: please merge https://github.com/python/peps/pull/741) and its reference implementation: - Added a new introductory section explaining the basic idea. - The C protocol no longer deals with __name__; a __name__ attribute is required but the prot

[Python-Dev] Specification of C function profiling?

2018-07-16 Thread Jeroen Demeyer
Hello, it seems to me that there is no clear specification for the sys.setprofile() event c_call: the docs say "A C function is about to be called" but it's not clear what that means exactly, in particular when that C function is an unbound method like list.append. I also noticed that Lib/te

[Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-21 Thread Jeroen Demeyer
Hello, I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython compilation of SageMath. By default, a function in Cython is an instance of builtin_function_or_method (analogously, method_descriptor for a

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-21 Thread Jeroen Demeyer
On 2018-07-21 19:07, INADA Naoki wrote: Good job. But I already +1 for adding support for extension callable type. Do you think this benchmark can be optimized more in future optimization which is possible by PEP 580, but not 576? I should clarify that the benchmark did not involve an implemen

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-21 Thread Jeroen Demeyer
On 2018-07-21 19:55, Guido van Rossum wrote: I don’t think we can safely assume Python 3.7 has the same performance, actually. A lot has changed. I'm not denying that some things have changed. Rather, I claim that those changes wouldn't invalidate the benchmarks. I am comparing calls through

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 08:27, INADA Naoki wrote: It's interesting... But I failed to build sage. What went wrong? It's build step is too different from normal Python package. That's true because Sage considers itself a distribution rather than a package. But it's possible to pick the distribution a

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 01:14, Guido van Rossum wrote: Jeroen was asked to provide benchmarks but only provided them for Python 2. The reasoning that not much has changed that could affect the benchmarks feels a bit optimistic, that’s all. The micro-benchmarks showed a clear difference on Python 3.8 (git

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 18:14, Guido van Rossum wrote: Sorry I don't have better news. I don't consider that particularly bad news (but not good news either). I feel like the status on PEP 580 isn't anywhere near accepted anyway. I just hope that Python development won't stall completely. Even if no

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 14:52, Stefan Behnel wrote: Someone has to maintain the *existing* code base and help newcomers to get into it and understand it. This is an important point that people seem to be overlooking. The complexity and maintenance burden of PEP 580 should be compared to the status-quo.

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-22 Thread Jeroen Demeyer
On 2018-07-22 22:32, Antoine Pitrou wrote: Two things: - the issue26110 changes are not very large, it's just two additional opcodes and a bit of support code I didn't mean to compare PEP 580 to that specific issue, it was just an example. Even if issue26110 is small, the total complexity a

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
I did exactly the same benchmark again with Python 3.7 and the results are similar. I'm copying and editing the original post for completeness: I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython compil

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
On 2018-07-23 01:54, Ivan Pozdeev via Python-Dev wrote: All the material to discuss that we have in this thread is a single test result that's impossible to reproduce and impossible to run in Py3. I just posted that it can be reproduced on Python 3.7: https://mail.python.org/pipermail/python-de

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
On 2018-07-23 00:28, Guido van Rossum wrote: So does your implementation of the PEP result in a net increase or decrease of the total lines of code? 12 files changed, 918 insertions(+), 704 deletions(-) That's a net increase, so there is no obvious win here. Still, I have various excuses for

Re: [Python-Dev] Benchmarks why we need PEP 576/579/580

2018-07-23 Thread Jeroen Demeyer
On 2018-07-23 12:13, Antoine Pitrou wrote: IMHO the main point of discussion should be judging the solution you are proposing Yes please. I would very much welcome a discussion about the actual content of the PEP instead of meta-discussions like how complicated it is. ___

[Python-Dev] PEP 576/579/580 benchmark: mistune

2018-07-27 Thread Jeroen Demeyer
Hello all, since my latest benchmark for PEP 580 [1] involved SageMath, which is quite a big project, I instead propose a much simpler benchmark involving mistune. mistune [2] is a Markdown parser implemented in the Python language. It optionally allows Cython compilation. It doesn't use any

Re: [Python-Dev] Let's change to C API!

2018-07-29 Thread Jeroen Demeyer
My first impression is that making things faster and hiding implementation details in the ABI are contrary goals. I agree with hiding implementation details in the API but not in the ABI. For example, you mention that you want to make Py_INCREF() a function call instead of a macro. But since P

[Python-Dev] Error message for wrong number of arguments

2018-07-30 Thread Jeroen Demeyer
Hello, I noticed an inconsistency in the error messages for the number of arguments to a method call. For Python methods, the "self" argument is counted. For built-in methods, the "self" argument is *not* counted: >>> class mylist(list): ... def append(self, val): super().append(val) >>>

Re: [Python-Dev] PEP 576/579/580 benchmark: mistune

2018-07-30 Thread Jeroen Demeyer
On 2018-07-30 13:11, INADA Naoki wrote: Like previous SageMath bench, this is caused by Cython's specialization; __Pyx_PyObject_CallOneArg. It specializing calling PyFunction and PyCFunction, but it doesn't specialized for calling CyFunction. Yes, I saw that too. But this is exactly what CPyth

Re: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython

2018-07-30 Thread Jeroen Demeyer
On 2018-07-30 15:35, INADA Naoki wrote: As repeatedly said, PEP 580 is very complicated protocol when just implementing callable object. Can you be more concrete what you find complicated? Maybe I can improve the PEP to explain it more. Also, I'm open to suggestions to make it less complicate

Re: [Python-Dev] Error message for wrong number of arguments

2018-07-30 Thread Jeroen Demeyer
On 2018-07-30 20:22, Chris Barker wrote: is it possible for the interpreter to know when this error is generated that this is a bound method expecting a "self", rather than an arbitrary function with n parameters? That would be quite hard. The error message is generated by the underlying funct

  1   2   3   >