[Python-Dev] Benchmarks why we need PEP 576/579/580
Hello, I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython compilation of SageMath. By default, a function in Cython is an instance of builtin_function_or_method (analogously, method_descriptor for a method), which has special optimizations in the CPython interpreter. But the option "binding=True" changes those to a custom class which is NOT optimized. I ran the full SageMath testsuite several times without and with binding=True to find out any significant differences. The most dramatic difference is multiplication for generic matrices. More precisely, with the following command: python -m timeit -s "from sage.all import MatrixSpace, GF; M = MatrixSpace(GF(9), 200).random_element()" "M * M" With binding=False, I got 10 loops, best of 3: 692 msec per loop With binding=True, I got 10 loops, best of 3: 1.16 sec per loop This is a big regression which should be gone completely with PEP 580. I should mention that this was done on Python 2.7.15 (SageMath is not yet ported to Python 3) but I see no reason why the conclusions shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 and Cython 0.28.4. I hope that this finally shows that the problems mentioned in PEP 579 are real. Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
On Sun, Jul 22, 2018 at 1:28 AM Jeroen Demeyer wrote: > > Hello, > > I finally managed to get some real-life benchmarks for why we need a > faster C calling protocol (see PEPs 576, 579, 580). Good job. But I already +1 for adding support for extension callable type. Do you think this benchmark can be optimized more in future optimization which is possible by PEP 580, but not 576? > > I should mention that this was done on Python 2.7.15 (SageMath is not > yet ported to Python 3) but I see no reason why the conclusions > shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 > and Cython 0.28.4. Do you mean you backport LOAD_METHOD and fastcall to Python 2.7 for benchmarking? Reproducing it seems hard job to me... -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
I don’t think we can safely assume Python 3.7 has the same performance, actually. A lot has changed. On Sat, Jul 21, 2018 at 10:10 AM INADA Naoki wrote: > On Sun, Jul 22, 2018 at 1:28 AM Jeroen Demeyer wrote: > > > > Hello, > > > > I finally managed to get some real-life benchmarks for why we need a > > faster C calling protocol (see PEPs 576, 579, 580). > > Good job. But I already +1 for adding support for extension callable type. > Do you think this benchmark can be optimized more in future optimization > which is possible by PEP 580, but not 576? > > > > > I should mention that this was done on Python 2.7.15 (SageMath is not > > yet ported to Python 3) but I see no reason why the conclusions > > shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 > > and Cython 0.28.4. > > Do you mean you backport LOAD_METHOD and fastcall to Python 2.7 > for benchmarking? > Reproducing it seems hard job to me... > > -- > INADA Naoki > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
On 2018-07-21 19:07, INADA Naoki wrote: Good job. But I already +1 for adding support for extension callable type. Do you think this benchmark can be optimized more in future optimization which is possible by PEP 580, but not 576? I should clarify that the benchmark did not involve an implementation of PEP 576 or PEP 580. It simply shows what kind of regressions one gets when *not* implementing something like those PEPs. So this can't be used to compare PEP 576 versus PEP 580. I still think that PEP 576 would slow down bound method calls but without a reference implementation, I can only guess. (I know that PEP 576 claims a reference implementation but it doesn't correspond to the PEP. I'm basing myself on the text of PEP 576, not the "reference implementation".) Do you mean you backport LOAD_METHOD and fastcall to Python 2.7 for benchmarking? No, I did not. This is just benchmarking the difference between tp_call and more specialized call functions (In Python 2.7, that is call_function()). Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
On 2018-07-21 19:55, Guido van Rossum wrote: I don’t think we can safely assume Python 3.7 has the same performance, actually. A lot has changed. I'm not denying that some things have changed. Rather, I claim that those changes wouldn't invalidate the benchmarks. I am comparing calls through tp_call (A) versus optimized call paths (B). I only need to assume that the speed improvements to (A) between 2.7 and 3.7 are not bigger than the speed improvements to (B). Most optimizations which have been done in Python 3.x target (B). In fact, I'm not aware of any optimization to (A) apart from maybe some minor code improvements. So I think it's a relatively safe assumption that the speed difference between (A) and (B) did not decrease from 2.7 to 3.7. Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
Given the cost of a mistake here I recommend a higher standard. But in the end it’s no longer my decision. On Sat, Jul 21, 2018 at 11:18 AM Jeroen Demeyer wrote: > On 2018-07-21 19:55, Guido van Rossum wrote: > > I don’t think we can safely assume Python 3.7 has the same performance, > > actually. A lot has changed. > > I'm not denying that some things have changed. Rather, I claim that > those changes wouldn't invalidate the benchmarks. > > I am comparing calls through tp_call (A) versus optimized call paths > (B). I only need to assume that the speed improvements to (A) between > 2.7 and 3.7 are not bigger than the speed improvements to (B). > > Most optimizations which have been done in Python 3.x target (B). In > fact, I'm not aware of any optimization to (A) apart from maybe some > minor code improvements. So I think it's a relatively safe assumption > that the speed difference between (A) and (B) did not decrease from 2.7 > to 3.7. > > > Jeroen. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
Guido van Rossum schrieb am 21.07.2018 um 22:46: > Given the cost of a mistake here I recommend a higher standard. May I ask what you think the "cost of a mistake" is here? Jeroen has already implemented most of this, and is willing to provide essentially a shrink-wrapped implementation. He has shown, using the current application benchmark suite, that his implementation does not degrade the application performance (that we know of). He has motivated in PEP form, and shown in his implementation, that the changes avoid much of the special casing that's currently littered in various spots of the interpreter and replace them by a much clearer protocol, thus reducing the overall maintenance cost. He has layed out a cleanup path to get rid of the current quirks in the split between function/method types, thus making the code easier to explain and lowering the entry barrier for newcomers to the code base. And, he has motivated that this protocol enables a future extension towards a specialised (faster) C level call protocol, which third party extensions would benefit from. Given all that, I'm having a hard time finding a "cost" in this. To me, it reads like a plain net win for all sides. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
The cost would be if we were to end up maintaining all that code and it wouldn’t make much difference. Jeroen was asked to provide benchmarks but only provided them for Python 2. The reasoning that not much has changed that could affect the benchmarks feels a bit optimistic, that’s all. The new BDFL may be less demanding though. :=) On Sat, Jul 21, 2018 at 2:39 PM Stefan Behnel wrote: > Guido van Rossum schrieb am 21.07.2018 um 22:46: > > Given the cost of a mistake here I recommend a higher standard. > > May I ask what you think the "cost of a mistake" is here? > > Jeroen has already implemented most of this, and is willing to provide > essentially a shrink-wrapped implementation. He has shown, using the > current application benchmark suite, that his implementation does not > degrade the application performance (that we know of). He has motivated in > PEP form, and shown in his implementation, that the changes avoid much of > the special casing that's currently littered in various spots of the > interpreter and replace them by a much clearer protocol, thus reducing the > overall maintenance cost. He has layed out a cleanup path to get rid of the > current quirks in the split between function/method types, thus making the > code easier to explain and lowering the entry barrier for newcomers to the > code base. And, he has motivated that this protocol enables a future > extension towards a specialised (faster) C level call protocol, which third > party extensions would benefit from. > > Given all that, I'm having a hard time finding a "cost" in this. To me, it > reads like a plain net win for all sides. > > Stefan > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
On Sun, Jul 22, 2018 at 12:14 AM, Guido van Rossum wrote: > [...] > The new BDFL may be less demanding though. :=) > I sincerely hope not. regards Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Benchmarks why we need PEP 576/579/580
> > I am comparing calls through tp_call (A) versus optimized call paths > (B). I only need to assume that the speed improvements to (A) between > 2.7 and 3.7 are not bigger than the speed improvements to (B). > It's interesting... But I failed to build sage. It's build step is too different from normal Python package. It tooks very long time to build. And "install from source" document only describe step to `./sage` command work. It doesn't describe step to `improt sage` works. I feel it's not suitable for target application for discussing future optimization. Target application should be easy to test, benchmark and profile for all of core-devs interesting in these PEPs. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
