[Cython] "long long" versus "PY_LONG_LONG"
At some point Cython lost the ability of using PY_LONG_LONG instead of "long long" in generated C code. The big offenders are the following two files: Cython/Utility/Overflow.c Cython/Utility/TypeConversion.c Do we want to continue using PY_LONG_LONG? Otherwise, what about Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite easy right now, let me know and I'll contribute the patch for the upcoming 0.22.1 release (if that ever happens). -- Lisandro Dalcin Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
On 18 February 2012 at 19:11, Stefan Behnel wrote: > Stefan Behnel, 18.02.2012 09:54: >> Stefan Behnel, 15.02.2012 12:32: >>> The current state of the discussion seems to be that PyPy provides ways to >>> talk to C code, but nothing as complete as CPython's C-API in the sense >>> that it allows efficient two-way communication between C code and Python >>> objects. Thus, we need to either improve this or look for alternatives. >>> >>> In order to get us more focussed on what can be done and what the >>> implications are, so that we may eventually be able to decide what should >>> be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement >>> Proposal). >>> >>> http://wiki.cython.org/enhancements/pypy >> >> The discussion so far makes me rather certain that the most promising >> short-term solution is to make Cython generate C code that PyPy's cpyext >> can handle. This should get us a rather broad set of running code somewhat >> quickly, while requiring the least design-from-scratch type of work in a >> direction that does not yet allow us to see if it will really make existing >> code work or not. > > Update: > > Amaury Forgeot d'Arc fiddled out a couple of fixes and hacks to make it run > (although with some clear bugs in the exception handling code). There is a > Jenkins job now to (try to) run the test suite of my own branch in the > latest PyPy nightly build: > > https://sage.math.washington.edu:8091/hudson/view/dev-scoder/job/cython-scoder-pypy-nightly/ > > It currently crashes rather badly at some point, but at least it looks like > it's actually getting somewhere. > One thing that Cython developers really need is PyPy defining a macro such as PYPY_VERSION_HEX in such a way us we can properly use conditional compilation. For example, a few days ago I was pushing PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the interpreter broke at runtime. This issue will be eventually fixed, I hope. Unce that happens, how can we know it is save to use the call for that pypy version and upwards? I mean, Cython should still support previous PyPy releases... -- Lisandro Dalcin Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "long long" versus "PY_LONG_LONG"
Lisandro Dalcin schrieb am 29.03.2015 um 12:17: > At some point Cython lost the ability of using PY_LONG_LONG instead of > "long long" in generated C code. The big offenders are the following > two files: > > Cython/Utility/Overflow.c > Cython/Utility/TypeConversion.c > > Do we want to continue using PY_LONG_LONG? Otherwise, what about > Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite > easy right now, let me know and I'll contribute the patch for the > upcoming 0.22.1 release (if that ever happens). If PyLL helps in any way, then, by all means, we should use it. And yes, there should be a 0.22.1 release. I selected changes for it already, and saw that you did, too. We should give the current branch another review to make sure we didn't forget anything else, and then release it. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "long long" versus "PY_LONG_LONG"
On 29 March 2015 at 13:33, Stefan Behnel wrote: > Lisandro Dalcin schrieb am 29.03.2015 um 12:17: >> At some point Cython lost the ability of using PY_LONG_LONG instead of >> "long long" in generated C code. The big offenders are the following >> two files: >> >> Cython/Utility/Overflow.c >> Cython/Utility/TypeConversion.c >> >> Do we want to continue using PY_LONG_LONG? Otherwise, what about >> Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite >> easy right now, let me know and I'll contribute the patch for the >> upcoming 0.22.1 release (if that ever happens). > > If PyLL helps in any way, then, by all means, we should use it. > > And yes, there should be a 0.22.1 release. I selected changes for it > already, and saw that you did, too. We should give the current branch > another review to make sure we didn't forget anything else, and then > release it. > OK, please give me a day to push the PyLL patch. Thank you! -- Lisandro Dalcin Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
Lisandro Dalcin schrieb am 29.03.2015 um 12:23: > One thing that Cython developers really need is PyPy defining a macro > such as PYPY_VERSION_HEX in such a way us we can properly use > conditional compilation. For example, a few days ago I was pushing > PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the > interpreter broke at runtime. This issue will be eventually fixed, I > hope. Unce that happens, how can we know it is save to use the call > for that pypy version and upwards? I mean, Cython should still support > previous PyPy releases... Yes, it's unfortunate that cpyext isn't versioned. But PY_VERSION_HEX should still change sometimes over PyPy releases, so that might at least help a bit. In general, however, you shouldn't expect any CPython internals to work in PyPy. Avoiding multiple C-API calls when a generic one exists is really the best of all strategies. For example, why read just the sign, when you can read the whole value? Everything else just risks aggregating slowness and running into buggy corners. That being said, if you find anything that can be optimised, I'm happy to learn about your changes. I think the general problem is that the PyPy developers loudly push for cffi adoption, and improving anything in cpyext appears to undermine that goal for them. So the current state of cpyext is a result of political decisions, general dislikes and the usual OSS project lack of workpower. Given all that, it actually works quite well. :) Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
On 29 March 2015 at 14:16, Stefan Behnel wrote:
> Lisandro Dalcin schrieb am 29.03.2015 um 12:23:
>> One thing that Cython developers really need is PyPy defining a macro
>> such as PYPY_VERSION_HEX in such a way us we can properly use
>> conditional compilation. For example, a few days ago I was pushing
>> PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the
>> interpreter broke at runtime. This issue will be eventually fixed, I
>> hope. Unce that happens, how can we know it is save to use the call
>> for that pypy version and upwards? I mean, Cython should still support
>> previous PyPy releases...
>
> Yes, it's unfortunate that cpyext isn't versioned. But PY_VERSION_HEX
> should still change sometimes over PyPy releases, so that might at least
> help a bit.
>
Well, you know that's not enough ;-)
> In general, however, you shouldn't expect any CPython internals to work in
> PyPy. Avoiding multiple C-API calls when a generic one exists is really the
> best of all strategies. For example, why read just the sign, when you can
> read the whole value? Everything else just risks aggregating slowness and
> running into buggy corners.
>
Well, I agree, however I do require _PyLong_Sign to work if I want to
report correct overflow error messages when converting PyLong values
to a C integral type. In CPython, we are using Py_SIZE for that, not
only for speed, but also to figure out the right overflow kind when
converting to unsigned integers.
> That being said, if you find anything that can be optimised, I'm happy to
> learn about your changes.
>
Again, this is not about optimization, but proper error reporting.
diff --git a/Cython/Utility/TypeConversion.c b/Cython/Utility/TypeConversion.c
index 4cd8229..fbac649 100644
--- a/Cython/Utility/TypeConversion.c
+++ b/Cython/Utility/TypeConversion.c
@@ -560,6 +560,10 @@ static CYTHON_INLINE {{TYPE}}
{{FROM_PY_FUNCTION}}(PyObject *x) {
if (unlikely(Py_SIZE(x) < 0)) {
goto raise_neg_overflow;
}
+#elif CYTHON_COMPILING_IN_PYPY
+if (unlikely(_PyLong_Sign(x) < 0)) {
+goto raise_neg_overflow;
+}
#endif
if (sizeof({{TYPE}}) <= sizeof(unsigned long)) {
__PYX_VERIFY_RETURN_INT({{TYPE}}, unsigned long,
PyLong_AsUnsignedLong(x))
--
Lisandro Dalcin
Research Scientist
Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
Numerical Porous Media Center (NumPor)
King Abdullah University of Science and Technology (KAUST)
http://numpor.kaust.edu.sa/
4700 King Abdullah University of Science and Technology
al-Khawarizmi Bldg (Bldg 1), Office # 4332
Thuwal 23955-6900, Kingdom of Saudi Arabia
http://www.kaust.edu.sa
Office Phone: +966 12 808-0459
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
Lisandro Dalcin schrieb am 29.03.2015 um 14:52:
>> Lisandro Dalcin schrieb am 29.03.2015 um 12:23:
>>> One thing that Cython developers really need is PyPy defining a macro
>>> such as PYPY_VERSION_HEX in such a way us we can properly use
>>> conditional compilation. For example, a few days ago I was pushing
>>> PyPy fixes to Cython. I tried to use _PyLong_Sign in my patch, but the
>>> interpreter broke at runtime.
>
> however I do require _PyLong_Sign to work if I want to
> report correct overflow error messages when converting PyLong values
> to a C integral type. In CPython, we are using Py_SIZE for that, not
> only for speed, but also to figure out the right overflow kind when
> converting to unsigned integers.
>
> diff --git a/Cython/Utility/TypeConversion.c b/Cython/Utility/TypeConversion.c
> index 4cd8229..fbac649 100644
> --- a/Cython/Utility/TypeConversion.c
> +++ b/Cython/Utility/TypeConversion.c
> @@ -560,6 +560,10 @@ static CYTHON_INLINE {{TYPE}}
> {{FROM_PY_FUNCTION}}(PyObject *x) {
> if (unlikely(Py_SIZE(x) < 0)) {
> goto raise_neg_overflow;
> }
> +#elif CYTHON_COMPILING_IN_PYPY
> +if (unlikely(_PyLong_Sign(x) < 0)) {
> +goto raise_neg_overflow;
> +}
> #endif
> if (sizeof({{TYPE}}) <= sizeof(unsigned long)) {
> __PYX_VERIFY_RETURN_INT({{TYPE}}, unsigned long,
> PyLong_AsUnsignedLong(x))
Why not call PyObject_RichCompareBool() to make cpyext itself compare the
value to 0?
Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] "long long" versus "PY_LONG_LONG"
On 29 March 2015 at 14:19, Lisandro Dalcin wrote: > On 29 March 2015 at 13:33, Stefan Behnel wrote: >> Lisandro Dalcin schrieb am 29.03.2015 um 12:17: >>> At some point Cython lost the ability of using PY_LONG_LONG instead of >>> "long long" in generated C code. The big offenders are the following >>> two files: >>> >>> Cython/Utility/Overflow.c >>> Cython/Utility/TypeConversion.c >>> >>> Do we want to continue using PY_LONG_LONG? Otherwise, what about >>> Microsoft compilers? Going back to PY_LONG_LONG everywhere is quite >>> easy right now, let me know and I'll contribute the patch for the >>> upcoming 0.22.1 release (if that ever happens). >> >> If PyLL helps in any way, then, by all means, we should use it. >> >> And yes, there should be a 0.22.1 release. I selected changes for it >> already, and saw that you did, too. We should give the current branch >> another review to make sure we didn't forget anything else, and then >> release it. >> > > OK, please give me a day to push the PyLL patch. Thank you! > Done: [0.22.x] https://github.com/cython/cython/commit/60f0e2f99c3e587bc6d267608ad13e0f384b527c [master] https://github.com/cython/cython/commit/2e245fc1e3881ed66ce9e8a5e07c44071b1a499e -- Lisandro Dalcin Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
On 29 March 2015 at 16:10, Stefan Behnel wrote: > Why not call PyObject_RichCompareBool() to make cpyext itself compare the > value to 0? That should definitely work. Let me try to put a patch together. -- Lisandro Dalcin Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
On 29 March 2015 at 20:30, Lisandro Dalcin wrote: > On 29 March 2015 at 16:10, Stefan Behnel wrote: >> Why not call PyObject_RichCompareBool() to make cpyext itself compare the >> value to 0? > > That should definitely work. Let me try to put a patch together. > Stefan, how to you feel about abusing of Py_False in the call to RichCompareBool ? -- Lisandro Dalcin Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
Lisandro Dalcin schrieb am 29.03.2015 um 19:40: > On 29 March 2015 at 20:30, Lisandro Dalcin wrote: >> On 29 March 2015 at 16:10, Stefan Behnel wrote: >>> Why not call PyObject_RichCompareBool() to make cpyext itself compare the >>> value to 0? >> >> That should definitely work. Let me try to put a patch together. > > Stefan, how to you feel about abusing of Py_False in the call to > RichCompareBool ? Wow - evil idea! :) But then, it's only for cpyext - as long as it works there and comes with a comment, it's certainly as simple as it gets. bool is clearly defined as an int subtype in Python. Just remember to check the error code. Py_SIZE() can't fail in CPython, but comparisons can, sadly. Stefan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: Bringing Cython and PyPy closer together
On 29 March 2015 at 21:02, Stefan Behnel wrote: > Lisandro Dalcin schrieb am 29.03.2015 um 19:40: >> On 29 March 2015 at 20:30, Lisandro Dalcin wrote: >>> On 29 March 2015 at 16:10, Stefan Behnel wrote: Why not call PyObject_RichCompareBool() to make cpyext itself compare the value to 0? >>> >>> That should definitely work. Let me try to put a patch together. >> >> Stefan, how to you feel about abusing of Py_False in the call to >> RichCompareBool ? > > Wow - evil idea! :) > > But then, it's only for cpyext - as long as it works there and comes with a > comment, it's certainly as simple as it gets. bool is clearly defined as an > int subtype in Python. > > Just remember to check the error code. Py_SIZE() can't fail in CPython, but > comparisons can, sadly. > Please review: https://github.com/cython/cython/pull/377 Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
