[Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
Hi folks! I have just ran into buffer protocol incompatibility problem with numpy-1.5.1, which lead me to discover the following ticket (discussed back in December 2010 on this list): http://trac.cython.org/cython_trac/ticket/630 In despair, I was about to try to see if there is anything I can do to fix it myself. To this end I cloned your git repository and set up my Python environment to use the latest bleeding edge version from there. To my surprise I discovered that my code started working and I don't have the buffer interface problem that I was facing before anymore. So my suggestion would be to maybe test it more thoroughly and if it is indeed the case, close the ticket. I tried to subscribe to it or leave a comment, but I need an account which I can't register on my own. Still, I have the following question: which risks am I running by using the bleeding edge version and is there any chance this could inflict cancer on me? Do you have plans to make a new point release which includes this very important improvement anytime soon? Thanks! P.S. I've met Stefan Behnel at FACETS CodeJam the year before last. Back then I didn't realize how much you rock guys and how I will need Cython in the future for my research... -- Sincerely yours, Yury V. Zaytsev ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On Fri, 2011-04-15 at 16:20 +0200, Yury V. Zaytsev wrote: > To my surprise I discovered that my code started working and I don't > have the buffer interface problem that I was facing before anymore. I am under impression that the culprit was this commit: Commit SHA1: dd3da6c283a0750c9c6514991be719ac064e79b4 Commit message: PEP 3118: fix for NULL Py_buffer arg Committer: Lisandro Dalcin 2011-04-08 00:25:45 -- Sincerely yours, Yury V. Zaytsev ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On 15 April 2011 11:20, Yury V. Zaytsev wrote: > Hi folks! > > I have just ran into buffer protocol incompatibility problem with > numpy-1.5.1, which lead me to discover the following ticket (discussed > back in December 2010 on this list): > > http://trac.cython.org/cython_trac/ticket/630 > > In despair, I was about to try to see if there is anything I can do to > fix it myself. To this end I cloned your git repository and set up my > Python environment to use the latest bleeding edge version from there. > > To my surprise I discovered that my code started working and I don't > have the buffer interface problem that I was facing before anymore. > > So my suggestion would be to maybe test it more thoroughly and if it is > indeed the case, close the ticket. I tried to subscribe to it or leave a > comment, but I need an account which I can't register on my own. > I'm opposed to close the ticket. Cython cannot currently parse format strings according to the full spec. It worked for you just because of some quick fixes I've pushed for simple cases. > Still, I have the following question: which risks am I running by using > the bleeding edge version and is there any chance this could inflict > cancer on me? > Well, usually the bleeding edge is more featured and bug-free than latest release. However, right now, generator support was merged and you could have some performance regressions. But I'm using the bleeding edge for every day development work of mpi4py/petsc4py. > > Do you have plans to make a new point release which includes this very > important improvement anytime soon? > Not sure, we still have some (performance) regressions to fix, all of them related to the new generators support. -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On 15 April 2011 11:48, Yury V. Zaytsev wrote: > On Fri, 2011-04-15 at 16:20 +0200, Yury V. Zaytsev wrote: > >> To my surprise I discovered that my code started working and I don't >> have the buffer interface problem that I was facing before anymore. > > I am under impression that the culprit was this commit: > > Commit SHA1: dd3da6c283a0750c9c6514991be719ac064e79b4 > Commit message: PEP 3118: fix for NULL Py_buffer arg > Committer: Lisandro Dalcin 2011-04-08 00:25:45 > This commit should be easy to cherry-pick and backport to latest release. However, there is a potential risk that user code implementing __getbuffer__ in extension classes do not make the check "if info == NULL: return", and then you get a SEGFAULT. AFAIK, all this business with Py_buffer pointer being NULL was added because of some obscure bug in Python==3.0.x, but I would expect people out there to be using Python >= 3.1.x, so not a big deal. -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On 04/15/2011 06:45 PM, Lisandro Dalcin wrote: On 15 April 2011 11:20, Yury V. Zaytsev wrote: Hi folks! I have just ran into buffer protocol incompatibility problem with numpy-1.5.1, which lead me to discover the following ticket (discussed back in December 2010 on this list): http://trac.cython.org/cython_trac/ticket/630 In despair, I was about to try to see if there is anything I can do to fix it myself. To this end I cloned your git repository and set up my Python environment to use the latest bleeding edge version from there. To my surprise I discovered that my code started working and I don't have the buffer interface problem that I was facing before anymore. So my suggestion would be to maybe test it more thoroughly and if it is indeed the case, close the ticket. I tried to subscribe to it or leave a comment, but I need an account which I can't register on my own. I'm opposed to close the ticket. Cython cannot currently parse format strings according to the full spec. It worked for you just because of some quick fixes I've pushed for simple cases. Pauli Virtanen fixed this and there's a pull request here: https://github.com/cython/cython/pull/17 I'll get to it in a couple of days if nobody beats me to it. BTW, did anyone figure out how to get emailed on pull requests? (Yes, I checked off all the "send me email" boxes etc., we were talking about this during the workshop -- it seems that org admins can't subscribe...) DS ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On 04/15/2011 07:57 PM, Dag Sverre Seljebotn wrote: On 04/15/2011 06:45 PM, Lisandro Dalcin wrote: On 15 April 2011 11:20, Yury V. Zaytsev wrote: Hi folks! I have just ran into buffer protocol incompatibility problem with numpy-1.5.1, which lead me to discover the following ticket (discussed back in December 2010 on this list): http://trac.cython.org/cython_trac/ticket/630 In despair, I was about to try to see if there is anything I can do to fix it myself. To this end I cloned your git repository and set up my Python environment to use the latest bleeding edge version from there. To my surprise I discovered that my code started working and I don't have the buffer interface problem that I was facing before anymore. So my suggestion would be to maybe test it more thoroughly and if it is indeed the case, close the ticket. I tried to subscribe to it or leave a comment, but I need an account which I can't register on my own. I'm opposed to close the ticket. Cython cannot currently parse format strings according to the full spec. It worked for you just because of some quick fixes I've pushed for simple cases. Pauli Virtanen fixed this and there's a pull request here: https://github.com/cython/cython/pull/17 I'll get to it in a couple of days if nobody beats me to it. BTW, did anyone figure out how to get emailed on pull requests? (Yes, I checked off all the "send me email" boxes etc., we were talking about this during the workshop -- it seems that org admins can't subscribe...) And BTW, the Cython test suite exposed a bug in NumPy as well -- you can see it in the test case comments. So there may be NumPy builds (and releases?) out there that fail the Cython test suite because of a bug in NumPy. It only affect unpacked structs though; I believe you're good with packed structs (and few use NumPy with unpacked structs). DS ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On Fri, 2011-04-15 at 19:57 +0200, Dag Sverre Seljebotn wrote: > BTW, did anyone figure out how to get emailed on pull requests? (Yes, I > checked off all the "send me email" boxes etc., we were talking about > this during the workshop -- it seems that org admins can't subscribe...) That's strange, I'm getting notifications, when I get pull requests for the projects I have in my organization. I don't remember ticking any extra boxes other than in the Account Settings -> Notification Center though. Maybe you are not actually watching the main repository? I would try to set the watch flag and see if it works... Thanks for you your hard work on Cython by the way! -- Sincerely yours, Yury V. Zaytsev ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Incompatibility with numpy-1.5.1 - fixed in master?!
On Fri, 2011-04-15 at 13:45 -0300, Lisandro Dalcin wrote: > I'm opposed to close the ticket. Cython cannot currently parse format > strings according to the full spec. It worked for you just because of > some quick fixes I've pushed for simple cases. I don't insist on that, especially now that I didn't have the full picture. Maybe if you have access to the track just add a note that most of the problems have been solved in the latest git master and it will be fixed for good before the next release? Thanks, Lisandro! -- Sincerely yours, Yury V. Zaytsev ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] speed.pypy.org
Stefan Behnel, 11.04.2011 15:08: I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython running on speed.pypy.org (that's what I wrote "cythonrun" for). If it works out well, we may have it up in a couple of days. ... or maybe not. It may take a little longer due to lack of time on his side. I would expect that Cython won't be a big winner in this game, given that it will only compile plain untyped Python code. It's also going to fail entirely in some of the benchmarks. But I think it's worth having it up there, simply as a way for us to see where we are performance-wise and to get quick (nightly) feed-back about optimisations we try. The benchmark suite is also a nice set of real-world Python code that will allow us to find compliance issues. Ok, here's what I have so far. I fixed a couple of bugs in Cython and got at least some of the benchmarks running. Note that they are actually simple ones, only a single module. Basically all complex benchmarks fail due to known bugs, such as Cython def functions not accepting attribute assignments (e.g. on wrapping). There's also a problem with code that uses platform specific names conditionally, such as WindowsError when running on Windows. Cython complains about non-builtin names here. I'm considering to turn that into a visible warning instead of an error, so that the name would instead be looked up dynamically to let the code fail at runtime *iff* it reaches the name lookup. Anyway, here are the numbers. I got them with "auto_cpdef" enabled, although that doesn't even seem to make that a big difference. The baseline is a self-compiled Python 2.7.1+ (about a month old). Stefan ### ai ### Min: 0.402407 -> 0.362190: 1.1110x faster Avg: 0.408784 -> 0.366898: 1.1142x faster Significant (t=10.017195, a=0.95) Stddev: 0.00824 -> 0.00442: 1.8668x smaller ### chaos ### (with a bug fixed in the benchmark itself) Min: 0.393362 -> 0.231932: 1.6960x faster Avg: 0.401941 -> 0.234089: 1.7170x faster Significant (t=36.128709, a=0.95) Stddev: 0.01004 -> 0.00267: 3.7538x smaller ### crypto_pyaes ### Min: 2.629560 -> 1.276433: 2.0601x faster Avg: 2.639409 -> 1.277742: 2.0657x faster Significant (t=368.652396, a=0.95) Stddev: 0.00812 -> 0.00153: 5.3215x smaller ### fannkuch ### Min: 1.512630 -> 0.853309: 1.7727x faster Avg: 1.522860 -> 0.860237: 1.7703x faster Significant (t=118.573908, a=0.95) Stddev: 0.00880 -> 0.00887: 1.0073x larger ### float ### Min: 0.452620 -> 0.343341: 1.3183x faster Avg: 0.475137 -> 0.349356: 1.3600x faster Significant (t=9.575876, a=0.95) Stddev: 0.02838 -> 0.00757: 3.7489x smaller ### go ### Min: 0.758998 -> 0.491929: 1.5429x faster Avg: 0.764110 -> 0.496518: 1.5389x faster Significant (t=90.848407, a=0.95) Stddev: 0.00400 -> 0.00523: 1.3096x larger ### nbody_modified ### Min: 0.399168 -> 0.197931: 2.0167x faster Avg: 0.401379 -> 0.203112: 1.9762x faster Significant (t=42.377829, a=0.95) Stddev: 0.00293 -> 0.01004: 3.4337x larger ### raytracesimple ### (module renamed from "raytrace-simple") Min: 2.016425 -> 1.182970: 1.7045x faster Avg: 2.030030 -> 1.192164: 1.7028x faster Significant (t=78.219481, a=0.95) Stddev: 0.02184 -> 0.00983: 2.2211x smaller ### richards ### Min: 0.286723 -> 0.162430: 1.7652x faster Avg: 0.289933 -> 0.165193: 1.7551x faster Significant (t=52.898468, a=0.95) Stddev: 0.00392 -> 0.00352: 1.1127x smaller ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] speed.pypy.org
On Fri, Apr 15, 2011 at 1:20 PM, Stefan Behnel wrote: > Stefan Behnel, 11.04.2011 15:08: >> >> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython >> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it >> works out well, we may have it up in a couple of days. > > ... or maybe not. It may take a little longer due to lack of time on his > side. > > >> I would expect that Cython won't be a big winner in this game, given that >> it will only compile plain untyped Python code. It's also going to fail >> entirely in some of the benchmarks. But I think it's worth having it up >> there, simply as a way for us to see where we are performance-wise and to >> get quick (nightly) feed-back about optimisations we try. The benchmark >> suite is also a nice set of real-world Python code that will allow us to >> find compliance issues. > > Ok, here's what I have so far. I fixed a couple of bugs in Cython and got at > least some of the benchmarks running. Note that they are actually simple > ones, only a single module. Basically all complex benchmarks fail due to > known bugs, such as Cython def functions not accepting attribute assignments > (e.g. on wrapping). There's also a problem with code that uses platform > specific names conditionally, such as WindowsError when running on Windows. > Cython complains about non-builtin names here. I'm considering to turn that > into a visible warning instead of an error, so that the name would instead > be looked up dynamically to let the code fail at runtime *iff* it reaches > the name lookup. Given the usefulness of the error, and the (relative) lack of issues with it so far, I'd rather not turn it into only a warning by default (though an option might be nice). Another option would be to whitelist the presumably small, finite set of names that are platform-dependent. > Anyway, here are the numbers. I got them with "auto_cpdef" enabled, although > that doesn't even seem to make that a big difference. The baseline is a > self-compiled Python 2.7.1+ (about a month old). Cool. So basically everything is faster, usually somewhere between a 50-100% improvement. There's lots of room for improvement, though a JIT has a significant advantage that we don't get for untyped code. - Robert > ### ai ### > Min: 0.402407 -> 0.362190: 1.1110x faster > Avg: 0.408784 -> 0.366898: 1.1142x faster > Significant (t=10.017195, a=0.95) > Stddev: 0.00824 -> 0.00442: 1.8668x smaller > > > ### chaos ### (with a bug fixed in the benchmark itself) > Min: 0.393362 -> 0.231932: 1.6960x faster > Avg: 0.401941 -> 0.234089: 1.7170x faster > Significant (t=36.128709, a=0.95) > Stddev: 0.01004 -> 0.00267: 3.7538x smaller > > > ### crypto_pyaes ### > Min: 2.629560 -> 1.276433: 2.0601x faster > Avg: 2.639409 -> 1.277742: 2.0657x faster > Significant (t=368.652396, a=0.95) > Stddev: 0.00812 -> 0.00153: 5.3215x smaller > > > ### fannkuch ### > Min: 1.512630 -> 0.853309: 1.7727x faster > Avg: 1.522860 -> 0.860237: 1.7703x faster > Significant (t=118.573908, a=0.95) > Stddev: 0.00880 -> 0.00887: 1.0073x larger > > > ### float ### > Min: 0.452620 -> 0.343341: 1.3183x faster > Avg: 0.475137 -> 0.349356: 1.3600x faster > Significant (t=9.575876, a=0.95) > Stddev: 0.02838 -> 0.00757: 3.7489x smaller > > > ### go ### > Min: 0.758998 -> 0.491929: 1.5429x faster > Avg: 0.764110 -> 0.496518: 1.5389x faster > Significant (t=90.848407, a=0.95) > Stddev: 0.00400 -> 0.00523: 1.3096x larger > > > ### nbody_modified ### > Min: 0.399168 -> 0.197931: 2.0167x faster > Avg: 0.401379 -> 0.203112: 1.9762x faster > Significant (t=42.377829, a=0.95) > Stddev: 0.00293 -> 0.01004: 3.4337x larger > > > ### raytracesimple ### (module renamed from "raytrace-simple") > Min: 2.016425 -> 1.182970: 1.7045x faster > Avg: 2.030030 -> 1.192164: 1.7028x faster > Significant (t=78.219481, a=0.95) > Stddev: 0.02184 -> 0.00983: 2.2211x smaller > > > ### richards ### > Min: 0.286723 -> 0.162430: 1.7652x faster > Avg: 0.289933 -> 0.165193: 1.7551x faster > Significant (t=52.898468, a=0.95) > Stddev: 0.00392 -> 0.00352: 1.1127x smaller > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
