Re: [Cython] (bug?) "cdef public" and Python attribute access
Hi, note that the cython-users mailing list would be a more appropriate place to discuss this. Please reply over there. David Warde-Farley, 20.03.2012 06:44: > All of this refers to Cython 0.15.1: > > I have a "cdef public void" function which takes several ndarrays, that > segfaults when I call it from other C code. Inside I use the ".size" > attribute, triggering a PyObject_GetAttr, which segfaults. Running in > gdb and looking at the generated C code, I see it is because > __pyx_n_s__size is NULL, presumably owing to __Pyx_InitStrings() never > being called. I take it you are embedding Cython code in a C program? Note that the module must be imported in order to work properly. See here: http://wiki.cython.org/EmbeddingCython > Am I to presume this is intended behaviour? Yes. > i.e. code that generates > Python C API (or more specifically GetAttr) calls simply doesn't > work/isn't intended to work from "public" functions called from C code? It does work, but it requires two things: a properly imported module and (unless declared "nogil") the GIL. > I don't see mention of it in the documentation, so it's kind of a nasty > surprise :) Yes, this can use some better documentation. > (The "api" keyword mentions the import_modulename() > business but no signature for that is generated in my header file.) It has been discussed before that the concepts "api" and "public" are not very clearly separated and should eventually be merged into one. Hasn't been done yet. Basically, the concepts are somewhat more focussed towards being used from inside of CPython than towards embedding CPython in C code. Both work, but embedding requires a little more manual setup. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] some pull requests for 0.16?
On 18 March 2012 11:58, Stefan Behnel wrote: > Hi, > > I put up two new pull requests on github: > > Implementation of PEP 380 (yield from): > https://github.com/cython/cython/pull/96 > > Rewrite of dict iteration: > https://github.com/cython/cython/pull/95 > > Given that the release of 0.16 has currently slowed down a bit, and given > that these are really nice features, could someone (and especially Mark, as > the responsible release manager) take a look at them and give an opinion > regarding their suitability for 0.16? Sorry about the slowdown, my BT internet connection at home got closed off, so working on a release is rather hard, hopefully that will be fixed within the next few days. Anyway, during the sprints Dag worked on the numpy attributes accesses to generate non-deprecated numpy code, I'm not sure how far along that is and if we can get that in as well, or have it wait until 0.16.1. I worked on enhancing fused types runtime dispatching (e.g. select the right specialization at runtime from the argument types), especially on matching buffers. It's also a bit of a rewrite as the complexity grew quite a bit and ndarray and fused types were broken in subtle ways (I would do the pull request now if I hadn't left the code at home :). Finally, there is a serious bug in the reversed(range()) optimization that Mike Graham discovered when working on the range() optimization with a runtime step, which is basically wrong for any step that is not 1 or -1 (it simply swaps the bounds with an offset by 1, but it actually has to figure out what the actual last value was in the range and use that as the start instead). I personally consider this a blocker, so I'll try contacting Mike and see if he is still (interested in) working on that. As for the two pull requests, both are quite large, but the dict iteration rewrite is more like an enhancement whereas the 'yield from' is really a new (somewhat big) feature. I would personally prefer to merge only enhancements and bug fixes, but not new features. If you feel confident enough that the dict iteration pull request is ready to be merged, then please do so. We can do an 0.16.1 release with the 'yield from', the 'lnotab' line number mapping from code objects to Cython source and probably the boundscheck and wraparound optimizations I started working on (and other things if and when they come up). We can release 0.16.1 somewhat shortly after releasing 0.16. I think a shorter release cycle with less changes would be beneficial and give us swifter feedback about bugs, missing features, wanted features, etc. So I propose that when we merged dict iteration, fused types enhancements and the range bug fix, we do another beta release. We (I) also need to look into the test failures on Gentoo. > Personally, I think they are ready, but they'd certainly require another > beta release for testing. > > You may notice that I also added the PyPy related preprocessor defines > "CYTHON_COMPILING_IN_CPYTHON" and "CYTHON_COMPILING_IN_PYPY" to the current > master and used them in a couple of places to reduce Cython's dependency on > CPython details. That doesn't mean we already support PyPy (there are still > open issues, mostly on their side). I only pushed the changes that were > non-intrusive and helped me in preparing the two branches above in a clean > way. I have some more changes in my PyPy branch: > > https://github.com/scoder/cython/tree/pypy > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] some pull requests for 0.16?
mark florisson, 20.03.2012 17:40: > On 18 March 2012 11:58, Stefan Behnel wrote: >> I put up two new pull requests on github: >> >> Implementation of PEP 380 (yield from): >> https://github.com/cython/cython/pull/96 >> >> Rewrite of dict iteration: >> https://github.com/cython/cython/pull/95 >> >> Given that the release of 0.16 has currently slowed down a bit, and given >> that these are really nice features, could someone (and especially Mark, as >> the responsible release manager) take a look at them and give an opinion >> regarding their suitability for 0.16? > > Sorry about the slowdown, my BT internet connection at home got closed > off, so working on a release is rather hard, hopefully that will be > fixed within the next few days. Anyway, during the sprints Dag worked > on the numpy attributes accesses to generate non-deprecated numpy > code, I'm not sure how far along that is and if we can get that in as > well, or have it wait until 0.16.1. I wouldn't mind if my changes waited for 0.16.1 as well, as long as that's not too long from now. There will certainly be things to fix after the release anyway. > I worked on enhancing fused types runtime dispatching (e.g. select the > right specialization at runtime from the argument types), especially > on matching buffers. It's also a bit of a rewrite as the complexity > grew quite a bit and ndarray and fused types were broken in subtle > ways (I would do the pull request now if I hadn't left the code at > home :). I took a tiny look at that when I wrote a test case and was surprised to see that it imports "sys" inside of the generated function. Looks like there's a bit left to optimise. I faintly remember a blog post (or more than one) by PJ Eby when he wrote a "generic functions" module. It uses type dispatching as well, and he wrote about the ways he found to tune that. Might be an interesting read. > Finally, there is a serious bug in the reversed(range()) optimization > that Mike Graham discovered when working on the range() optimization > with a runtime step, which is basically wrong for any step that is not > 1 or -1 (it simply swaps the bounds with an offset by 1, but it > actually has to figure out what the actual last value was in the range > and use that as the start instead). I personally consider this a > blocker, so I'll try contacting Mike and see if he is still > (interested in) working on that. Interesting. Yes, I can well imagine that being wrong. These things are exceedingly tricky. However, it's easy to disable the optimisation for a specific case - likely easier than fixing it. Also, I don't consider this a blocker. It's not like it works in the current release, so it's not a regression compared to that. > As for the two pull requests, both are quite large, but the dict > iteration rewrite is more like an enhancement whereas the 'yield from' > is really a new (somewhat big) feature. I would personally prefer to > merge only enhancements and bug fixes, but not new features. Funny. I would have proposed the opposite, because I consider the "yield from" implementation (as a new language feature) safer than the dict iteration (which impacts existing code). As I said, I wouldn't mind postponing both to 0.16.1, but I think it would be better to get them out and have people use them, so that we can also fix the bugs earlier. > If you > feel confident enough that the dict iteration pull request is ready to > be merged, then please do so. I'll take another look over it and decide. Thanks. > We can do an 0.16.1 release with the > 'yield from', the 'lnotab' line number mapping from code objects to > Cython source Note that that's not useful yet - it requires changes to the code that *uses* the code objects as well, to a) actually reuse the code objects (iff C line numbers are disabled in traces) and b) write the proper relative line into the byte code offset field. I just put it up to have a place where we can discuss further changes. > and probably the boundscheck and wraparound > optimizations I started working on (and other things if and when they > come up). We can release 0.16.1 somewhat shortly after releasing 0.16. > I think a shorter release cycle with less changes would be beneficial > and give us swifter feedback about bugs, missing features, wanted > features, etc. +1 > So I propose that when we merged dict iteration, fused types > enhancements and the range bug fix, we do another beta release. We (I) > also need to look into the test failures on Gentoo. Whatever you choose. In case the PyPy developers manage to get one change in that I'd like to see on their side (two new C-API functions for exception handling), I'll also push some more Cython changes based on it that I have lying around. That would hugely improve the compatibility for both. In general, I started using the "CYTHON_COMPILING_IN_..." #defines wherever they made sense, and specifically where we rely on CPython internals and also in some places
Re: [Cython] OS X 10.7 Lion: GCC __builtin_expect unrecognized inside OpenMP blocks
On Sun, Mar 11, 2012 at 9:09 AM, mark florisson wrote: > On 11 March 2012 09:26, Lisandro Dalcin wrote: >> On 11 March 2012 09:46, mark florisson wrote: >>> On 10 March 2012 14:00, Stefan Behnel wrote: Lisandro Dalcin, 10.03.2012 10:51: > On 10 March 2012 03:41, mark florisson wrote: >> On 9 March 2012 14:22, Lisandro Dalcin wrote: >>> I'm basically experiencing the issues here: >>> http://www.cocoabuilder.com/archive/xcode/310299-error-calling-builtin-expect-inside-omp-parallel-for.html >>> >>> Can you imagine any way to workaround it? >> >> What a lovely C compiler bug... Did you file a bug report with gcc or >> xcode? What version of gcc does Lion ship with? >> > > I'm using a Mac just by accident, no idea (nor motivation) to report > bugs to Apple. > > $ gcc --version > i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. > build 5658) (LLVM build 2336.9.00) > > Could that be the the bug is actually in the LLVM backend? > >> I suppose the macro would have to be disabled for OS X Lion inside >> parallel sections (there seems to be __APPLE__ and __OSX__, I don't >> know about detecting the OS X version), that's easy to do (undef and >> redefine to no-ops before parallel section and redefine it again after >> the section). I'll try fixing it during the sprints next week. > > Perhaps switching to use a "omp_likely"/"omp_unlikely" macros inside > OpenMP blocks would be nicer than defining/undefining? Could that be coded into the macro or would it require to change the generated code? But at least it sounds like it would not impact code in functions that are being called from within the OpenMP blocks, would it? Just the code straight inside the block. A work-around could still have a substantial impact if it requires changes to the generated code. >>> >>> Yeah, that's why I suggested the undef/re-def approach around OpenMP >>> blocks. It's some code bloat, but only for the C preprocessor, so it >>> should be fine. >>> >> >> I still feel bad about this. What about just disabling branch >> prediction if OpenMP is ever used? Or perhaps just protect the >> definitions of likely/unlikely with some guard, such as users can >> disable them using a -D definition? >> > > Why? That would disable all compiler optimizations in all code that > could be gotten through the hints, due to a stupid C compiler bug when > using cython.parallel. It would be better to do an easy fix, a nasty > workaround for a nasty bug, but it shouldn't slow down other code. The > only problem with it is a couple of extra bytes in your .c file, and > the fact that it makes us feel less fuzzy inside because we realize > our world isn't what we want it to be. +1 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
