Re: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
Romain Guillebert, 07.04.2011 17:01: I proposed the Summer of Code project regarding the Python backend for Cython. As I said in my proposal this would translate Cython code to Python + FFI code (I don't know yet if it will use ctypes or something specific to PyPy). PyPy's ctypes is now really fast and this will allow people to port their Cython code to PyPy. For the moment I've been mostly in touch with the PyPy people and they seem happy with my proposal. Of course I'm available for questions. Hi Romain, it's usually required for GSoC students to provide a patch for the project they will be participating in. It appears that you have provided patches for PyPy that were well accepted, but since you'd be working mostly on Cython, it would be helpful for us to get an idea about how well you know the Cython source code by now. So, here's a little exam. If you were to implement support for the globals() builtin in Cython, what would you consider the necessary parts of the implementation? How would you approach this task? Feel free to ask for any information or pointers you need to solve this. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 16 April 2011 18:42, Dag Sverre Seljebotn wrote: > (Moving discussion from http://markflorisson.wordpress.com/, where Mark > said:) Ok, sure, it was just an issue I was wondering about at that moment, but it's a tricky issue, so thanks. > """ > Started a new branch https://github.com/markflorisson88/cython/tree/openmp . > > Now the question is whether sharing attributes should be propagated > outwards. e.g. if you do > > for i in prange(m): > for j in prange(n): > sum += i * j > > then ‘sum’ is a reduction for the inner parallel loop, but not for the outer > one. So the user would currently have to rewrite this to > > for i in prange(m): > for j in prange(n): > sum += i * j > sum += 0 > > which seems a bit silly . Of course, we could just disable nested > parallelism, or tell the users to use a prange and a ‘for from’ in such > cases. > """ > > Dag: Interesting. The first one is definitely the behaviour we want, as long > as it doesn't cause unintended consequences. > > I don't really think it will -- the important thing is that that the order > of loop iteration evaluation must be unimportant. And that is still true > (for the outer loop, as well as for the inner) in your first example. > > Question: When you have nested pranges, what will happen is that two nested > OpenMP parallel blocks are used, right? And do you know if there is complete > freedom/"reentrancy" in that variables that are thread-private in an outer > parallel block and be shared in an inner one, and vice versa? An implementation may or may not support it, and if it is supported the behaviour can be configured through omp_set_nested(). So we should consider the case where it is supported and enabled. If you have a lastprivate or reduction, and after the loop these are (reduced and) assigned to the original variable. So if that happens inside a parallel construct which does not declare the variable private to the construct, you actually have a race. So e.g. the nested prange currently races in the outer parallel range. > If so I'd think that this algorithm should work and feel natural: > > - In each prange, for the purposes of variable private/shared/reduction > inference, consider all internal "prange" just as if they had been "range"; > no special treatment. > > - Recurse to children pranges. Right, that is most natural. Algorithmically, reductions and lastprivates (as those can have races if placed in inner parallel constructs) propagate outwards towards the outermost parallel block, or up to the first parallel with block, or up to the first construct that already determined the sharing attribute. e.g. with parallel: with parallel: for i in prange(n): for j in prange(n): sum += i * j # sum is well-defined here # sum is undefined here Here 'sum' is a reduction for the two innermost loops. 'sum' is not private for the inner parallel with block, as a prange in a parallel with block is a worksharing loop that binds to that parallel with block. However, the outermost parallel with block declares sum (and i and j) private, so after that block all those variables become undefined. However, in the outermost parallel with block, sum will have to be initialized to 0 before anything else, or be declared firstprivate, otherwise 'sum' is undefined to begin with. Do you think declaring it firstprivate would be the way to go, or should we make it private and issue a warning or perhaps even an error? > DS > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Recent bugs in generators
2011/4/18 Stefan Behnel : > Vitja Makarov, 18.04.2011 06:38: >> >> 2011/4/18 Stefan Behnel: >>> >>> Vitja Makarov, 17.04.2011 17:57: 3. check_yield_in_exception() >>> >>> I added this because I found a failing pyregr test that uses it (testing >>> the >>> @contextmanager decorator). >>> >>> Cython calls __Pyx_ExceptionReset when except block is done, so when yield is there no exception reset is called. I'm not sure how to fix this. >>> >>> I'm not completely sure either. >>> >>> import sys def foo(): """ >>> list(foo()) [, None] """ try: raise ValueError except ValueError: yield sys.exc_info()[0] yield sys.exc_info()[0] # exc_info is lost here >>> >>> I think (!), the difference here is that CPython actually keeps the >>> exception in the generator frame. We don't have a frame, so we have to >>> emulate it using the closure class. I guess we'll have to store away the >>> exception into the closure when we yield while an exception is being >>> handled, and restore it afterwards. Note: this is not the exception that >>> is >>> freshly *being* raised (the "_cur*" fields in the thread state), it's the >>> exception that *was* raised and is now being handled, i.e. the thread >>> state >>> fields without the "_cur", that are reflected by sys.exc_info(). >> >> Interesting difference between py2 and py3: >> >> def foo(): >> try: >> raise ValueError >> except ValueError: >> yield >> raise >> list(foo()) >> >> File "xxx.py", line 7, in >> list(foo()) >> File "xxx.py", line 6, in foo >> raise >> TypeError: exceptions must be old-style classes or derived from >> BaseException, not NoneType >> >> It seems that exception info is completely lost (tried 2.6, 2.7) and >> seems to be fixed in python3. > > Not surprising. The implementation is completely different in Py2 and Py3, > both in CPython and in Cython. It's actually much simpler in Cython under > Py3, due to better semantics and C-API support. That also implies that > there's much less Cython can do wrong in that environment. ;-) > > >> Btw exception info temps are already saved and restored between yields. > > Right, but the exc_info itself is not reset and recovered around the yield. > As I said above, generators have their own lifetime frame in CPython, and > exceptions don't leak from that. So, whenever it's the generator (or code > called by it) that raises an exception, that must be kept local to the > generator. > There is one more interesting thing: def test_yield_inside_genexp(): """ >>> o = test_yield_inside_genexp() >>> list(o) [0, None, 1, None, 2, None] """ return ((yield i) for i in range(3)) -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 18 April 2011 13:06, mark florisson wrote: > On 16 April 2011 18:42, Dag Sverre Seljebotn > wrote: >> (Moving discussion from http://markflorisson.wordpress.com/, where Mark >> said:) > > Ok, sure, it was just an issue I was wondering about at that moment, > but it's a tricky issue, so thanks. > >> """ >> Started a new branch https://github.com/markflorisson88/cython/tree/openmp . >> >> Now the question is whether sharing attributes should be propagated >> outwards. e.g. if you do >> >> for i in prange(m): >> for j in prange(n): >> sum += i * j >> >> then ‘sum’ is a reduction for the inner parallel loop, but not for the outer >> one. So the user would currently have to rewrite this to >> >> for i in prange(m): >> for j in prange(n): >> sum += i * j >> sum += 0 >> >> which seems a bit silly . Of course, we could just disable nested >> parallelism, or tell the users to use a prange and a ‘for from’ in such >> cases. >> """ >> >> Dag: Interesting. The first one is definitely the behaviour we want, as long >> as it doesn't cause unintended consequences. >> >> I don't really think it will -- the important thing is that that the order >> of loop iteration evaluation must be unimportant. And that is still true >> (for the outer loop, as well as for the inner) in your first example. >> >> Question: When you have nested pranges, what will happen is that two nested >> OpenMP parallel blocks are used, right? And do you know if there is complete >> freedom/"reentrancy" in that variables that are thread-private in an outer >> parallel block and be shared in an inner one, and vice versa? > > An implementation may or may not support it, and if it is supported > the behaviour can be configured through omp_set_nested(). So we should > consider the case where it is supported and enabled. > > If you have a lastprivate or reduction, and after the loop these are > (reduced and) assigned to the original variable. So if that happens > inside a parallel construct which does not declare the variable > private to the construct, you actually have a race. So e.g. the nested > prange currently races in the outer parallel range. > >> If so I'd think that this algorithm should work and feel natural: >> >> - In each prange, for the purposes of variable private/shared/reduction >> inference, consider all internal "prange" just as if they had been "range"; >> no special treatment. >> >> - Recurse to children pranges. > > Right, that is most natural. Algorithmically, reductions and > lastprivates (as those can have races if placed in inner parallel > constructs) propagate outwards towards the outermost parallel block, > or up to the first parallel with block, or up to the first construct > that already determined the sharing attribute. > > e.g. > > with parallel: > with parallel: > for i in prange(n): > for j in prange(n): > sum += i * j > # sum is well-defined here > # sum is undefined here > > Here 'sum' is a reduction for the two innermost loops. 'sum' is not > private for the inner parallel with block, as a prange in a parallel > with block is a worksharing loop that binds to that parallel with > block. However, the outermost parallel with block declares sum (and i > and j) private, so after that block all those variables become > undefined. > > However, in the outermost parallel with block, sum will have to be > initialized to 0 before anything else, or be declared firstprivate, > otherwise 'sum' is undefined to begin with. Do you think declaring it > firstprivate would be the way to go, or should we make it private and > issue a warning or perhaps even an error? > >> DS >> ___ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > Everything seems to be working, although now the user has to be careful with nested parallel blocks as variables can be private there (and not firstprivate), i.e., the user has to do initialization at the right place (e.g. in the outermost parallel block that determines it private). I'm thinking of adding a warning, as the C compiler does. Two issues are remaining: 1) explicit declarations of firstprivates Do we still want those? 2) buffer auxiliary vars When unpacking numpy buffers and using typed numpy arrays, can reassignment or updates of a buffer-related variable ever occur in nogil code sections? I'm thinking this is not possible and therefore all buffer variables may be shared in parallel (for) sections? ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
(apologies for top post) This all seems to scream 'disallow' to me, in particular since some openmp implementations may not support it etc. At any rate I feel 'parallel/parallel/prange/prange' is going to far; so next step could be to only allowing 'parallel/prange/parallel/prange'. But really, my feeling is that if you really do need this then you can always write a seperate function for the inner loop (I honestly can't think of a usecase anyway...). So I'd really drop it; at least until the rest of the gsoc project is completed :) DS -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. mark florisson wrote: On 16 April 2011 18:42, Dag Sverre Seljebotn wrote: > (Moving discussion from http://markflorisson.wordpress.com/, where Mark > said:) Ok, sure, it was just an issue I was wondering about at that moment, but it's a tricky issue, so thanks. > """ > Started a new branch https://github.com/markflorisson88/cython/tree/openmp . > > Now the question is whether sharing attributes should be propagated > outwards. e.g. if you do > > for i in prange(m): >for j in prange(n): >sum += i * j > > then ‘sum’ is a reduction for the inner parallel loop, but not for the outer > one. So the user would currently have to rewrite this to > > for i in prange(m): > for j in prange(n): >sum += i * j >sum += 0 > > which seems a bit silly . Of course, we could just disable nested > parallelism, or tell the users to use a prange and a ‘for from’ in such > cases. > """ > > Dag: Interesting. The first one is definitely the behaviour we want, as long > as it doesn't cause unintended consequences. > > I don't really think it will > -- the important thing is that that the order > of loop iteration evaluation > must be unimportant. And that is still true > (for the outer loop, as well as > for the inner) in your first example. > > Question: When you have nested > pranges, what will happen is that two nested > OpenMP parallel blocks are > used, right? And do you know if there is complete > freedom/"reentrancy" in > that variables that are thread-private in an outer > parallel block and be > shared in an inner one, and vice versa? An implementation may or may not > support it, and if it is supported the behaviour can be configured through > omp_set_nested(). So we should consider the case where it is supported and > enabled. If you have a lastprivate or reduction, and after the loop these are > (reduced and) assigned to the original variable. So if that happens inside a > parallel construct which does not declare the variable private to the > construct , you actually have a race. So e.g. the nested prange currently races in the outer parallel range. > If so I'd think that this algorithm should work and feel natural: > > - In each prange, for the purposes of variable private/shared/reduction > inference, consider all internal "prange" just as if they had been "range"; > no special treatment. > > - Recurse to children pranges. Right, that is most natural. Algorithmically, reductions and lastprivates (as those can have races if placed in inner parallel constructs) propagate outwards towards the outermost parallel block, or up to the first parallel with block, or up to the first construct that already determined the sharing attribute. e.g. with parallel: with parallel: for i in prange(n): for j in prange(n): sum += i * j # sum is well-defined here # sum is undefined here Here 'sum' is a reduction for the two innermost loops. 'sum' is not private for the inner parallel with block, as a prange in a parallel with block is a worksharin g loop that binds to that parallel with block. However, the outermost parallel with block declares sum (and i and j) private, so after that block all those variables become undefined. However, in the outermost parallel with block, sum will have to be initialized to 0 before anything else, or be declared firstprivate, otherwise 'sum' is undefined to begin with. Do you think declaring it firstprivate would be the way to go, or should we make it private and issue a warning or perhaps even an error? > DS >_ > cython-devel mailing list > cython-devel@python.org > > http://mail.python.org/mailman/listinfo/cython-devel > >_ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] prange CEP updated
On 18 April 2011 16:01, Dag Sverre Seljebotn wrote: > (apologies for top post) No problem, it means I have to scroll less :) > This all seems to scream 'disallow' to me, in particular since some openmp > implementations may not support it etc. > > At any rate I feel 'parallel/parallel/prange/prange' is going to far; so > next step could be to only allowing 'parallel/prange/parallel/prange'. > > But really, my feeling is that if you really do need this then you can > always write a seperate function for the inner loop (I honestly can't think > of a usecase anyway...). So I'd really drop it; at least until the rest of > the gsoc project is completed :) Ok, sure, I'll disallow it. Then the user won't be able to make mistakes and I don't have to detect the case and issue a warning for inner reductions or lastprivates :). > DS > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > mark florisson wrote: >> >> On 16 April 2011 18:42, Dag Sverre Seljebotn >> wrote: > (Moving discussion from http://markflorisson.wordpress.com/, where >> Mark > said:) Ok, sure, it was just an issue I was wondering about at that >> moment, but it's a tricky issue, so thanks. > """ > Started a new branch >> https://github.com/markflorisson88/cython/tree/openmp . > > Now the question >> is whether sharing attributes should be propagated > outwards. e.g. if you >> do > > for i in prange(m): > for j in prange(n): > sum += i * j > >> > then ‘sum’ is a reduction for the inner parallel loop, but not for the >> outer > one. So the user would currently have to rewrite this to > > for i >> in prange(m): > for j in prange(n): > sum += i * j > sum += 0 > >> > which seems a bit silly . Of course, we could just disable nested > >> parallelism, or tell the users to use a prange and a ‘for from’ in such > >> cases. > """ > > Dag: Interesting. The first one is definitely the behaviour >> we want, as long > as it doesn't cause unintended consequences. > > I don't >> really think it will -- the important thing is that that the order > of loop >> iteration evaluation must be unimportant. And that is still true > (for the >> outer loop, as well as for the inner) in your first example. > > Question: >> When you have nested pranges, what will happen is that two nested > OpenMP >> parallel blocks are used, right? And do you know if there is complete > >> freedom/"reentrancy" in that variables that are thread-private in an outer > >> parallel block and be shared in an inner one, and vice versa? An >> implementation may or may not support it, and if it is supported the >> behaviour can be configured through omp_set_nested(). So we should consider >> the case where it is supported and enabled. If you have a lastprivate or >> reduction, and after the loop these are (reduced and) assigned to the >> original variable. So if that happens inside a parallel construct which does >> not declare the variable private to the construct, you actually have a race. >> So e.g. the nested prange currently races in the outer parallel range. > If >> so I'd think that this algorithm should work and feel natural: > > - In >> each prange, for the purposes of variable private/shared/reduction > >> inference, consider all internal "prange" just as if they had been "range"; >> > no special treatment. > > - Recurse to children pranges. Right, that is >> most natural. Algorithmically, reductions and lastprivates (as those can >> have races if placed in inner parallel constructs) propagate outwards >> towards the outermost parallel block, or up to the first parallel with >> block, or up to the first construct that already determined the sharing >> attribute. e.g. with parallel: with parallel: for i in prange(n): for j in >> prange(n): sum += i * j # sum is well-defined here # sum is undefined here >> Here 'sum' is a reduction for the two innermost loops. 'sum' is not private >> for the inner parallel with block, as a prange in a parallel with block is a >> worksharing loop that binds to that parallel with block. However, the >> outermost parallel with block declares sum (and i and j) private, so after >> that block all those variables become undefined. However, in the outermost >> parallel with block, sum will have to be initialized to 0 before anything >> else, or be declared firstprivate, otherwise 'sum' is undefined to begin >> with. Do you think declaring it firstprivate would be the way to go, or >> should we make it private and issue a warning or perhaps even an error? > DS >> > >> >> > cython-devel mailing list > cython-devel@python.org > >> > http://mail.python.org/mailman/listinfo/cython-devel > >> >> cython-devel mailing list cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-
Re: [Cython] prange CEP updated
Excellent! Sounds great! (as I won't have my laptop for some days I can't have a look yet but I will later) You're right about (the current) buffers and the gil. A testcase explicitly for them would be good. Firstprivate etc: i think it'd be nice myself, but it is probably better to take a break from it at this point so that we can think more about that and not do anything rash; perhaps open up a specific thread on them and ask for more general input. Perhaps you want to take a break or task-switch to something else (fused types?) until I can get around to review and merge what you have so far? You'll know best what works for you though. If you decide to implement explicit threadprivate variables because you've got the flow I certainly wom't object myself. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. mark florisson wrote: On 18 April 2011 13:06, mark florisson wrote: > On 16 April 2011 18:42, Dag Sverre Seljebotn wrote: >> (Moving discussion from http://markflorisson.wordpress.com/, where Mark >> said:) > > Ok, sure, it was just an issue I was wondering about at that moment, > but it's a tricky issue, so thanks. > >> """ >> Started a new branch https://github.com/markflorisson88/cython/tree/openmp . >> >> Now the question is whether sharing attributes should be propagated >> outwards. e.g. if you do >> >> for i in prange(m): >>for j in prange(n): >>sum += i * j >> >> then ‘sum’ is a reduction for the inner parallel loop, but not for the outer >> one. So the user would currently have to rewrite this to >> >> for i in prange(m): >>for j in prange(n): >>sum += i * j >>sum += 0 >> >> which seems a bit silly . Of course, we could just disable nested >> parallelism, or tell the users to use a prange and a ‘fo r from’ in such >> cases. >> """ >> >> Dag: Interesting. The first one is definitely the behaviour we want, as long >> as it doesn't cause unintended consequences. >> >> I don't really think it will -- the important thing is that that the order >> of loop iteration evaluation must be unimportant. And that is still true >> (for the outer loop, as well as for the inner) in your first example. >> >> Question: When you have nested pranges, what will happen is that two nested >> OpenMP parallel blocks are used, right? And do you know if there is complete >> freedom/"reentrancy" in that variables that are thread-private in an outer >> parallel block and be shared in an inner one, and vice versa? > > An implementation may or may not support it, and if it is supported > the behaviour can be configured through omp_set_nested(). So we should > consider the case where it is supported and enabled. > > If you have a lastprivate or reduction, and after the loop these are > (reduced and) as signed to the original variable. So if that happens > inside a parallel construct which does not declare the variable > private to the construct, you actually have a race. So e.g. the nested > prange currently races in the outer parallel range. > >> If so I'd think that this algorithm should work and feel natural: >> >> - In each prange, for the purposes of variable private/shared/reduction >> inference, consider all internal "prange" just as if they had been "range"; >> no special treatment. >> >> - Recurse to children pranges. > > Right, that is most natural. Algorithmically, reductions and > lastprivates (as those can have races if placed in inner parallel > constructs) propagate outwards towards the outermost parallel block, > or up to the first parallel with block, or up to the first construct > that already determined the sharing attribute. > > e.g. > > with parallel: > with parallel: >for i in prange(n): > for j in prange(n): >sum += i * j > # sum is well-defined here > # sum is undefined here > > Here 'sum' is a reduction for the two innermost loops. 'sum' is not > private for the inner parallel with block, as a prange in a parallel > with block is a worksharing loop that binds to that parallel with > block. However, the outermost parallel with block declares sum (and i > and j) private, so after that block all those variables become > undefined. > > However, in the outermost parallel with block, sum will have to be > initialized to 0 before anything else, or be declared firstprivate, > otherwise 'sum' is undefined to begin with. Do you think declaring it > firstprivate would be the way to go, or should we make it private and > issue a warning or perhaps even an error? > >> DS >>_ >> cython-devel mailing list >> cython-devel@python.org >> >> http://mail.python.org/mailman/listinfo/cython-devel >> > Everything seems >> to be working, although now the user has to be careful with nested parallel >> blocks as variables can be private there (and not firstprivate), i.e., the >> user has to do initialization at the right
Re: [Cython] prange CEP updated
On 18 April 2011 16:41, Dag Sverre Seljebotn wrote: > Excellent! Sounds great! (as I won't have my laptop for some days I can't > have a look yet but I will later) > > You're right about (the current) buffers and the gil. A testcase explicitly > for them would be good. > > Firstprivate etc: i think it'd be nice myself, but it is probably better to > take a break from it at this point so that we can think more about that and > not do anything rash; perhaps open up a specific thread on them and ask for > more general input. Perhaps you want to take a break or task-switch to > something else (fused types?) until I can get around to review and merge > what you have so far? You'll know best what works for you though. If you > decide to implement explicit threadprivate variables because you've got the > flow I certainly wom't object myself. > Ok, cool, I'll move on :) I already included a test with a prange and a numpy buffer with indexing. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
Hi I investigated the code produced by Cython, and I see 3 cases which should be handled : * A pure python variable which has a value assigned (including None) * A pure python variable which has no value assigned * A C variable (we can't test if they are set of not) The first and second one seem relatively easy to implement it can reuse __pyx_string_tab. The code will call __Pyx_GetName on each element of the array, if it returns NULL, don't put the variable in the dictionary, else put the name and it's value in the dictionary. For the last one, it will probably need the C name, C type and Python name of each C variable, add the python name in the dictionary and wrap the C value into a Python object. However this will only provide read access to the globals. To offer read/write access, making a proxy dictionary is probably the most straightforward implementation (it can be rather inefficient though since we need to do an if/else to compare the value we want to set with every C globals). Do you think I'm on the right way ? Thanks Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] gilnanny
Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get() for every refnanny inc- and decref, so that it will issue a fatal error whenever reference counting is done without the gil, to make sure we never do any illegal things in nogil code blocks. While I'm at it, I also want to add a pystate.pxd to the cpython includes. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] gilnanny
On Mon, Apr 18, 2011 at 12:08 PM, mark florisson wrote: > Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get() > for every refnanny inc- and decref, so that it will issue a fatal > error whenever reference counting is done without the gil, to make > sure we never do any illegal things in nogil code blocks. Sounds like a good idea to me. > While I'm at it, I also want to add a pystate.pxd to the cpython includes. Sure, corresponding to pystate.h? Have you looked into how stable this API is (e.g. Py2 vs. Py3)? - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] gilnanny
Den 18.04.2011 22:26, skrev Robert Bradshaw: On Mon, Apr 18, 2011 at 12:08 PM, mark florisson wrote: Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get() for every refnanny inc- and decref, so that it will issue a fatal error whenever reference counting is done without the gil, to make sure we never do any illegal things in nogil code blocks. Sounds like a good idea to me. Have you ever considered to allow a "with gil:" statement? It seems this could be implemented using the simplified GIL API, i.e. the same way ctypes synchronizes callbacks to Python. Usecases would e.g. be computational code that sometimes needs to touch Python objects. E.g. append something to a list, slice a NumPy array, unbox a buffer into local scope, etc. A "with gil" statement could allow us to grab the GIL back for that. Sturla ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] gilnanny
Sturla Molden, 19.04.2011 00:18: Den 18.04.2011 22:26, skrev Robert Bradshaw: On Mon, Apr 18, 2011 at 12:08 PM, mark florisson wrote: Can I add a gilnanny to refnanny? I want to do a PyThreadState_Get() for every refnanny inc- and decref, so that it will issue a fatal error whenever reference counting is done without the gil, to make sure we never do any illegal things in nogil code blocks. Sounds like a good idea to me. Have you ever considered to allow a "with gil:" statement? Yes, that's what this is all about. https://github.com/markflorisson88/cython/commits/master Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Recent bugs in generators
Vitja Makarov, 18.04.2011 15:19: There is one more interesting thing: def test_yield_inside_genexp(): """ >>> o = test_yield_inside_genexp() >>> list(o) [0, None, 1, None, 2, None] """ return ((yield i) for i in range(3)) Impressively ugly. I guess we should start testing these things with other Python implementations than CPython, in order to see if these corner cases are really being accepted and knowingly being implemented, or if it's just something that no-one has really cared about so far and that's actually worth discussing. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Recent bugs in generators
2011/4/19 Stefan Behnel : > Vitja Makarov, 18.04.2011 15:19: >> >> There is one more interesting thing: >> >> def test_yield_inside_genexp(): >> """ >> >>> o = test_yield_inside_genexp() >> >>> list(o) >> [0, None, 1, None, 2, None] >> """ >> return ((yield i) for i in range(3)) > > Impressively ugly. > > I guess we should start testing these things with other Python > implementations than CPython, in order to see if these corner cases are > really being accepted and knowingly being implemented, or if it's just > something that no-one has really cared about so far and that's actually > worth discussing. > The case above is yield inside genexp and it works now. Btw we've found one more funny thing this time with parser: 0lor(1) -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
Romain Guillebert, 18.04.2011 17:55: I investigated the code produced by Cython, and I see 3 cases which should be handled : * A pure python variable which has a value assigned (including None) * A pure python variable which has no value assigned * A C variable (we can't test if they are set of not) The first and second one seem relatively easy to implement it can reuse __pyx_string_tab. The code will call __Pyx_GetName on each element of the array, if it returns NULL, don't put the variable in the dictionary, else put the name and it's value in the dictionary. For the last one, it will probably need the C name, C type and Python name of each C variable, add the python name in the dictionary and wrap the C value into a Python object. However this will only provide read access to the globals. To offer read/write access, making a proxy dictionary is probably the most straightforward implementation (it can be rather inefficient though since we need to do an if/else to compare the value we want to set with every C globals). Do you think I'm on the right way ? Thanks, Romain. Certainly sounds "good enough" to me. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
