[Cython] 'with gil:' statement
I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? The 'with gil:' statement can now be used in the same way as 'with nogil:'. Exceptions raised from GIL blocks will be propagated if possible (i.e., if the return type is 'object'). Otherwise it will jump to the end of the function and use the usual __Pyx_WriteUnraisable, there's not really anything new there. For functions declared 'nogil' that contain 'with gil:' statements, it will safely lock around around the initialization of any Python objects and set up the refnanny context (with appropriate preprocessor guards). At the end of the function it will safely lock around the teardown of the refnanny context. With 'safely' I mean that it will create a thread state if it was not already created and may be called even while the GIL is already held (using PyGILState_Ensure()). This means variables are declared and initialized in the same way as in normal GIL-holding functions (except that there is additional locking), and of course the GIL-checking code ensures that errors are issued if those variables are attempted to be used outside any GIL blocks. Could someone review the patch (which is attached)? Maybe check if I haven't missed any side cases and such? with_gil_statement.patch Description: Binary data ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/16/2011 11:28 AM, mark florisson wrote: I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # => if True: This is because one may want to change "with nogil" to "with gil" for debugging purposes (allow printing debug information). Another feedback is that I wonder whether we should put the "gil" and "nogil" psuedo-context managers both in cython namespace, and sort of deprecate the "global" nogil, rather than introduce yet another name that can't be used safely for all kinds of variables. -- Dag The 'with gil:' statement can now be used in the same way as 'with nogil:'. Exceptions raised from GIL blocks will be propagated if possible (i.e., if the return type is 'object'). Otherwise it will jump to the end of the function and use the usual __Pyx_WriteUnraisable, there's not really anything new there. For functions declared 'nogil' that contain 'with gil:' statements, it will safely lock around around the initialization of any Python objects and set up the refnanny context (with appropriate preprocessor guards). At the end of the function it will safely lock around the teardown of the refnanny context. With 'safely' I mean that it will create a thread state if it was not already created and may be called even while the GIL is already held (using PyGILState_Ensure()). This means variables are declared and initialized in the same way as in normal GIL-holding functions (except that there is additional locking), and of course the GIL-checking code ensures that errors are issued if those variables are attempted to be used outside any GIL blocks. Could someone review the patch (which is attached)? Maybe check if I haven't missed any side cases and such? ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] PyCharm
The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the Cython project a free Open Source license, which means that anyone developing Cython may freely use PyCharm to develop Cython. They prefer to license to remain unpublic, so if you develop Cython and want a free PyCharm license, send me an email and I will send you the license key. It is valid until 13 March 2012. Cheers, Mark ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: > On 03/16/2011 11:28 AM, mark florisson wrote: > > I implemented the 'with gil:' statement, and have added error checking > for nested 'with gil' or 'with nogil' statements. For instance, with > the patch applied Cython wil issue an error when you have e.g. > > with nogil: > with nogil: > ... > > (or the same thing for 'with gil:'). This because nested 'nogil' or > 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort > the Python interpreter. However, if people are acquiring the GIL > manually in nogil blocks and then want to release the GIL with a > 'nogil' block, it will incorrectly issue an error. I do think this > kind of code is uncommon, but perhaps we want to issue a warning > instead? > > I think we should make nested nogil-s noops, i.e. > > with nogil: > with nogil: # => if True: > > This is because one may want to change "with nogil" to "with gil" for > debugging purposes (allow printing debug information). Interesting, that does sound convenient, but I'm not if mere convenience should move us to simply ignore what is technically most likely incorrect code (unless there is intermediate manual locking). In any case, I wouldn't really be against that. If you simply want to allow this for debugging, we could also allow print statements in nogil sections, by either rewriting it using 'with gil:', or by inserting a simple printf (in which case you probably want to place a few restrictions). > Another feedback is that I wonder whether we should put the "gil" and > "nogil" psuedo-context managers both in cython namespace, and sort of > deprecate the "global" nogil, rather than introduce yet another name that > can't be used safely for all kinds of variables. Hmm, good catch. Actually, 'with cython.nogil:' is already possible, but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to simply be ignored). So I guess I'll have to fix that :) > -- Dag > > The 'with gil:' statement can now be used in the same way as 'with > nogil:'. Exceptions raised from GIL blocks will be propagated if > possible (i.e., if the return type is 'object'). Otherwise it will > jump to the end of the function and use the usual > __Pyx_WriteUnraisable, there's not really anything new there. > > For functions declared 'nogil' that contain 'with gil:' statements, it > will safely lock around around the initialization of any Python > objects and set up the refnanny context (with appropriate preprocessor > guards). At the end of the function it will safely lock around the > teardown of the refnanny context. With 'safely' I mean that it will > create a thread state if it was not already created and may be called > even while the GIL is already held (using PyGILState_Ensure()). This > means variables are declared and initialized in the same way as in > normal GIL-holding functions (except that there is additional > locking), and of course the GIL-checking code ensures that errors are > issued if those variables are attempted to be used outside any GIL > blocks. > > Could someone review the patch (which is attached)? Maybe check if I > haven't missed any side cases and such? > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
>> Another feedback is that I wonder whether we should put the "gil" and >> "nogil" psuedo-context managers both in cython namespace, and sort of >> deprecate the "global" nogil, rather than introduce yet another name that >> can't be used safely for all kinds of variables. > > Hmm, good catch. Actually, 'with cython.nogil:' is already possible, > but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to > simply be ignored). > So I guess I'll have to fix that :) > I attached a patch for 'with cython.gil:'. with_cython.gil_statement.patch Description: Binary data ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 13:01, mark florisson wrote: >>> Another feedback is that I wonder whether we should put the "gil" and >>> "nogil" psuedo-context managers both in cython namespace, and sort of >>> deprecate the "global" nogil, rather than introduce yet another name that >>> can't be used safely for all kinds of variables. >> >> Hmm, good catch. Actually, 'with cython.nogil:' is already possible, >> but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to >> simply be ignored). >> So I guess I'll have to fix that :) >> > > I attached a patch for 'with cython.gil:'. > Attached a patch to disallow invalid directives in with statements. I ran the test suite to ensure I didn't break anything. error_on_invalid_with_directives.patch Description: Binary data ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/16/2011 12:54 PM, mark florisson wrote: On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: On 03/16/2011 11:28 AM, mark florisson wrote: I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # => if True: This is because one may want to change "with nogil" to "with gil" for debugging purposes (allow printing debug information). Interesting, that does sound convenient, but I'm not if mere convenience should move us to simply ignore what is technically most likely incorrect code (unless there is intermediate manual locking). I'm not sure if I understand what you mean here. How does simply ignoring redundant "with [no]gil" statements cause incorrect code? Or do you mean this is a I'm just trying to minimize the "language getting in your way" factor. It is pretty useless to write if x: if x: ... as well, but Python does allow it. Warnings on nested "with nogil" is more the role of a "cylint" in my opinion. In any case, I wouldn't really be against that. If you simply want to allow this for debugging, we could also allow print statements in nogil sections, by either rewriting it using 'with gil:', or by inserting a simple printf (in which case you probably want to place a few restrictions). It's not only print statements. I.e., if I think something is wrong with an array, I'll stick in code like print np.std(x), np.mean(x), np.any(np.isnan(x)) or something more complex that may require temporaries. Or even plot the vector: plt.plot(x) plt.show() # blocks until I close plot window Or, launch a debugger: if np.any(np.isnan(x)): import pdb; pdb.set_trace() so I guess I should stop saying this is only about printing. In general, it's nice to use Python during debugging. I find myself replacing "with nogil" with "if True" in such situations, to avoid reindenting. I guess I can soon start using "with gil" around the debug code though. Again, the restriction on nested nogil/gil is not a big problem, just an instance of "the language getting in your way". Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 13:37, Dag Sverre Seljebotn wrote: > On 03/16/2011 12:54 PM, mark florisson wrote: >> >> On 16 March 2011 11:58, Dag Sverre Seljebotn >> wrote: >>> >>> On 03/16/2011 11:28 AM, mark florisson wrote: >>> >>> I implemented the 'with gil:' statement, and have added error checking >>> for nested 'with gil' or 'with nogil' statements. For instance, with >>> the patch applied Cython wil issue an error when you have e.g. >>> >>> with nogil: >>> with nogil: >>> ... >>> >>> (or the same thing for 'with gil:'). This because nested 'nogil' or >>> 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort >>> the Python interpreter. However, if people are acquiring the GIL >>> manually in nogil blocks and then want to release the GIL with a >>> 'nogil' block, it will incorrectly issue an error. I do think this >>> kind of code is uncommon, but perhaps we want to issue a warning >>> instead? >>> >>> I think we should make nested nogil-s noops, i.e. >>> >>> with nogil: >>> with nogil: # => if True: >>> >>> This is because one may want to change "with nogil" to "with gil" for >>> debugging purposes (allow printing debug information). >> >> Interesting, that does sound convenient, but I'm not if mere >> convenience should move us to simply ignore what is technically most >> likely incorrect code (unless there is intermediate manual locking). > > I'm not sure if I understand what you mean here. How does simply ignoring > redundant "with [no]gil" statements cause incorrect code? Or do you mean > this is a > > I'm just trying to minimize the "language getting in your way" factor. It is > pretty useless to write > > if x: > if x: > ... > > as well, but Python does allow it. > > Warnings on nested "with nogil" is more the role of a "cylint" in my > opinion. > Perhaps you're right. However, I just think it is important for users to realize that in general, they cannot unblock threads recursively. Currently the error checking code catches multiple nested 'with (no)gil', but it doesn't catch this: cdef void func() nogil: with nogil: pass with nogil: func() But the problem is that it does abort the interpreter. So I thought that perhaps emphasizing that that code is incorrect for at least the easy-to-catch cases, we might make users somewhat more aware. Because if the above code aborts Python, but a nested 'with nogil:' is valid code, there might be a source for confusion. >> In any case, I wouldn't really be against that. If you simply want to >> allow this for debugging, we could also allow print statements in >> nogil sections, by either rewriting it using 'with gil:', or by >> inserting a simple printf (in which case you probably want to place a >> few restrictions). > > It's not only print statements. I.e., if I think something is wrong with an > array, I'll stick in code like > > print np.std(x), np.mean(x), np.any(np.isnan(x)) > > or something more complex that may require temporaries. Or even plot the > vector: > > plt.plot(x) > plt.show() # blocks until I close plot window > > Or, launch a debugger: > > if np.any(np.isnan(x)): > import pdb; pdb.set_trace() > > so I guess I should stop saying this is only about printing. In general, > it's nice to use Python during debugging. I find myself replacing "with > nogil" with "if True" in such situations, to avoid reindenting. > > I guess I can soon start using "with gil" around the debug code though. > Again, the restriction on nested nogil/gil is not a big problem, just an > instance of "the language getting in your way". I understand. If your code doesn't introduce Cython-level blocks, you could at least put it on one line so you can easily comment it out. > Dag > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 13:55, mark florisson wrote: > On 16 March 2011 13:37, Dag Sverre Seljebotn > wrote: >> On 03/16/2011 12:54 PM, mark florisson wrote: >>> >>> On 16 March 2011 11:58, Dag Sverre Seljebotn >>> wrote: On 03/16/2011 11:28 AM, mark florisson wrote: I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # => if True: This is because one may want to change "with nogil" to "with gil" for debugging purposes (allow printing debug information). >>> >>> Interesting, that does sound convenient, but I'm not if mere >>> convenience should move us to simply ignore what is technically most >>> likely incorrect code (unless there is intermediate manual locking). >> >> I'm not sure if I understand what you mean here. How does simply ignoring >> redundant "with [no]gil" statements cause incorrect code? Or do you mean >> this is a >> >> I'm just trying to minimize the "language getting in your way" factor. It is >> pretty useless to write >> >> if x: >> if x: >> ... >> >> as well, but Python does allow it. >> >> Warnings on nested "with nogil" is more the role of a "cylint" in my >> opinion. >> > > Perhaps you're right. However, I just think it is important for users > to realize that in general, they cannot unblock threads recursively. > Currently the error checking code catches multiple nested 'with > (no)gil', but it doesn't catch this: > > cdef void func() nogil: > with nogil: > pass > > with nogil: > func() > > But the problem is that it does abort the interpreter. So I thought > that perhaps emphasizing that that code is incorrect for at least the > easy-to-catch cases, we might make users somewhat more aware. Because > if the above code aborts Python, but a nested 'with nogil:' is valid > code, there might be a source for confusion. > I have to mention, though, that the converse is not true. The gil may be acquired recursively, i.e., 'with gil:' blocks may be nested. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/16/2011 01:55 PM, mark florisson wrote: On 16 March 2011 13:37, Dag Sverre Seljebotn wrote: On 03/16/2011 12:54 PM, mark florisson wrote: On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: On 03/16/2011 11:28 AM, mark florisson wrote: I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # =>if True: This is because one may want to change "with nogil" to "with gil" for debugging purposes (allow printing debug information). Interesting, that does sound convenient, but I'm not if mere convenience should move us to simply ignore what is technically most likely incorrect code (unless there is intermediate manual locking). I'm not sure if I understand what you mean here. How does simply ignoring redundant "with [no]gil" statements cause incorrect code? Or do you mean this is a I'm just trying to minimize the "language getting in your way" factor. It is pretty useless to write if x: if x: ... as well, but Python does allow it. Warnings on nested "with nogil" is more the role of a "cylint" in my opinion. Perhaps you're right. However, I just think it is important for users to realize that in general, they cannot unblock threads recursively. Currently the error checking code catches multiple nested 'with (no)gil', but it doesn't catch this: cdef void func() nogil: with nogil: pass with nogil: func() But the problem is that it does abort the interpreter. So I thought that perhaps emphasizing that that code is incorrect for at least the easy-to-catch cases, we might make users somewhat more aware. Because if the above code aborts Python, but a nested 'with nogil:' is valid code, there might be a source for confusion. Ah, right. I guess I agree with disallowing nested "with nogil" statements for the time being then. Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
mark florisson, 16.03.2011 13:28: On 16 March 2011 13:01, mark florisson wrote: Another feedback is that I wonder whether we should put the "gil" and "nogil" psuedo-context managers both in cython namespace, and sort of deprecate the "global" nogil, rather than introduce yet another name that can't be used safely for all kinds of variables. Hmm, good catch. Actually, 'with cython.nogil:' is already possible, but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to simply be ignored). So I guess I'll have to fix that :) I attached a patch for 'with cython.gil:'. Looks ok. Attached a patch to disallow invalid directives in with statements. +1, except that I'd print the directive name as part of the error message. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Dag Sverre Seljebotn, 16.03.2011 11:58: On 03/16/2011 11:28 AM, mark florisson wrote: I implemented the 'with gil:' statement, and have added error checking for nested 'with gil' or 'with nogil' statements. For instance, with the patch applied Cython wil issue an error when you have e.g. with nogil: with nogil: ... (or the same thing for 'with gil:'). This because nested 'nogil' or 'gil' blocks, without intermediate GIL-acquiring/releasing, will abort the Python interpreter. However, if people are acquiring the GIL manually in nogil blocks and then want to release the GIL with a 'nogil' block, it will incorrectly issue an error. I do think this kind of code is uncommon, but perhaps we want to issue a warning instead? I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # => if True: -1, the compiler should bark at clearly nonsensical code. The above, if encountered in the wild, is most likely a copy&paste bug that wasn't intended that way. (ok, you'd most likely get other errors as well due to GIL checking, but that doesn't change the fact that the code is nonsensical) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Dag Sverre Seljebotn, 16.03.2011 13:37: On 03/16/2011 12:54 PM, mark florisson wrote: On 16 March 2011 11:58, Dag Sverre Seljebotn wrote: I think we should make nested nogil-s noops, i.e. with nogil: with nogil: # => if True: This is because one may want to change "with nogil" to "with gil" for debugging purposes (allow printing debug information). Interesting, that does sound convenient, but I'm not if mere convenience should move us to simply ignore what is technically most likely incorrect code (unless there is intermediate manual locking). I'm just trying to minimize the "language getting in your way" factor. It is pretty useless to write if x: if x: ... as well, but Python does allow it. That's because it's not necessarily useless. It can have side-effects. In any case, I wouldn't really be against that. If you simply want to allow this for debugging, we could also allow print statements in nogil sections, by either rewriting it using 'with gil:', or by inserting a simple printf (in which case you probably want to place a few restrictions). It's not only print statements. I.e., if I think something is wrong with an array, I'll stick in code like print np.std(x), np.mean(x), np.any(np.isnan(x)) or something more complex that may require temporaries. Or even plot the vector: plt.plot(x) plt.show() # blocks until I close plot window Or, launch a debugger: if np.any(np.isnan(x)): import pdb; pdb.set_trace() All of these are better expressed using an explicit "with gil", also in debug code. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PyCharm
mark florisson, 16.03.2011 11:33: The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the Cython project a free Open Source license, which means that anyone developing Cython may freely use PyCharm to develop Cython. Looks like they don't support Cython code, though: http://youtrack.jetbrains.net/issue/PY-1046 (although, maybe that's the reason for the offer ;) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PyCharm
On 16 March 2011 15:26, Stefan Behnel wrote: > mark florisson, 16.03.2011 11:33: >> >> The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the >> Cython project a free Open Source license, which means that anyone >> developing Cython may freely use PyCharm to develop Cython. > > Looks like they don't support Cython code, though: > > http://youtrack.jetbrains.net/issue/PY-1046 > > (although, maybe that's the reason for the offer ;) > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > Indeed, that is rather unfortunate. I only use it for plain Python code, so I work with two editors, which is somewhat inconvenient. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 15:07, Stefan Behnel wrote: > mark florisson, 16.03.2011 13:28: >> >> On 16 March 2011 13:01, mark florisson wrote: > > Another feedback is that I wonder whether we should put the "gil" and > "nogil" psuedo-context managers both in cython namespace, and sort of > deprecate the "global" nogil, rather than introduce yet another name > that > can't be used safely for all kinds of variables. Hmm, good catch. Actually, 'with cython.nogil:' is already possible, but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to simply be ignored). So I guess I'll have to fix that :) >>> >>> I attached a patch for 'with cython.gil:'. > > Looks ok. Does that mean I should push the changes, or do you want some more elaborate reviewing first? > >> Attached a patch to disallow invalid directives in with statements. > > +1, except that I'd print the directive name as part of the error message. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
Robert Bradshaw, 11.03.2011 19:33: On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel wrote: Stefan Behnel, 11.03.2011 15:08: Vitja Makarov, 11.03.2011 15:04: 2011/3/11 Stefan Behnel: Personally, I think it would be nice to keep up Python's semantics, but when I implemented this I broke quite some code in Sage (you may have noticed that the sage-build project in Hudson has been red for a while). There are things in C and especially in C++ that cannot be easily copied into a temporary variable in order to make sure they are evaluated before the following arguments. This is not a problem for Python function calls where all arguments end up being copied (and often converted) anyway. It is a problem for C function calls, though. f(g(a), a.x, h(a)) Why could not this be translated into: tmp1 = g(a) tmp2 = a.x tmp3 = h(a) f(tmp1, tmp2, tmp3) See above. To be a little clearer here, it's a problem in C for example with struct values. Copying them by value into a temp variable can be expensive, potentially twice as expensive as simply passing them into the function normally. Yep, and some types (e.g. array types) can't be assigned to at all. FWIW, the issues with Sage is that many libraries use the "stack allocated, pass-by-reference" trick typedef foo_struct foo_t[1] but we have declared them to be of type "void*" because we don't care about or want to muck with the internals. Cleaning this up is something we should do, but is taking a while, and aside from that it makes Cython even more dependent on correct type declarations (and backwards incompatible in this regard). Sage is a great test for the buildbot, so keeping it red for so long is not a good thing either. Not sure what kind of additional devilry C++ provides here, but I'd expect that object values can exhibit bizarre behaviour when being assigned. Maybe others can enlighten me here. Yes, C++ allows overloading of the assignment operator, so assigning may lead to arbitrary code (as well as probably an expensive copy, as with structs, and structs in C++ are really just classes with a different visibility). I have no idea how many cases there actually are that we can't handle or that may lead to a performance degradation when using temps, but the problem is that they exist at all. Note that this applies not just to function arguments, but a host of other places, e.g. the order of evaluation of "f() + g()" in C is unspecified. Fleshing this out completely will lead to a lot more temps and verbose C code. And then we'd just cross our fingers that the C compiler was able to optimize all these temps away (though still possibly producing inferior code if it were able to switch order of execution). Whatever we do, we need a flag/directive. Perhaps there's a way to guarantee correct ordering for all valid Python code Plain Python code should never suffer from the original problem (with the obvious exception of cpdef functions), but I also see how type inference can affect this guarantee for the evaluation order in expressions. I agree with Greg, though, that it has a code smell if (multiple) subexpressions in an expression have side-effects, especially when they are impacted by the evaluation order. That means that the average innocent future maintainer could break the code by manually rearranging the expression in a way that's perfectly valid mathematically, e.g. as a readability fix. even in the presence of (function and variable) type inference, but allow some leeway for explicitly declared C functions. I'm actually leaning towards not guaranteeing the order of execution if C doesn't do it either. If this is really required, it's easy to work around for users, but it's severely hard to fix for Cython in all cases, and the gain is truly small. After all, we'd only make it easier for users to write bad code. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Wed, 16 Mar 2011 14:10:29 +0100, Dag Sverre Seljebotn wrote: > Ah, right. I guess I agree with disallowing nested "with nogil" > statements for the time being then. Could you make the inner nested "with nogil" statements no-ops instead, if the GIL is already released? Does the Cython compiler keep track if GIL is acquired or not? -- Pauli Virtanen ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
mark florisson, 16.03.2011 11:28: I implemented the 'with gil:' statement [...] The 'with gil:' statement can now be used in the same way as 'with nogil:'. Exceptions raised from GIL blocks will be propagated if possible (i.e., if the return type is 'object'). Otherwise it will jump to the end of the function and use the usual __Pyx_WriteUnraisable, there's not really anything new there. I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Would you do the same when calling a cdef function that uses "with gil" in its signature? We don't currently do that, but it feels like it's the same situation. For functions declared 'nogil' that contain 'with gil:' statements, it will safely lock around around the initialization of any Python objects and set up the refnanny context (with appropriate preprocessor guards). At the end of the function it will safely lock around the teardown of the refnanny context. With 'safely' I mean that it will create a thread state if it was not already created and may be called even while the GIL is already held (using PyGILState_Ensure()). This means variables are declared and initialized in the same way as in normal GIL-holding functions (except that there is additional locking), and of course the GIL-checking code ensures that errors are issued if those variables are attempted to be used outside any GIL blocks. I find that surprising semantics. So the GIL will always be acquired at least twice in the following example, regardless of the input? cdef int callme(bint flag) nogil: if flag: with gil: x = object() Could someone review the patch (which is attached)? Maybe check if I haven't missed any side cases and such? From a first look, the test file you added seems far too short. I would expect that this feature requires a lot more testing in combination with declared and undeclared local variables, type inference, several exception raising and catching situations (e.g. raise in one block, catch in an outer block, try-finally, ...) or looping. It may also have an impact on Vitja's control flow analysis branch that's worth considering. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
mark florisson, 16.03.2011 15:29: On 16 March 2011 15:07, Stefan Behnel wrote: mark florisson, 16.03.2011 13:28: On 16 March 2011 13:01, mark florisson wrote: Another feedback is that I wonder whether we should put the "gil" and "nogil" psuedo-context managers both in cython namespace, and sort of deprecate the "global" nogil, rather than introduce yet another name that can't be used safely for all kinds of variables. Hmm, good catch. Actually, 'with cython.nogil:' is already possible, but cython.gil isn't (in fact, 'with cython.somethingrandom:' seems to simply be ignored). So I guess I'll have to fix that :) I attached a patch for 'with cython.gil:'. Looks ok. Does that mean I should push the changes, or do you want some more elaborate reviewing first? I was only commenting on the two simpler follow-up changes. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/16/2011 02:17 PM, Pauli Virtanen wrote: Wed, 16 Mar 2011 14:10:29 +0100, Dag Sverre Seljebotn wrote: Ah, right. I guess I agree with disallowing nested "with nogil" statements for the time being then. Could you make the inner nested "with nogil" statements no-ops instead, if the GIL is already released? Does the Cython compiler keep track if GIL is acquired or not? That's what I initially suggested. See Mark's posted code -- when calling another function, you can't know whether the nogil is "nested" or not (without actually checking with CPython...) Within-Cython solutions are bad because calls may cross between different Cython modules. Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 16:14, Stefan Behnel wrote:
> mark florisson, 16.03.2011 11:28:
>>
>> I implemented the 'with gil:' statement
>> [...]
>> The 'with gil:' statement can now be used in the same way as 'with
>> nogil:'. Exceptions raised from GIL blocks will be propagated if
>> possible (i.e., if the return type is 'object'). Otherwise it will
>> jump to the end of the function and use the usual
>> __Pyx_WriteUnraisable, there's not really anything new there.
>
> I'm not sure if this is a good idea. "nogil" blocks don't have a way to
> handle exceptions, so simply jumping out of them because an inner 'with gil'
> block raised an exception can have unexpected side effects.
>
> Would you do the same when calling a cdef function that uses "with gil" in
> its signature? We don't currently do that, but it feels like it's the same
> situation.
>
I beg to differ. Consider this code:
with nogil:
with gil:
raise SomeError
printf("something\n")
With normal 'with' statements the printf (in the snippet above) would
be skipped, and so it is with 'with gil:' blocks, it follows the exact
semantics of the Python behaviour as you expect it. With statements
are like try/finally, and the 'with (no)gil:' blocks are implemented
as such. The above would translate to:
try:
release gil
try:
acquire gil
raise SomeError
finally:
release gil
printf("something\n")
finally:
acquire gil
If users need to do (C-level) cleanup, they need to wrap the code in
the 'with gil:' in something like try/finally, or perhaps try/except.
In fact, if you declare a function 'with gil', you will want to either
handle any exception in that function, or you want your caller to
check for errors while returning an error indicator. So in that case,
you should still do error checking, if you don't, exceptions will
simply be ignored.
Another related issue, you cannot declare your function with both
'with gil' and 'except NULL' at the same time, as calling such a
function without having the GIL will segfault your program. In fact, I
think we should add a check for this, as e.g. the following code will
segfault in func() when it calls __Pyx_WriteUnraisable, as it will try
to get the exception from a NULL thread-state.
from cpython.ref cimport PyObject
from libc.stdio cimport printf
cdef PyObject *func2() except NULL with gil:
raise Exception("blah")
cdef void func() nogil:
cdef PyObject *p = func2()
printf("This is not printed!\n")
func()
Of course, you cannot really declare the return type of func2 as
'object', because you wouldn't be able to call it from 'nogil'
sections that way.
I guess my point is, if the user isn't very careful about the way
exceptions are handled, undesired behaviour may occur. With both
functions and with gil blocks, the user will have to override the
default behaviour if needed. I think "python semantics" most
accurately reflect the syntax for 'with gil:' blocks.
>> For functions declared 'nogil' that contain 'with gil:' statements, it
>> will safely lock around around the initialization of any Python
>> objects and set up the refnanny context (with appropriate preprocessor
>> guards). At the end of the function it will safely lock around the
>> teardown of the refnanny context. With 'safely' I mean that it will
>> create a thread state if it was not already created and may be called
>> even while the GIL is already held (using PyGILState_Ensure()). This
>> means variables are declared and initialized in the same way as in
>> normal GIL-holding functions (except that there is additional
>> locking), and of course the GIL-checking code ensures that errors are
>> issued if those variables are attempted to be used outside any GIL
>> blocks.
>
> I find that surprising semantics. So the GIL will always be acquired at
> least twice in the following example, regardless of the input?
>
> cdef int callme(bint flag) nogil:
> if flag:
> with gil: x = object()
>
Yes. It should be noted to users that 'with gil:' does not come
without its repercussions.
>> Could someone review the patch (which is attached)? Maybe check if I
>> haven't missed any side cases and such?
>
> From a first look, the test file you added seems far too short. I would
> expect that this feature requires a lot more testing in combination with
> declared and undeclared local variables, type inference, several exception
> raising and catching situations (e.g. raise in one block, catch in an outer
> block, try-finally, ...) or looping. It may also have an impact on Vitja's
> control flow analysis branch that's worth considering.
I agree. I think I'll start a branch and work on some more tests.
> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.or
Re: [Cython] 'with gil:' statement
>>> Could someone review the patch (which is attached)? Maybe check if I >>> haven't missed any side cases and such? >> >> From a first look, the test file you added seems far too short. I would >> expect that this feature requires a lot more testing in combination with >> declared and undeclared local variables, type inference, several exception >> raising and catching situations (e.g. raise in one block, catch in an outer >> block, try-finally, ...) or looping. It may also have an impact on Vitja's >> control flow analysis branch that's worth considering. > > I agree. I think I'll start a branch and work on some more tests. I also have to adjust the function teardown locking. I'll start a branch and report back if I get positive results. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
mark florisson wrote: I just think it is important for users to realize that in general, they cannot unblock threads recursively. I don't think that nested 'with nogil' blocks in any way suggest that they can. Saying 'with nogil' means "I want this code executed with the gil released". If it's already released, the logical meaning is to do nothing further. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/16/2011 01:55 PM, mark florisson wrote: but it doesn't catch this: cdef void func() nogil: with nogil: pass with nogil: func() I would say that in that case, func() should acquire the gil on entry so that it can be released in the 'with nogil' block. Either that or find some way to generate code that tests whether the gil is currently held and releases it if it is. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
Stefan Behnel wrote: That means that the average innocent future maintainer could break the code by manually rearranging the expression in a way that's perfectly valid mathematically, Another thing is that when the subexpressions concerned are function arguments, it only works if the order of the arguments happens to be the same as the required order of evaluation. Changes in the function signature can therefore lead to subtle breakage. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Stefan Behnel wrote: I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: > Robert Bradshaw, 11.03.2011 19:33: >> >> On Fri, Mar 11, 2011 at 9:36 AM, Stefan Behnel >> wrote: >>> >>> Stefan Behnel, 11.03.2011 15:08: Vitja Makarov, 11.03.2011 15:04: > > 2011/3/11 Stefan Behnel: >> >> Personally, I think it would be nice to keep up Python's semantics, >> but >> when >> I implemented this I broke quite some code in Sage (you may have >> noticed >> that the sage-build project in Hudson has been red for a while). There >> are >> things in C and especially in C++ that cannot be easily copied into a >> temporary variable in order to make sure they are evaluated before the >> following arguments. This is not a problem for Python function calls >> where >> all arguments end up being copied (and often converted) anyway. It is >> a >> problem for C function calls, though. > >> f(g(a), a.x, h(a)) > > Why could not this be translated into: > > tmp1 = g(a) > tmp2 = a.x > tmp3 = h(a) > > f(tmp1, tmp2, tmp3) See above. >>> >>> To be a little clearer here, it's a problem in C for example with struct >>> values. Copying them by value into a temp variable can be expensive, >>> potentially twice as expensive as simply passing them into the function >>> normally. >> >> Yep, and some types (e.g. array types) can't be assigned to at all. >> FWIW, the issues with Sage is that many libraries use the "stack >> allocated, pass-by-reference" trick >> >> typedef foo_struct foo_t[1] >> >> but we have declared them to be of type "void*" because we don't care >> about or want to muck with the internals. Cleaning this up is >> something we should do, but is taking a while, and aside from that it >> makes Cython even more dependent on correct type declarations (and >> backwards incompatible in this regard). Sage is a great test for the >> buildbot, so keeping it red for so long is not a good thing either. >> >>> Not sure what kind of additional devilry C++ provides here, but I'd >>> expect >>> that object values can exhibit bizarre behaviour when being assigned. >>> Maybe >>> others can enlighten me here. >> >> Yes, C++ allows overloading of the assignment operator, so assigning >> may lead to arbitrary code (as well as probably an expensive copy, as >> with structs, and structs in C++ are really just classes with a >> different visibility). >> >>> I have no idea how many cases there actually are that we can't handle or >>> that may lead to a performance degradation when using temps, but the >>> problem >>> is that they exist at all. >> >> Note that this applies not just to function arguments, but a host of >> other places, e.g. the order of evaluation of "f() + g()" in C is >> unspecified. Fleshing this out completely will lead to a lot more >> temps and verbose C code. And then we'd just cross our fingers that >> the C compiler was able to optimize all these temps away (though still >> possibly producing inferior code if it were able to switch order of >> execution). >> >> Whatever we do, we need a flag/directive. Perhaps there's a way to >> guarantee correct ordering for all valid Python code > > Plain Python code should never suffer from the original problem (with the > obvious exception of cpdef functions), but I also see how type inference can > affect this guarantee for the evaluation order in expressions. In particular, we need to be careful if we ever automatically cpdef a function (as part of a future optimization). > I agree with Greg, though, that it has a code smell if (multiple) > subexpressions in an expression have side-effects, especially when they are > impacted by the evaluation order. That means that the average innocent > future maintainer could break the code by manually rearranging the > expression in a way that's perfectly valid mathematically, e.g. as a > readability fix. > > >> even in the >> presence of (function and variable) type inference, but allow some >> leeway for explicitly declared C functions. > > I'm actually leaning towards not guaranteeing the order of execution if C > doesn't do it either. If this is really required, it's easy to work around > for users, but it's severely hard to fix for Cython in all cases, and the > gain is truly small. After all, we'd only make it easier for users to write > bad code. Yep. Lets keep the code in for the above case. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
