Re: [Cython] 'with gil:' statement
On 03/17/2011 12:24 AM, Greg Ewing wrote: Stefan Behnel wrote: I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. -1. I consider the fact that exceptions don't propagate from some functions a "currently unfixable bug". We should plan for it being fixed some day. Having a "with" statement alter execution flow in this way is totally unintuitive to me. If you want this, it's better to introduce a new keyword like "trywithgil: ... except:" (not that I'm in favour of that). We could perhaps fix exception propagation from nogil functions by using some conventions + setjmp/longjmp. Mono does this when calling into native code, and I recently did it manually in Cython to propagate exceptions through the Fortran wrappers in SciPy. Also, the GIL may not be around forever even in CPython? (All arguments I've seen for keeping it has been along the lines of "it slows down serial code", not that it is considered a good thing.) Designing a language around the GIL feels like a dead-end to me. I'm OK with being practical in the face of the limitations of today; but let's keep "with gil" and "with nogil" something that can become noops in the future without too much pain. Yes, I know that if the GIL goes it will break Stefan's lxml code, and I'm sure other code -- I'm just saying that we shouldn't make the language design even more GIL-centric than it already is. Anyway, I'm off to write some computational Fortran+OpenMP code because dealing with threading and Python is just more than I can deal with... Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/17/2011 08:38 AM, Dag Sverre Seljebotn wrote: On 03/17/2011 12:24 AM, Greg Ewing wrote: Stefan Behnel wrote: I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. -1. I consider the fact that exceptions don't propagate from some functions a "currently unfixable bug". We should plan for it being fixed some day. Having a "with" statement alter execution flow in this way is totally unintuitive to me. If you want this, it's better to introduce a new keyword like "trywithgil: ... except:" (not that I'm in favour of that). We could perhaps fix exception propagation from nogil functions by using some conventions + setjmp/longjmp. Mono does this when calling into native code, and I recently did it manually in Cython to propagate exceptions through the Fortran wrappers in SciPy. Also, the GIL may not be around forever even in CPython? (All arguments I've seen for keeping it has been along the lines of "it slows down serial code", not that it is considered a good thing.) Heh. I obviously meant that "removing it would slow down serial code". Dag Designing a language around the GIL feels like a dead-end to me. I'm OK with being practical in the face of the limitations of today; but let's keep "with gil" and "with nogil" something that can become noops in the future without too much pain. Yes, I know that if the GIL goes it will break Stefan's lxml code, and I'm sure other code -- I'm just saying that we shouldn't make the language design even more GIL-centric than it already is. Anyway, I'm off to write some computational Fortran+OpenMP code because dealing with threading and Python is just more than I can deal with... Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Dag Sverre Seljebotn, 17.03.2011 08:38: On 03/17/2011 12:24 AM, Greg Ewing wrote: Stefan Behnel wrote: I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. -1. I consider the fact that exceptions don't propagate from some functions a "currently unfixable bug". We should plan for it being fixed some day. It can't be fixed in general, because there are cases where exceptions simply cannot be propagated. Think of C callbacks, for example. C doesn't have a "normal" way of dealing with exceptions, so if an exception that originated from a callback simply leads to returning from the function, it may mean that the outer C code will simply continue to execute normally. Nothing's won in that case. In code: cdef void c_callback(...) nogil: ... do some C stuff ... with gil: ... do some Python stuff ... ... do some more C stuff ... So far, there are two proposed ways of doing this. 1) acquire the GIL on entry and exit, handling unraisable exceptions right before exiting. 2) keep all GIL requiring code inside of the "with gil" block, including unraisable exceptions. I find (2) a *lot* more intuitive, as well as much safer. We can't know what effects the surrounding "do C stuff" code has. It may contain thread-safe C level cleanup code for the "with gil" block, for example, or preparation code that enables returning into the calling C code. Simply jumping out of the GIL block without executing the trailing code may simply not work at all. We could perhaps fix exception propagation from nogil functions by using some conventions + setjmp/longjmp. Mono does this when calling into native code, and I recently did it manually in Cython to propagate exceptions through the Fortran wrappers in SciPy. Regardless of the topic of this thread, it would be nice to have longjmp support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work around several quirks in that area. Also, the GIL may not be around forever even in CPython? (All arguments I've seen for keeping it has been along the lines of "it slows down serial code", not that it is considered a good thing.) If it ever gets removed, there will surely have to be an emulation layer for C modules. Many of them simply use it as thread-lock, and that's totally reasonable IMHO. Designing a language around the GIL feels like a dead-end to me. We keep having diverging opinions about the GIL. I like it, and I keep repeating myself by saying that "threading should be explicit". Having a way to lock the whole interpreter and to keep parallel execution and reentry points to well defined places in your code is a great feature. I'm OK with being practical in the face of the limitations of today; but let's keep "with gil" and "with nogil" something that can become noops in the future without too much pain. Yes, I know that if the GIL goes it will break Stefan's lxml code, and I'm sure other code -- I'm just saying that we shouldn't make the language design even more GIL-centric than it already is. It's not. Even a removal of the GIL won't remove the fact that C can't propagate exceptions. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/17/2011 09:27 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 17.03.2011 08:38: On 03/17/2011 12:24 AM, Greg Ewing wrote: Stefan Behnel wrote: I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. -1. I consider the fact that exceptions don't propagate from some functions a "currently unfixable bug". We should plan for it being fixed some day. It can't be fixed in general, because there are cases where exceptions simply cannot be propagated. Think of C callbacks, for example. C doesn't have a "normal" way of dealing with exceptions, so if an exception that originated from a callback simply leads to returning from the function, it may mean that the outer C code will simply continue to execute normally. Nothing's won in that case. Yes, that's a good point. (This is what I used setjmp/longjmp to work around BTW, to longjmp across the calling Fortran code. I knew it wasn't doing any mallocs/frees, let alone any file handling etc., so this was safe.) I'll admit that I'm mostly focused on code like def f(): with nogil: for ...: A if something_exceptional: with gil: raise Exception(...) B C where I'd say it's up to me to make sure that B and C can safely be skipped. It would be a major pain to have my raised exception here be "trapped" -- in fact, it would make the "with gil" statement unusable for my purposes. In code: cdef void c_callback(...) nogil: ... do some C stuff ... with gil: ... do some Python stuff ... ... do some more C stuff ... So far, there are two proposed ways of doing this. 1) acquire the GIL on entry and exit, handling unraisable exceptions right before exiting. 2) keep all GIL requiring code inside of the "with gil" block, including unraisable exceptions. I find (2) a *lot* more intuitive, as well as much safer. We can't know what effects the surrounding "do C stuff" code has. It may contain thread-safe C level cleanup code for the "with gil" block, for example, or preparation code that enables returning into the calling C code. Simply jumping out of the GIL block without executing the trailing code may simply not work at all. I think you find (2) more intuitive because you have a very detailed knowledge of Cython and CPython, but that somebody new to Cython would expect a "with" statement to have the same control flow logic as the Python with statement. Of course, I don't have any data for that. How about this compromise: We balk on the code you wrote with: Error line 345: Exceptions propagating from "with gil" block cannot be propagated out of function, please insert try/except and handle exception So that we require this: with gil: try: ... except: warnings.warning(...) # or even cython.unraisable(e) This keeps me happy about not abusing the with statement for strange control flow, and makes the "with gil" useful for raising exceptions inside regular def functions with nogil blocks. We could perhaps fix exception propagation from nogil functions by using some conventions + setjmp/longjmp. Mono does this when calling into native code, and I recently did it manually in Cython to propagate exceptions through the Fortran wrappers in SciPy. Regardless of the topic of this thread, it would be nice to have longjmp support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work around several quirks in that area. Not sure what you mean here, I used longjmp (in a function without any Python objects) and it seems to work just fine. Did I miss anything? Also, the GIL may not be around forever even in CPython? (All arguments I've seen for keeping it has been along the lines of "it slows down serial code", not that it is considered a good thing.) If it ever gets removed, there will surely have to be an emulation layer for C modules. Many of them simply use it as thread-lock, and that's totally reasonable IMHO. Good point. But there may be an option to disable said emulation layer that we want to make use of in Cython... (This is relevant today for Cython-on-.NET, for instance.) Designing a language around the GIL feels like a dead-end to me. We keep having diverging opinions about the GIL. I like it, and I keep repeating myself by saying that "threading should be explicit". Having a way to lock the whole interpreter and to keep parallel execution and reent
Re: [Cython] 'with gil:' statement
On 17 March 2011 10:08, Dag Sverre Seljebotn wrote: > On 03/17/2011 09:27 AM, Stefan Behnel wrote: >> >> Dag Sverre Seljebotn, 17.03.2011 08:38: >>> >>> On 03/17/2011 12:24 AM, Greg Ewing wrote: Stefan Behnel wrote: > I'm not sure if this is a good idea. "nogil" blocks don't have a way to > handle exceptions, so simply jumping out of them because an inner 'with > gil' block raised an exception can have unexpected side effects. Seems to me that the __Pyx_WriteUnraisable should be done at the end of the 'with gil' block, and execution should then continue from there. In other words, the effect on exception handling should be the same as if the 'with gil' block had been factored out into a separate function having no exception return value. >>> >>> -1. >>> >>> I consider the fact that exceptions don't propagate from some functions a >>> "currently unfixable bug". We should plan for it being fixed some day. >> >> It can't be fixed in general, because there are cases where exceptions >> simply cannot be propagated. Think of C callbacks, for example. C doesn't >> have a "normal" way of dealing with exceptions, so if an exception that >> originated from a callback simply leads to returning from the function, it >> may mean that the outer C code will simply continue to execute normally. >> Nothing's won in that case. > > Yes, that's a good point. (This is what I used setjmp/longjmp to work around > BTW, to longjmp across the calling Fortran code. I knew it wasn't doing any > mallocs/frees, let alone any file handling etc., so this was safe.) > > I'll admit that I'm mostly focused on code like > > def f(): > with nogil: > for ...: > A > if something_exceptional: > with gil: > raise Exception(...) > B > C > > where I'd say it's up to me to make sure that B and C can safely be skipped. > It would be a major pain to have my raised exception here be "trapped" -- in > fact, it would make the "with gil" statement unusable for my purposes. > > > >> >> In code: >> >> cdef void c_callback(...) nogil: >> ... do some C stuff ... >> with gil: >> ... do some Python stuff ... >> ... do some more C stuff ... >> >> So far, there are two proposed ways of doing this. >> >> 1) acquire the GIL on entry and exit, handling unraisable exceptions right >> before exiting. >> >> 2) keep all GIL requiring code inside of the "with gil" block, including >> unraisable exceptions. >> >> I find (2) a *lot* more intuitive, as well as much safer. We can't know >> what effects the surrounding "do C stuff" code has. It may contain >> thread-safe C level cleanup code for the "with gil" block, for example, or >> preparation code that enables returning into the calling C code. Simply >> jumping out of the GIL block without executing the trailing code may simply >> not work at all. > > I think you find (2) more intuitive because you have a very detailed > knowledge of Cython and CPython, but that somebody new to Cython would > expect a "with" statement to have the same control flow logic as the Python > with statement. Of course, I don't have any data for that. > > How about this compromise: We balk on the code you wrote with: > > Error line 345: Exceptions propagating from "with gil" block cannot be > propagated out of function, please insert try/except and handle exception > > So that we require this: > > with gil: > try: > ... > except: > warnings.warning(...) # or even cython.unraisable(e) > > This keeps me happy about not abusing the with statement for strange control > flow, and makes the "with gil" useful for raising exceptions inside regular > def functions with nogil blocks. > I agree with your previous statement, but not with your compromise :). We have to differentiate between two cases, similar to Stefan's cases, but different in a very important way that matter for nested GIL blocks. 1) Exceptions can propagate to some outer GIL section (in or outside the current function) 2) Exceptions can't propagate, because there is no outer GIL section and the function has a non-object return type With your compromise, with 1) exceptions cannot propagate, but with 2) you win forcing the user to be explicit. But then you still need to write to some variable indicating that an exception occurred and adjust control flow accordingly in your nogil section (unless you want to clean up and return immediately). If you have Python with-statement semantics, you can do the following, for instance: cdef void func() nogil: with gil: try: with nogil: with gil: code that may raise an exception this is not executed except ExceptionRaisedFromInnerWithGilBlock: handle exception here The point is, if you have case 2), and you want to use GIL code, you need to h
Re: [Cython] 'with gil:' statement
On 17 March 2011 09:27, Stefan Behnel wrote: > Dag Sverre Seljebotn, 17.03.2011 08:38: >> >> On 03/17/2011 12:24 AM, Greg Ewing wrote: >>> >>> Stefan Behnel wrote: >>> I'm not sure if this is a good idea. "nogil" blocks don't have a way to handle exceptions, so simply jumping out of them because an inner 'with gil' block raised an exception can have unexpected side effects. >>> >>> Seems to me that the __Pyx_WriteUnraisable should be done at >>> the end of the 'with gil' block, and execution should then >>> continue from there. >>> >>> In other words, the effect on exception handling should be >>> the same as if the 'with gil' block had been factored out into >>> a separate function having no exception return value. >>> >> >> -1. >> >> I consider the fact that exceptions don't propagate from some functions a >> "currently unfixable bug". We should plan for it being fixed some day. > > It can't be fixed in general, because there are cases where exceptions > simply cannot be propagated. Think of C callbacks, for example. C doesn't > have a "normal" way of dealing with exceptions, so if an exception that > originated from a callback simply leads to returning from the function, it > may mean that the outer C code will simply continue to execute normally. > Nothing's won in that case. > > In code: > > cdef void c_callback(...) nogil: > ... do some C stuff ... > with gil: > ... do some Python stuff ... > ... do some more C stuff ... > > So far, there are two proposed ways of doing this. > > 1) acquire the GIL on entry and exit, handling unraisable exceptions right > before exiting. > > 2) keep all GIL requiring code inside of the "with gil" block, including > unraisable exceptions. > > I find (2) a *lot* more intuitive, as well as much safer. We can't know what > effects the surrounding "do C stuff" code has. It may contain thread-safe C > level cleanup code for the "with gil" block, for example, or preparation > code that enables returning into the calling C code. Simply jumping out of > the GIL block without executing the trailing code may simply not work at > all. Which is exactly why users will have to handle exceptions if they want their C code to execute. > >> We could perhaps fix exception propagation from nogil functions by using >> some conventions + setjmp/longjmp. Mono does this when calling into native >> code, and I recently did it manually in Cython to propagate exceptions >> through the Fortran wrappers in SciPy. > > Regardless of the topic of this thread, it would be nice to have longjmp > support in Cython. Lupa, my Cython wrapper for LuaJIT, currently has to work > around several quirks in that area. > > >> Also, the GIL may not be around >> forever even in CPython? (All arguments I've seen for keeping it has been >> along the lines of "it slows down serial code", not that it is considered >> a >> good thing.) > > If it ever gets removed, there will surely have to be an emulation layer for > C modules. Many of them simply use it as thread-lock, and that's totally > reasonable IMHO. > > >> Designing a language around the GIL feels like a dead-end to me. > > We keep having diverging opinions about the GIL. I like it, and I keep > repeating myself by saying that "threading should be explicit". Having a way > to lock the whole interpreter and to keep parallel execution and reentry > points to well defined places in your code is a great feature. > > >> I'm OK >> with being practical in the face of the limitations of today; but let's >> keep "with gil" and "with nogil" something that can become noops in the >> future without too much pain. Yes, I know that if the GIL goes it will >> break Stefan's lxml code, and I'm sure other code -- I'm just saying that >> we shouldn't make the language design even more GIL-centric than it >> already >> is. > > It's not. Even a removal of the GIL won't remove the fact that C can't > propagate exceptions. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 03/17/2011 11:16 AM, mark florisson wrote: On 17 March 2011 10:08, Dag Sverre Seljebotn wrote: How about this compromise: We balk on the code you wrote with: Error line 345: Exceptions propagating from "with gil" block cannot be propagated out of function, please insert try/except and handle exception So that we require this: with gil: try: ... except: warnings.warning(...) # or even cython.unraisable(e) This keeps me happy about not abusing the with statement for strange control flow, and makes the "with gil" useful for raising exceptions inside regular def functions with nogil blocks. I agree with your previous statement, but not with your compromise :). We have to differentiate between two cases, similar to Stefan's cases, but different in a very important way that matter for nested GIL blocks. 1) Exceptions can propagate to some outer GIL section (in or outside the current function) 2) Exceptions can't propagate, because there is no outer GIL section and the function has a non-object return type With your compromise, with 1) exceptions cannot propagate, but with 2) you win forcing the user to be explicit. But then you still need to write to some variable indicating that an exception occurred and adjust control flow accordingly in your nogil section (unless you want to clean up and return immediately). If you have Python with-statement semantics, you can do the following, for instance: cdef void func() nogil: with gil: try: with nogil: with gil: code that may raise an exception this is not executed except ExceptionRaisedFromInnerWithGilBlock: handle exception here The point is, if you have case 2), and you want to use GIL code, you need to handle exceptions in some way. Forcing the user to not propagate anything doesn't sound right, unless this holds only for the outermost 'with gil' block. I would be OK with that, although it would be inconsistent with how exceptions in normal cdef functions with non-object return work, so I would say that we'd have to force it in the same manner there. I think we should perhaps look at enforcing explicit exception-ignoring everywhere.. there's a lot of details to hash out, and there's the issue of backwards compatability, but it could be dealt with with a couple of releases where we only raise a warning and so on. It could involve a *very* limited subset of exception handling for use in nogil mode (i.e., only a bare "except:" statement allowed, where one can call either "cython.unraisable()", "pass", or set a flag). Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 17 March 2011 11:35, Dag Sverre Seljebotn wrote: > On 03/17/2011 11:16 AM, mark florisson wrote: >> >> On 17 March 2011 10:08, Dag Sverre Seljebotn >> wrote: >>> >>> How about this compromise: We balk on the code you wrote with: >>> >>> Error line 345: Exceptions propagating from "with gil" block cannot be >>> propagated out of function, please insert try/except and handle exception >>> >>> So that we require this: >>> >>> with gil: >>> try: >>> ... >>> except: >>> warnings.warning(...) # or even cython.unraisable(e) >>> >>> This keeps me happy about not abusing the with statement for strange >>> control >>> flow, and makes the "with gil" useful for raising exceptions inside >>> regular >>> def functions with nogil blocks. >>> >> I agree with your previous statement, but not with your compromise :). >> We have to differentiate between two cases, similar to Stefan's cases, >> but different in a very important way that matter for nested GIL >> blocks. >> >> 1) Exceptions can propagate to some outer GIL section (in or outside >> the current function) >> 2) Exceptions can't propagate, because there is no outer GIL section >> and the function has a non-object return type >> >> With your compromise, with 1) exceptions cannot propagate, but with 2) >> you win forcing the user to be explicit. But then you still need to >> write to some variable indicating that an exception occurred and >> adjust control flow accordingly in your nogil section (unless you want >> to clean up and return immediately). >> >> If you have Python with-statement semantics, you can do the following, >> for instance: >> >> cdef void func() nogil: >> with gil: >> try: >> >> with nogil: >> with gil: >> code that may raise an exception >> >> this is not executed >> >> except ExceptionRaisedFromInnerWithGilBlock: >> handle exception here >> >> The point is, if you have case 2), and you want to use GIL code, you >> need to handle exceptions in some way. Forcing the user to not >> propagate anything doesn't sound right, unless this holds only for the >> outermost 'with gil' block. I would be OK with that, although it would >> be inconsistent with how exceptions in normal cdef functions with >> non-object return work, so I would say that we'd have to force it in >> the same manner there. > > I think we should perhaps look at enforcing explicit exception-ignoring > everywhere.. there's a lot of details to hash out, and there's the issue of > backwards compatability, but it could be dealt with with a couple of > releases where we only raise a warning and so on. > > It could involve a *very* limited subset of exception handling for use in > nogil mode (i.e., only a bare "except:" statement allowed, where one can > call either "cython.unraisable()", "pass", or set a flag). I don't think we should allow the forced exception handling in nogil sections, but in gil sections (and you'd have to hold the GIL as exceptions are stored on the thread state). So forced exception handling for functions declared 'with gil' and for outermost 'with gil' blocks in cdef functions with non-object return. > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On 3/16/11 11:05 PM, Robert Bradshaw wrote: On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: I'm actually leaning towards not guaranteeing the order of execution if C doesn't do it either. If this is really required, it's easy to work around for users, but it's severely hard to fix for Cython in all cases, and the gain is truly small. After all, we'd only make it easier for users to write bad code. Yep. Lets keep the code in for the above case. Is there a huge big warning in the docs? Maybe on this page would be a good place: http://docs.cython.org/src/userguide/limitations.html Jason ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout wrote: > On 3/16/11 11:05 PM, Robert Bradshaw wrote: >> >> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >> wrote: >>> >>> I'm actually leaning towards not guaranteeing the order of execution if C >>> doesn't do it either. If this is really required, it's easy to work >>> around >>> for users, but it's severely hard to fix for Cython in all cases, and the >>> gain is truly small. After all, we'd only make it easier for users to >>> write >>> bad code. >> >> Yep. Lets keep the code in for the above case. > > > Is there a huge big warning in the docs? Maybe on this page would be a good > place: http://docs.cython.org/src/userguide/limitations.html This doesn't affect Python functions at all, so I'm not sure it belongs on that page. I agree that it should be mentioned that c(p)def functions have C calling semantics, *including* an unspecified order or argument evaluation. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
2011/3/17 Robert Bradshaw : > On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout > wrote: >> On 3/16/11 11:05 PM, Robert Bradshaw wrote: >>> >>> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel >>> wrote: I'm actually leaning towards not guaranteeing the order of execution if C doesn't do it either. If this is really required, it's easy to work around for users, but it's severely hard to fix for Cython in all cases, and the gain is truly small. After all, we'd only make it easier for users to write bad code. >>> >>> Yep. Lets keep the code in for the above case. >> >> >> Is there a huge big warning in the docs? Maybe on this page would be a good >> place: http://docs.cython.org/src/userguide/limitations.html > > This doesn't affect Python functions at all, so I'm not sure it > belongs on that page. I agree that it should be mentioned that c(p)def > functions have C calling semantics, *including* an unspecified order > or argument evaluation. > As you noticed above, how should be handled def function inlining? I guess there are restrictions on def function argument type, so side-effect isn't issue here. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Thu, Mar 17, 2011 at 1:48 PM, Vitja Makarov wrote: > 2011/3/17 Robert Bradshaw : >> On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout >> wrote: >>> On 3/16/11 11:05 PM, Robert Bradshaw wrote: On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: > > I'm actually leaning towards not guaranteeing the order of execution if C > doesn't do it either. If this is really required, it's easy to work > around > for users, but it's severely hard to fix for Cython in all cases, and the > gain is truly small. After all, we'd only make it easier for users to > write > bad code. Yep. Lets keep the code in for the above case. >>> >>> >>> Is there a huge big warning in the docs? Maybe on this page would be a good >>> place: http://docs.cython.org/src/userguide/limitations.html >> >> This doesn't affect Python functions at all, so I'm not sure it >> belongs on that page. I agree that it should be mentioned that c(p)def >> functions have C calling semantics, *including* an unspecified order >> or argument evaluation. >> > > As you noticed above, how should be handled def function inlining? > > I guess there are restrictions on def function argument type, so > side-effect isn't issue here. Yep, or at least not near as much of an issue. (I think the C compiler can optimize away an, e.g, extra copy of, e.g, int, double and PyObject* temps in most cases.) - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
Robert Bradshaw, 17.03.2011 05:05: On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: I'm actually leaning towards not guaranteeing the order of execution if C doesn't do it either. If this is really required, it's easy to work around for users, but it's severely hard to fix for Cython in all cases, and the gain is truly small. After all, we'd only make it easier for users to write bad code. Yep. Lets keep the code in for the above case. Erm, the current status is that we try to guarantee the order by pushing everything into temps, thus breaking Sage. What code exactly did you intend to keep here? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
On Thu, Mar 17, 2011 at 2:25 PM, Stefan Behnel wrote: > Robert Bradshaw, 17.03.2011 05:05: >> >> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: >>> >>> I'm actually leaning towards not guaranteeing the order of execution if C >>> doesn't do it either. If this is really required, it's easy to work >>> around >>> for users, but it's severely hard to fix for Cython in all cases, and the >>> gain is truly small. After all, we'd only make it easier for users to >>> write >>> bad code. >> >> Yep. Lets keep the code in for the above case. > > Erm, the current status is that we try to guarantee the order by pushing > everything into temps, thus breaking Sage. What code exactly did you intend > to keep here? I was thinking about guarding it with an if False (or flag in Options.py). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)
Robert Bradshaw, 17.03.2011 22:26: On Thu, Mar 17, 2011 at 2:25 PM, Stefan Behnel wrote: Robert Bradshaw, 17.03.2011 05:05: On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel wrote: I'm actually leaning towards not guaranteeing the order of execution if C doesn't do it either. If this is really required, it's easy to work around for users, but it's severely hard to fix for Cython in all cases, and the gain is truly small. After all, we'd only make it easier for users to write bad code. Yep. Lets keep the code in for the above case. Erm, the current status is that we try to guarantee the order by pushing everything into temps, thus breaking Sage. What code exactly did you intend to keep here? I was thinking about guarding it with an if False (or flag in Options.py). Ah - the "poor man's VCS" approach? ;) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Dag Sverre Seljebotn wrote: def f(): with nogil: for ...: A if something_exceptional: with gil: raise Exception(...) B C If that's to be supported, the following really ought to be supported as well: def f(): with nogil: try: ... with gil: raise Exception() finally: ...do some cleanup... -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 18 March 2011 00:32, Greg Ewing wrote: > Dag Sverre Seljebotn wrote: > >> def f(): >> with nogil: >> for ...: >> A >> if something_exceptional: >> with gil: >> raise Exception(...) >> B >> C > > If that's to be supported, the following really ought to be > supported as well: > > def f(): > with nogil: > try: > ... > with gil: > raise Exception() > finally: > ...do some cleanup... > Why? I assume in his example the for loop was a Cython C for loop, not one that deals with Python objects. If you want to do cleanup you can catch the exception in the 'with gil:' block, or use try/finally in the 'with gil:' block or outside the 'with nogil:' block. Special-casing try/finally in nogil sections for only 'with gil:' sounds somewhat weird. On the other hand, it may be somewhat more convenient, and I think we could support it without having to acquire the GIL in the finally clause. I wouldn't be particularly opposed to that. > Greg > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
On 16 March 2011 20:16, mark florisson wrote: Could someone review the patch (which is attached)? Maybe check if I haven't missed any side cases and such? >>> >>> From a first look, the test file you added seems far too short. I would >>> expect that this feature requires a lot more testing in combination with >>> declared and undeclared local variables, type inference, several exception >>> raising and catching situations (e.g. raise in one block, catch in an outer >>> block, try-finally, ...) or looping. It may also have an impact on Vitja's >>> control flow analysis branch that's worth considering. >> >> I agree. I think I'll start a branch and work on some more tests. > > I also have to adjust the function teardown locking. I'll start a > branch and report back if I get positive results. > I added more tests, the code can be found in this fork: https://github.com/markflorisson88/cython . There is currently no compile-time checking for exceptions that might be swallowed, it still works in the same way as normal cdef functions with a non-object return type. Should we issue warnings for such cases instead of relying on the implicit swallow-and-print, as Dag suggested? ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
mark florisson wrote: I think we could support it without having to acquire the GIL in the finally clause. That was the intention -- the code in the finally clause would be subject to the same nogil restrictions as the rest of the nogil block. My point is that as long as you're allowing exceptions to be tunnelled through nogil blocks, they should respect any finally clauses that they pass through on the way. -- Greg ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 'with gil:' statement
Greg Ewing, 18.03.2011 01:18: mark florisson wrote: I think we could support it without having to acquire the GIL in the finally clause. That was the intention -- the code in the finally clause would be subject to the same nogil restrictions as the rest of the nogil block. My point is that as long as you're allowing exceptions to be tunnelled through nogil blocks, they should respect any finally clauses that they pass through on the way. +1 Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
