Re: [Cython] Scoped expressions and generators problem
Vitja Makarov, 19.05.2011 22:34: I don't know how to handle scopedexprs in generators, here are some examples: [(yield i) for i in l] or [i for i in yield] In Python3 language mode scoped expression is created for list comprehension and loop variable is moved there. Not only that. In Python 3 (and also for set/dict comprehensions in Py2.7), the above are basically generator expressions and the behaviour of yield inside of a generator expression is weird at best. >>> a = [(yield i) for i in (1,2,3)] >>> a at 0x1be2a00> >>> next(a) 1 >>> a.send(5) 2 >>> next(a) 3 >>> next(a) [5, None, None] The second case is simpler and more obvious as the yield expression only determines the iterable, which happens before entering the loop. Note, however, that the iterable is currently being evaluated inside of the wrong scope. http://trac.cython.org/cython_trac/ticket/600 So now it isn't stored inside generator closure and is lost between yields. Right, it's a separate scope. I guess this means that expression scopes must behave differently when one of the surrounding scopes is a generator scope. They either have to somehow declare their local names in that outer scope, or the closure class generator would have to descend into these inner scopes as well to inject additional names into the closure, or we could let the yield expression node explicitly copy over local names from surrounding non-generator scopes into temps. The current workings of declaring a block local C variable would match best with the third way IMHO, even if that's not the cleanest solution. Otherwise, we'd also have to change the way scoped expressions work. Btw there is one more problem I hope that's easy to solve: yield expression inside scoped expression is counted twice. That means that the YieldNodeCollector must stop at scoped expression boundaries. However, as ticket #600 shows, this may not be completely trivial to fix. It might work to let the scoped expression nodes own the iterable node, and to use a CloneNode in the place where the iterable is used inside of the loop. That way, the scope node can directly decide which scope to evaluate the iterable node in. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] nonecheck directive
Hi, why is the "nonecheck" directive set to False by default? Shouldn't it rather be a "I know what I'm doing" option that allows advanced users to trade speed for safety? The reason I'm asking is that I just enabled its evaluation in NoneCheckNode and immediately got crashes in the test suite. So its currently only half-heartedly safe because it's not being evaluated in a lot of places. That's a rather fragile situation, not only for refactorings. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] nonecheck directive
On Fri, May 20, 2011 at 8:13 AM, Stefan Behnel wrote: > Hi, > > why is the "nonecheck" directive set to False by default? Shouldn't it > rather be a "I know what I'm doing" option that allows advanced users to > trade speed for safety? > > The reason I'm asking is that I just enabled its evaluation in NoneCheckNode > and immediately got crashes in the test suite. So its currently only > half-heartedly safe because it's not being evaluated in a lot of places. > That's a rather fragile situation, not only for refactorings. The reasoning was that we didn't want to have a major performance regression on existing code has already been written knowing these semantics, and also that we eventually plan to solve this more gracefully using control flow. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] nonecheck directive
Robert Bradshaw, 20.05.2011 17:33: On Fri, May 20, 2011 at 8:13 AM, Stefan Behnel wrote: why is the "nonecheck" directive set to False by default? Shouldn't it rather be a "I know what I'm doing" option that allows advanced users to trade speed for safety? Erm, trade safety for speed, obviously ... The reason I'm asking is that I just enabled its evaluation in NoneCheckNode and immediately got crashes in the test suite. So its currently only half-heartedly safe because it's not being evaluated in a lot of places. That's a rather fragile situation, not only for refactorings. The reasoning was that we didn't want to have a major performance regression on existing code has already been written knowing these semantics, and also that we eventually plan to solve this more gracefully using control flow. I can see that there could have been a slight, potential performance regression due to additional None checks, even considering that the C compiler can often drop many of them due to its own control flow analysis, and even though the CPU's branch prediction can be expected to handle this quite well even in loops. However, for users, it's hard to predict where Cython can avoid None checks and where it cannot, so having to explicitly tell it to do None checks in a specific code section means that users encounter and analyse a crash first, potentially when switching to a newer Cython version. The opt-out way would have allowed them to disable it only for code sections where it is really getting in the way, and would have made it clear in their own code that something potentially unsafe is happening where they are on their own. I think that even in the face of future control flow analysis in Cython, it would still have been better to make it an opt-out rather than opt-in option, but I would expect that we can still switch the default setting when a suitable CFA step becomes available. In the future, I think we should be more careful with potentially harmful options, and always prefer safety over speed - *especially* when we know that the safe way will improve at some point. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
