Re: [Cython] [cython-users] Calling gil-requiring function not allowed without gil
[moving this away from cython-users] Dag Sverre Seljebotn, 11.08.2011 14:53: On 08/11/2011 02:13 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 11.08.2011 13:58: On 08/11/2011 01:43 PM, Ting Zhou wrote: here is my pyx file. When I compile it, it says "Calling gil-requiring function not allowed without gil" at calling dabs(x[i]). Why my dabs function is a gil-requiring function? from cython.parallel import * import numpy as np cimport numpy as np cimport cython cdef double dabs(double x): This should be cdef double dabs(double x) nogil: Note that Cython cannot infer this automatically. Even a trivial function may require an exclusive global lock for some reason, and it's common to use the GIL for that. So the programmer must be explicit here. Are you still against this mini-CEP?: with cython.global_lock(): ... Where global_lock() is GIL-requiring noop. This a) Improves code readability vastly. Having a critical section take effect because of the *lack* of a keyword is just very odd to anyone who's not shoulder deep in CPython internals The GIL is more than a critical section. It's just there - and that's a good thing. Just take it as a part of the runtime environment, and disable it when you are sure you can *safely* do without it. A global lock makes life a lot easier, as it prevents unconditional (and unexpected) concurrency. b) Allows inferring 'nogil' No it doesn't, because existing code doesn't use it. Only because you use one little statement in one part of your sources doesn't mean you know what you're doing, and that Cython can happily assume reversed semantics for the rest. Don't forget that threading is a dangerous abstraction, hard to get right and likely even the most error prone concurrency mechanism in widespread use. Not everything is "trivially parallelisable", and even those cases that are still produce enough problems. Actually, it's worth asking if even the prange loop was a good idea, given the amount of confusion driven traffic that it currently produces on the mailing list. I'm not arguing against it, but it seems to be a trickier concept than it appears at first sight. IMHO, that's worth working on *before* we introduce even more hard-to-understand concepts and hard-to-control compiler trickery. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Calling gil-requiring function not allowed without gil
On 08/11/2011 04:00 PM, Stefan Behnel wrote: Actually, it's worth asking if even the prange loop was a good idea, given the amount of confusion driven traffic that it currently produces on the mailing list. I'm not arguing against it, but it seems to be a trickier concept than it appears at first sight. IMHO, that's worth working on *before* we introduce even more hard-to-understand concepts and hard-to-control compiler trickery. Honestly. Have you even read the posts regarding the prange statements? One was a missing DLL (similar in nature to "I can't find my Python.h"), and another tried to implement something which was fundamentally not parallizable using *any* conceivable framework for expressing any parallelism at all. Nobody has as far as I remember had a problem that wouldn't be a problem using multiprocessing or some such instead. I think you're very far from arguing rational in any sense about this, so I'll try my best to just shut up. (Sorry all, but the statement about the prange loop blew my fuses...) Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] What now?
Hi, now that I've nailed Cython code, I'd like to get into something more funny. Currently, I'm working on a set of macros to seamlessy integrate Cython into CMake build process (in fact, I love CMake). But, I'd like to work also on something more essential, so... Is there any updated TODO list? (ToDo.txt in git repo dates back to more than 8 months ago). I may also work on the wiki Cheers, --SKS ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] What now?
Stefano, 11.08.2011 16:24: I'd like to work also on something more essential, so... Is there any updated TODO list? (ToDo.txt in git repo dates back to more than 8 months ago). The most up-to-date list of things to do is the bug tracker. http://trac.cython.org/cython_trac/report/1?asc=0&sort=ticket It contains a broad list of bugs and enhancements, anything from small issues to larger language features. Just give it a look and see if you find something that sounds interesting to you. You can also use the query page to select by components, such as potential optimisations, Python compatibility issues or Cython specific language features. http://trac.cython.org/cython_trac/query Note that some tickets may be incomprehensible without some background knowledge, but most of them should be ok. If you have any questions or if you find something that you want to tackle, please ask on this list about it. We'll try to get you going. There's also the enhancement proposal list with some bigger topics, but most of them haven't been decided upon (just some wild ideas) and even the acceptable ones may be too large to get started. Also, the page is somewhat outdated in corners. http://wiki.cython.org/enhancements Hope that helps, Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] What now?
On 08/11/2011 04:24 PM, Stefano wrote: Hi, now that I've nailed Cython code, I'd like to get into something more funny. Currently, I'm working on a set of macros to seamlessy integrate Cython into CMake build process (in fact, I love CMake). But, I'd like to work also on something more essential, so... Is there any updated TODO list? (ToDo.txt in git repo dates back to more than 8 months ago). I may also work on the wiki In addition to Stefan's suggestions, here's a small mini-CEP: Error testcases in tests/errors are currently rather inconvenient, since changing anything often shifts a lot of error message line numbers. Something like this format might be better, where the line number is implied from the position of the comment: # mode: error with nogil: print 1 #ERROR:4:print statement not allowed in nogil block I just filed this as http://trac.cython.org/cython_trac/ticket/710 Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] What now?
Thursday 11th August 2011 19:51:14, Dag Sverre Seljebotn wrote: > Error testcases in tests/errors are currently rather inconvenient, since > changing anything often shifts a lot of error message line numbers. > Something like this format might be better, where the line number is > implied from the position of the comment: > > # mode: error > > with nogil: > print 1 #ERROR:4:print statement not allowed in nogil block > > I just filed this as http://trac.cython.org/cython_trac/ticket/710 That's a good point. I've started working on it. Actually, runtests.py should be split into a module and a public run-main stub, to make it load faster. I'll try to cope with this too. --SKS ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] What now?
On Thu, Aug 11, 2011 at 1:45 PM, Stefano wrote: > Thursday 11th August 2011 19:51:14, Dag Sverre Seljebotn wrote: >> Error testcases in tests/errors are currently rather inconvenient, since >> changing anything often shifts a lot of error message line numbers. >> Something like this format might be better, where the line number is >> implied from the position of the comment: >> >> # mode: error >> >> with nogil: >> print 1 #ERROR:4:print statement not allowed in nogil block >> >> I just filed this as http://trac.cython.org/cython_trac/ticket/710 > > That's a good point. I've started working on it. That would be really nice. > Actually, runtests.py should > be split into a module and a public run-main stub, to make it load faster. > I'll try to cope with this too. I don't think this is really a concern. In terms of the enhancements wiki, it's always a good idea to look at the history to get an idea of how current/out of date a given CEP is. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython bug ?
Hi I tried to compiled Demos/primes.pyx using the ctypes backend and I think I've found a bug : The Entry for the kmax parameter does not set is_arg to 1. However if I turn the def into a cdef, it works fine. Is this a bug or a known hack ? Thanks Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython bug ? - Entry.is_arg flag not set for all arguments
[fixed subject] Romain Guillebert, 12.08.2011 03:19: I tried to compiled Demos/primes.pyx using the ctypes backend and I think I've found a bug : The Entry for the kmax parameter does not set is_arg to 1. However if I turn the def into a cdef, it works fine. Is this a bug or a known hack ? Vitja already brought this up, too: http://article.gmane.org/gmane.comp.python.cython.devel/12385 A quick grep on the sources gives me this: """ Cython/Compiler/Buffer.py:if entry.is_arg: Cython/Compiler/Buffer.py:if entry.is_arg: Cython/Compiler/ExprNodes.py:return entry and (entry.is_local or entry.is_arg) and not entry.in_closure Cython/Compiler/FlowControl.py:return (entry.is_local or entry.is_pyclass_attr or entry.is_arg or Cython/Compiler/FlowControl.py:self.is_arg = False Cython/Compiler/FlowControl.py:self.is_arg = True Cython/Compiler/FlowControl.py:if assmt.is_arg: Cython/Compiler/FlowControl.py:# TODO: starred args entries are not marked with is_arg flag Cython/Compiler/FlowControl.py:if assmt.is_arg: Cython/Compiler/FlowControl.py:is_arg = True Cython/Compiler/FlowControl.py:is_arg = False Cython/Compiler/FlowControl.py:if is_arg: Cython/Compiler/Symtab.py:# is_arg booleanIs the arg of a method Cython/Compiler/Symtab.py:is_arg = 0 Cython/Compiler/Symtab.py:entry.is_arg = 1 """ This doesn't look like it would be wrong (or even just unsafe) to consider the unset flag a bug and fix it. Basically, it's almost unused in the original sources, all places where the flag is being read currently are somewhat recent code that looks reasonable to me and that appear to assume that the flag is actually set. So, I'd say, if anyone wants to properly clean this up, please go ahead and do so, but please do it right on the master branch and send it through Jenkins. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython bug ? - Entry.is_arg flag not set for all arguments
2011/8/12 Stefan Behnel : > [fixed subject] > > Romain Guillebert, 12.08.2011 03:19: >> >> I tried to compiled Demos/primes.pyx using the ctypes backend and I >> think I've found a bug : >> The Entry for the kmax parameter does not set is_arg to 1. However if I >> turn the def into a cdef, it works fine. >> >> Is this a bug or a known hack ? > > Vitja already brought this up, too: > > http://article.gmane.org/gmane.comp.python.cython.devel/12385 > > A quick grep on the sources gives me this: > > """ > Cython/Compiler/Buffer.py: if entry.is_arg: > Cython/Compiler/Buffer.py: if entry.is_arg: > Cython/Compiler/ExprNodes.py: return entry and (entry.is_local or > entry.is_arg) and not entry.in_closure > Cython/Compiler/FlowControl.py: return (entry.is_local or > entry.is_pyclass_attr or entry.is_arg or > Cython/Compiler/FlowControl.py: self.is_arg = False > Cython/Compiler/FlowControl.py: self.is_arg = True > Cython/Compiler/FlowControl.py: if assmt.is_arg: > Cython/Compiler/FlowControl.py: # TODO: starred args entries are > not marked with is_arg flag > Cython/Compiler/FlowControl.py: if assmt.is_arg: > Cython/Compiler/FlowControl.py: is_arg = True > Cython/Compiler/FlowControl.py: is_arg = False > Cython/Compiler/FlowControl.py: if is_arg: > Cython/Compiler/Symtab.py: # is_arg boolean Is the arg of a > method > Cython/Compiler/Symtab.py: is_arg = 0 > Cython/Compiler/Symtab.py: entry.is_arg = 1 > """ > > This doesn't look like it would be wrong (or even just unsafe) to consider > the unset flag a bug and fix it. Basically, it's almost unused in the > original sources, all places where the flag is being read currently are > somewhat recent code that looks reasonable to me and that appear to assume > that the flag is actually set. > > So, I'd say, if anyone wants to properly clean this up, please go ahead and > do so, but please do it right on the master branch and send it through > Jenkins. > Yeah, that would be really nice if all args including starred ones will have is_arg attribute set. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Yet another Python to C compiler
Hi! Recently I've found one more Python to C compiler, that translates python bytecode into C source. And author says about 100% Python compatibility. The project is a signle 800KB python file. http://code.google.com/p/2c-python/ I was wondering when found that 2c beats Cython in some benchmarks. For instance, it's about 2 times faster than Cython in pystone test (pure python mode). Think we should investigate performance differences and make cython faster. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
