Re: [Cython] Yet another Python to C compiler
2011/8/12 Vitja Makarov : > Hi! > > Recently I've found one more Python to C compiler, that translates > python bytecode into C source. > And author says about 100% Python compatibility. The project is a > signle 800KB python file. > > http://code.google.com/p/2c-python/ > > I was wondering when found that 2c beats Cython in some benchmarks. > For instance, it's about 2 times faster than Cython in pystone test > (pure python mode). > > Think we should investigate performance differences and make cython faster. > Performance boost is because of --direct-call flag. With --no-direct-call flag performance is about the same. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Yet another Python to C compiler
Vitja Makarov, 12.08.2011 08:49: Recently I've found one more Python to C compiler, that translates python bytecode into C source. And author says about 100% Python compatibility. That's clearly incorrect. From the code, it appears that at least the builtins are static, which means that it's *not* Python compatible (even less so than Cython currently is). I'm also not sure about the integer type handling - worth some investigation, but it looks somewhat sloppy. I mean, it's well known that you can generate fast code by diverting from Python semantics. Shedskin excels in that. They mostly seem to be using the same tricks that Cython uses as well, namely static builtins, as well as optimistic optimisations for likely types based on usage patterns. And a huge amount of special casing, maybe even more than Cython currently applies. They also seem to make excessive use of CPython internals and internal APIs. Not sure if that's a good idea. Specifically, they didn't care a bit about Python 3 compatibility, it seems. The project is a signle 800KB python file. That's just plain sick. http://code.google.com/p/2c-python/ I was wondering when found that 2c beats Cython in some benchmarks. For instance, it's about 2 times faster than Cython in pystone test PyStone is known to be a particularly bad benchmark. The other benchmark results are somewhat surprising and (IMHO) hint mostly at a lack of Python compatibility. Again, it's well known that you can make specific benchmarks fast by diverting from Python semantics in general. For example, Cython runs richards.py ~70% faster, whereas they claim ~90%. Cython is ~50% faster on slowpickle, they claim ~80%. Not really that much of a difference actually, and easily achieved by tuning the language semantics to the code. Think we should investigate performance differences and make cython faster. You could start by contacting the authors. From the project site, it appears that it's basically Russian(?)-only. My guess is that they simply use more special casing and slightly better type inference than Cython currently does. Look at these, for example: http://code.google.com/p/2c-python/source/browse/2c.py?r=23d5c350a56e21d5a3e12e153d1fbe91ae1f5d56#15583 http://code.google.com/p/2c-python/source/browse/2c.py?r=23d5c350a56e21d5a3e12e153d1fbe91ae1f5d56#15869 They infer some more return types of builtin methods and their compiler knows about some stdlib modules (such as math): http://code.google.com/p/2c-python/source/browse/2c.py?r=23d5c350a56e21d5a3e12e153d1fbe91ae1f5d56#578 Overriding external modules statically means diverting from Python semantics. Cython would want to require user interaction for this, e.g. an explicit external .pxd file. Basically, I think that Cython could do a lot better with control flow driven type inference. Another thing is that it would be nice to extend the type system so that it knows about data types in Python containers. What we should definitely do is to use Mark's fused types for optimisations, e.g. when default arguments hint at a specific input type, or even just when we find a function call inside the module with a specific combination of input types. Also, I would expect that eventually optimising the CyFunction type would give us another bit of performance. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Calling gil-requiring function not allowed without gil
[second try in moving this discussion to cython-devel] Dag Sverre Seljebotn, 12.08.2011 08:50: On 08/12/2011 06:44 AM, Robert Bradshaw wrote: On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn wrote: On 08/11/2011 02:13 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 11.08.2011 13:58: On 08/11/2011 01:43 PM, Ting Zhou wrote: here is my pyx file. When I compile it, it says "Calling gil-requiring function not allowed without gil" at calling dabs(x[i]). Why my dabs function is a gil-requiring function? from cython.parallel import * import numpy as np cimport numpy as np cimport cython cdef double dabs(double x): This should be cdef double dabs(double x) nogil: Note that Cython cannot infer this automatically. Even a trivial function may require an exclusive global lock for some reason, and it's common to use the GIL for that. So the programmer must be explicit here. Are you still against this mini-CEP?: with cython.global_lock(): ... Where global_lock() is GIL-requiring noop. Just reading this, it's not immediately obvious what this means. (I thought at first this was different syntax for "with gil"...) True. "cython.synchronized"? The synchronized keyword in Java does not quite the same thing but almost. This a) Improves code readability vastly. Having a critical section take effect because of the *lack* of a keyword is just very odd to anyone who's not shoulder deep in CPython internals I'm not following you here. The only way to run into this is if you have explicitly release it. Presumably you can learn both keywords at the same time. Perhaps extern function could be nogil by default. I'll try to explain again. Consider code like this: cdef call_c(): magic_c_function(3) This code is perfectly fine even if magic_c_function is not reentrant -- but that's hardly well documented! So it's conceivable that another programmer comes along (who doesn't know the C library well) and decides that "somebody has just been too lazy to add the nogil specifier" and slaps it on -- at which point you have a bug. This is more to the point in creating readable code IMO: cdef call_c(): with cython.synchronized(): magic_c_function(3) I think I'm as confused as Robert here. Is that the GIL or some separate lock? If the latter, where would that lock be stored? Module-wide? While there are certainly use cases for that, I think it's much more common to either a) use the GIL or b) use an explicit lock at some well defined point, e.g. at an object or class level. I don't think we disagree when I say that locking should be explicit, but that includes the "thing" that keeps the lock during its lifetime. Also, consider your classical race conditions: cdef init(): global resource if resource == NULL: resource = malloc(sizeof(resource_t)) ... This is safe code. Well, it's safe code as long as there is no Python code interaction in between. Calling back into the interpreter may trigger a thread switch. Admittedly, this is not obvious and tends to introduce bugs. I learned that the hard way in lxml, actually twice. If your "synchronised" directive refers to the GIL, then it would suffer from that problem as well. I think that's very undesirable for an explicit critical section statement. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Calling gil-requiring function not allowed without gil
On 08/12/2011 02:45 PM, Stefan Behnel wrote: > [second try in moving this discussion to cython-devel] > > Dag Sverre Seljebotn, 12.08.2011 08:50: >> On 08/12/2011 06:44 AM, Robert Bradshaw wrote: >>> On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn >>> wrote: Are you still against this mini-CEP?: with cython.global_lock(): ... Where global_lock() is GIL-requiring noop. >>> >>> Just reading this, it's not immediately obvious what this means. (I >>> thought at first this was different syntax for "with gil"...) >> >> True. "cython.synchronized"? The synchronized keyword in Java does not >> quite the same thing but almost. >> This a) Improves code readability vastly. Having a critical section take effect because of the *lack* of a keyword is just very odd to anyone who's not shoulder deep in CPython internals >>> >>> I'm not following you here. The only way to run into this is if you >>> have explicitly release it. Presumably you can learn both keywords at >>> the same time. Perhaps extern function could be nogil by default. >> >> I'll try to explain again. Consider code like this: >> >> cdef call_c(): >> magic_c_function(3) >> >> This code is perfectly fine even if magic_c_function is not reentrant -- >> but that's hardly well documented! So it's conceivable that another >> programmer comes along (who doesn't know the C library well) and decides >> that "somebody has just been too lazy to add the nogil specifier" and >> slaps >> it on -- at which point you have a bug. >> >> This is more to the point in creating readable code IMO: >> >> cdef call_c(): >> with cython.synchronized(): >> magic_c_function(3) > > I think I'm as confused as Robert here. Is that the GIL or some separate > lock? > > If the latter, where would that lock be stored? Module-wide? While there > are certainly use cases for that, I think it's much more common to > either a) use the GIL or b) use an explicit lock at some well defined > point, e.g. at an object or class level. I intended it to be the GIL in current CPython. In GIL-less environments (whether Java, .NET or some future CPython) one would manuall ensure a fully global lock (inserting a _cython module in sys.path etc.). But: > I don't think we disagree when I say that locking should be explicit, > but that includes the "thing" that keeps the lock during its lifetime. > > >> Also, consider your classical race conditions: >> >> cdef init(): >> global resource >> if resource == NULL: >> resource = malloc(sizeof(resource_t)) ... >> >> This is safe code. > > Well, it's safe code as long as there is no Python code interaction in > between. Calling back into the interpreter may trigger a thread switch. > > Admittedly, this is not obvious and tends to introduce bugs. I learned > that the hard way in lxml, actually twice. > > If your "synchronised" directive refers to the GIL, then it would suffer > from that problem as well. I think that's very undesirable for an > explicit critical section statement. This is a good point. Hmm. This all started with your comment "Even a trivial function may require an exclusive global lock for some reason, and it's common to use the GIL for that. So the programmer must be explicit here." And my problem is that it's too implicit -- when you /require/ the global lock, you /leave out/ the nogil modifier. I just wanted to find some way about being explicit about what you already do. So it may be a question of naming. As you mention, using the GIL has drawbacks, but it performs better than a seperate lock when you know there won't be callbacks into Python... Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Vitja's CyFunction branch
2011/7/30 Stefan Behnel : > Robert Bradshaw, 30.07.2011 18:49: >> >> The only reason I haven't pushed a release branch is that last >> time I did that it kept getting the mainline development pulled into >> it > > That was just an accident on my side when I wasn't aware of the new branch > you had created. Won't happen again. Just open a new branch when you deem it > appropriate. > 0.15 is out. Is it time to merge CyFunction? ps: I'm afraid of merge button -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Vitja's CyFunction branch
Vitja Makarov, 13.08.2011 06:41: 2011/7/30 Stefan Behnel: Robert Bradshaw, 30.07.2011 18:49: The only reason I haven't pushed a release branch is that last time I did that it kept getting the mainline development pulled into it That was just an accident on my side when I wasn't aware of the new branch you had created. Won't happen again. Just open a new branch when you deem it appropriate. 0.15 is out. Is it time to merge CyFunction? ps: I'm afraid of merge button There were no objections to merging it, so I hit the button. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Vitja's CyFunction branch
2011/8/13 Stefan Behnel : > Vitja Makarov, 13.08.2011 06:41: >> >> 2011/7/30 Stefan Behnel: >>> >>> Robert Bradshaw, 30.07.2011 18:49: The only reason I haven't pushed a release branch is that last time I did that it kept getting the mainline development pulled into it >>> >>> That was just an accident on my side when I wasn't aware of the new >>> branch >>> you had created. Won't happen again. Just open a new branch when you deem >>> it >>> appropriate. >> >> 0.15 is out. Is it time to merge CyFunction? >> >> ps: I'm afraid of merge button > > There were no objections to merging it, so I hit the button. > Thanks! To be clear: we see increase in pyregr tests beacause of special hack in runtests.py that enables binding for pyregr. And of course it's disabled by default. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
