Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: > Hi, > > there seems to be quite some interest in a project to get parts of CPython > and specifically its stdlib rewritten in Cython. I've copied the latest > python-dev mail below. The relevant part of the thread is here: > > http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 Interesting. > In short, we have strong supporters, but Guido has understandable doubts > against a new (and quite large) dependency and potential semantic > deviations. Reading the list, I think others on the list overestimate the semantic differences. Mostly we're talking about things like is vs. equality for floating point numbers and tracebacks (at least for un-annotated code). It's a valid point that Cython is still under such active development. > But there seem to be cases where slight changes would be > acceptable that Cython compiled modules might introduce, such as emitting > different exception messages, changing Python classes into extension > classes, or even preventing monkey patching in modules that are backed by C > modules anyway. > > It would be helpful to get support from the side of external distributors > that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If > they agreed to test the Cython generated stdlib modules in their > distributions, we could get user feedback that would allow python-dev to > take a well founded decision. > > Do we have any volunteers for trying this out? Both on the side of > distributors and implementors? I think Sage might be willing to give it a try. I'll ask tomorrow as part of a talk I'm giving. Note that due to the way Python's import mechanism works, it would be easy (as a first pass) to make a "cythonize this Python install" which would just compile (a subset of) the .py files and drop .so files next to them. This would require no messing with the Python build system or distribution, easy to test and benchmark, and be easy to clean up. > At the current state of affairs, the implementation could still be financed > by a Python backed GSoC project, although it would be cool if more users > could just step up and simply try to compile and optimise stdlib modules > (preferably without major changes to the code). It's certainly a great way > to show off your Cython skills :). I gave it a try with difflib and it > turned out to be quite easy. > > http://blog.behnel.de/index.php?p=155 > > Reimplementing existing C modules in Cython might, however, be more > interesting for python-dev, but also be a larger undertaking. So a GSoC > might be worth running on that. I think that's a great idea. Would you be willing to mentor such a project. > Note that the latest Cython release does not have generator support yet, and > Vitja's branch on github is not very stable. We will try to get it up to > speed and merged during the workshop next week, at which point it will make > more sense to get this project started than right now. > > Stefan > > > Guido van Rossum, 22.03.2011 00:04: >> >> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. Löwis" wrote: >>> >>> Am 21.03.2011 11:58, schrieb Stefan Behnel: Guido van Rossum, 21.03.2011 03:46: > > Thanks for the clarifications. I now have a much better understanding > of what Cython is. But I'm not sold. For one, your attitude about > strict language compatibility worries me when it comes to the stdlib. Not sure what you mean exactly. Given our large user base, we do worry a lot about things like backwards compatibility, for example. If you are referring to compatibility with Python, I don't think anyone in the project really targets Cython as a a drop-in replacement for a Python runtime. We aim to compile Python code, yes, and there's a hand-wavy idea in the back of our head that we may want a plain Python compatibility mode at some point that will disable several important optimisations. >>> >>> I think that's the attitude Guido worries about: if you don't have the >>> desire to provide 100% Python compatibility under all circumstances >>> (i.e. including if someone passes parameters of "incorrect" types), >>> then there is very little chance that we would replace a Python module >>> with a Cython-compiled one. >>> >>> The only exception would be cases where the Python semantics is murky >>> (e.g. where Jython or so actually behaves differently for the same >>> Python code, and still claims language conformance). E.g. the exact >>> message on a TypeError might change when compiling with Cython, >>> but the cases in which you get a TypeError must not change. >> >> One other significant use case is the situation where we have an >> optional replacement module written in C (e.g. heapqmodule.c vs. >> heapq.py). There are usually many semantic differences between the C >> and pure-python module that we don't care about (e.g. monkeypatching >> won't work). >> >> The size of Cython as a de
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
Robert Bradshaw, 22.03.2011 08:14: On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: there seems to be quite some interest in a project to get parts of CPython and specifically its stdlib rewritten in Cython. [...] In short, we have strong supporters, but Guido has understandable doubts against a new (and quite large) dependency and potential semantic deviations. Reading the list, I think others on the list overestimate the semantic differences. Mostly we're talking about things like is vs. equality for floating point numbers and tracebacks (at least for un-annotated code). I think so too, that's what I tried to make clearer with my last reply. I think Cython is actually pretty close to Python semantics overall, and almost all deviations are explicitly triggered by type annotations in the code. It's a valid point that Cython is still under such active development. Absolutely. Eventually, we'd have to settle on a specific version for the compiler used in the stdlib, and support that at least as long as the CPython version that uses it. It would be helpful to get support from the side of external distributors that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If they agreed to test the Cython generated stdlib modules in their distributions, we could get user feedback that would allow python-dev to take a well founded decision. Do we have any volunteers for trying this out? Both on the side of distributors and implementors? I think Sage might be willing to give it a try. I'll ask tomorrow as part of a talk I'm giving. Cool. Note that due to the way Python's import mechanism works, it would be easy (as a first pass) to make a "cythonize this Python install" which would just compile (a subset of) the .py files and drop .so files next to them. This would require no messing with the Python build system or distribution, easy to test and benchmark, and be easy to clean up. Sure, I implemented that in pyximport ages ago, when Cython was really far from being able to compile much in the stdlib. We should totally give it a try once Vitja's branch is in shape. At the current state of affairs, the implementation could still be financed by a Python backed GSoC project, although it would be cool if more users could just step up and simply try to compile and optimise stdlib modules (preferably without major changes to the code). It's certainly a great way to show off your Cython skills :). I gave it a try with difflib and it turned out to be quite easy. http://blog.behnel.de/index.php?p=155 Reimplementing existing C modules in Cython might, however, be more interesting for python-dev, but also be a larger undertaking. So a GSoC might be worth running on that. I think that's a great idea. Would you be willing to mentor such a project. As usual, I'm not sure I'll have the time, but if no-one else steps up, I'd consider it. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PyCharm
On 3/16/11 5:33 AM, mark florisson wrote: The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the Cython project a free Open Source license, which means that anyone developing Cython may freely use PyCharm to develop Cython. They prefer to license to remain unpublic, so if you develop Cython and want a free PyCharm license, send me an email and I will send you the license key. It is valid until 13 March 2012. Is it the offer that is valid for a year, or is it the license that is valid for only a year? Thanks, Jason ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] cython crash: default parameters in methods of template cppclass
Hi,
I found a little bug in Cython 0.14.1.
The following causes the Cython compiler to throw an exception:
---8<---
cdef extern from "foo.h":
cdef cppclass foo[ T ]:
bar( int b = 0 )
cdef foo[ int ] a
a.bar( 1 )
---8<---
The exception is "AttributeError: 'CFuncType' object has no attribute
'op_arg_struct'", thrown in line 3044, in generate_result_code. Full
stack trace attached.
The file compiles without crash if one either
- removes the last line, i.e. the call to the class's method, or
- removes the template argument, i.e., the "[ T ]" in line 2 and
the "[ int ]" in line 5.
Hence, the issue only appears when calling methods of _templated_
classes, which have a _default_ value for their argument.
Simon
+---
| Dr. Simon Anders, Dipl.-Phys.
| European Molecular Biology Laboratory (EMBL), Heidelberg
| office phone +49-6221-387-8632
| preferred (permanent) e-mail: sand...@fs.tum.de
$ cython --cplus test2.pyx
Traceback (most recent call last):
File "/usr/local/bin/cython", line 9, in
load_entry_point('Cython==0.14.1', 'console_scripts', 'cython')()
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 789, in setuptools_main
return main(command_line = 1)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 806, in main
result = compile(sources, options)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 781, in compile
return compile_multiple(source, options)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 753, in compile_multiple
result = run_pipeline(source, options)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 617, in run_pipeline
err, enddata = context.run_pipeline(pipeline, source)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 244, in run_pipeline
data = phase(data)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Main.py",
line 166, in generate_pyx_code
module_node.process_implementation(options, result)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 73, in process_implementation
self.generate_c_code(env, options, result)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 293, in generate_c_code
self.generate_module_init_func(modules[:-1], env,
globalstate['init_module'])
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 1811, in generate_module_init_func
self.body.generate_execution_code(code)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 362, in generate_execution_code
stat.generate_execution_code(code)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 3361, in generate_execution_code
self.expr.generate_evaluation_code(code)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 479, in generate_evaluation_code
self.generate_result_code(code)
File
"/usr/local/lib/python2.6/dist-packages/Cython-0.14.1-py2.6-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3044, in generate_result_code
func_type.op_arg_struct.base_type, manage_ref=True)
AttributeError: 'CFuncType' object has no attribute 'op_arg_struct'
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PyCharm
On 22 March 2011 15:33, Jason Grout wrote: > On 3/16/11 5:33 AM, mark florisson wrote: >> >> The PyCharm IDE (http://www.jetbrains.com/pycharm/) has granted the >> Cython project a free Open Source license, which means that anyone >> developing Cython may freely use PyCharm to develop Cython. They >> prefer to license to remain unpublic, so if you develop Cython and >> want a free PyCharm license, send me an email and I will send you the >> license key. It is valid until 13 March 2012. > > > Is it the offer that is valid for a year, or is it the license that is valid > for only a year? The license, after that we have to renew it. > Thanks, > > Jason > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
I think it's a good idea, but I think it'd be better to use pure mode to get code that runs either way, or some sort of preprocessor (I've used m4 with good luck for this, though it doesn't syntax highlight nicely) to automatically derive pure python and cython from the same source file. For me at least, the branch of Cython that supports generators has worked flawlessly - except for one buglet that prevented use on 2.5.x. It was quickly fixed when reported though. Cython probably should get out of the tiny-version-number phase first though. I often feel that opensource projects use tiny version numbers for years as a sort of cop out. well after people have begun relying on the code for production use. On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: > Hi, > > there seems to be quite some interest in a project to get parts of CPython > and specifically its stdlib rewritten in Cython. I've copied the latest > python-dev mail below. The relevant part of the thread is here: > > http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 > > In short, we have strong supporters, but Guido has understandable doubts > against a new (and quite large) dependency and potential semantic > deviations. But there seem to be cases where slight changes would be > acceptable that Cython compiled modules might introduce, such as emitting > different exception messages, changing Python classes into extension > classes, or even preventing monkey patching in modules that are backed by C > modules anyway. > > It would be helpful to get support from the side of external distributors > that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If > they agreed to test the Cython generated stdlib modules in their > distributions, we could get user feedback that would allow python-dev to > take a well founded decision. > > Do we have any volunteers for trying this out? Both on the side of > distributors and implementors? > > At the current state of affairs, the implementation could still be financed > by a Python backed GSoC project, although it would be cool if more users > could just step up and simply try to compile and optimise stdlib modules > (preferably without major changes to the code). It's certainly a great way > to show off your Cython skills :). I gave it a try with difflib and it > turned out to be quite easy. > > http://blog.behnel.de/index.php?p=155 > > Reimplementing existing C modules in Cython might, however, be more > interesting for python-dev, but also be a larger undertaking. So a GSoC > might be worth running on that. > > Note that the latest Cython release does not have generator support yet, > and Vitja's branch on github is not very stable. We will try to get it up to > speed and merged during the workshop next week, at which point it will make > more sense to get this project started than right now. > > Stefan > > > Guido van Rossum, 22.03.2011 00:04: > >> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. Löwis" wrote: >> >>> Am 21.03.2011 11:58, schrieb Stefan Behnel: >>> Guido van Rossum, 21.03.2011 03:46: > Thanks for the clarifications. I now have a much better understanding > of what Cython is. But I'm not sold. For one, your attitude about > strict language compatibility worries me when it comes to the stdlib. > Not sure what you mean exactly. Given our large user base, we do worry a lot about things like backwards compatibility, for example. If you are referring to compatibility with Python, I don't think anyone in the project really targets Cython as a a drop-in replacement for a Python runtime. We aim to compile Python code, yes, and there's a hand-wavy idea in the back of our head that we may want a plain Python compatibility mode at some point that will disable several important optimisations. >>> >>> I think that's the attitude Guido worries about: if you don't have the >>> desire to provide 100% Python compatibility under all circumstances >>> (i.e. including if someone passes parameters of "incorrect" types), >>> then there is very little chance that we would replace a Python module >>> with a Cython-compiled one. >>> >>> The only exception would be cases where the Python semantics is murky >>> (e.g. where Jython or so actually behaves differently for the same >>> Python code, and still claims language conformance). E.g. the exact >>> message on a TypeError might change when compiling with Cython, >>> but the cases in which you get a TypeError must not change. >>> >> >> One other significant use case is the situation where we have an >> optional replacement module written in C (e.g. heapqmodule.c vs. >> heapq.py). There are usually many semantic differences between the C >> and pure-python module that we don't care about (e.g. monkeypatching >> won't work). >> >> The size of Cython as a dependency and its development speed are still >> problems though. In general for the core I don't
Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython
On Tue, Mar 22, 2011 at 4:09 PM, Dan Stromberg wrote: > > I think it's a good idea, but I think it'd be better to use pure mode to get > code that runs either way, or some sort of preprocessor (I've used m4 with > good luck for this, though it doesn't syntax highlight nicely) to > automatically derive pure python and cython from the same source file. It doesn't hurt to explore the potential before coming up with the actual solution. Ideally, the .py files would not have to be modified at all. > For me at least, the branch of Cython that supports generators has worked > flawlessly - except for one buglet that prevented use on 2.5.x. It was > quickly fixed when reported though. > > Cython probably should get out of the tiny-version-number phase first > though. I often feel that opensource projects use tiny version numbers for > years as a sort of cop out. well after people have begun relying on the code > for production use. We have a clear 1.0 goal, being able to compile the full Python language. We're not there yet, but very close. It may make sense at that point to also clean up any cruft we don't want to continue supporting forever. I agree, until that point, there's no way we would be a Python development dependency. - Robert > On Mon, Mar 21, 2011 at 11:10 PM, Stefan Behnel wrote: >> >> Hi, >> >> there seems to be quite some interest in a project to get parts of CPython >> and specifically its stdlib rewritten in Cython. I've copied the latest >> python-dev mail below. The relevant part of the thread is here: >> >> http://thread.gmane.org/gmane.comp.python.devel/122273/focus=122798 >> >> In short, we have strong supporters, but Guido has understandable doubts >> against a new (and quite large) dependency and potential semantic >> deviations. But there seem to be cases where slight changes would be >> acceptable that Cython compiled modules might introduce, such as emitting >> different exception messages, changing Python classes into extension >> classes, or even preventing monkey patching in modules that are backed by C >> modules anyway. >> >> It would be helpful to get support from the side of external distributors >> that use Cython already, e.g. Sage, Enthought/SciPy, ActiveState, etc. If >> they agreed to test the Cython generated stdlib modules in their >> distributions, we could get user feedback that would allow python-dev to >> take a well founded decision. >> >> Do we have any volunteers for trying this out? Both on the side of >> distributors and implementors? >> >> At the current state of affairs, the implementation could still be >> financed by a Python backed GSoC project, although it would be cool if more >> users could just step up and simply try to compile and optimise stdlib >> modules (preferably without major changes to the code). It's certainly a >> great way to show off your Cython skills :). I gave it a try with difflib >> and it turned out to be quite easy. >> >> http://blog.behnel.de/index.php?p=155 >> >> Reimplementing existing C modules in Cython might, however, be more >> interesting for python-dev, but also be a larger undertaking. So a GSoC >> might be worth running on that. >> >> Note that the latest Cython release does not have generator support yet, >> and Vitja's branch on github is not very stable. We will try to get it up to >> speed and merged during the workshop next week, at which point it will make >> more sense to get this project started than right now. >> >> Stefan >> >> >> Guido van Rossum, 22.03.2011 00:04: >>> >>> On Mon, Mar 21, 2011 at 3:44 PM, "Martin v. Löwis" wrote: Am 21.03.2011 11:58, schrieb Stefan Behnel: > > Guido van Rossum, 21.03.2011 03:46: >> >> Thanks for the clarifications. I now have a much better understanding >> of what Cython is. But I'm not sold. For one, your attitude about >> strict language compatibility worries me when it comes to the stdlib. > > Not sure what you mean exactly. Given our large user base, we do worry > a > lot about things like backwards compatibility, for example. > > If you are referring to compatibility with Python, I don't think anyone > in the project really targets Cython as a a drop-in replacement for a > Python runtime. We aim to compile Python code, yes, and there's a > hand-wavy idea in the back of our head that we may want a plain Python > compatibility mode at some point that will disable several important > optimisations. I think that's the attitude Guido worries about: if you don't have the desire to provide 100% Python compatibility under all circumstances (i.e. including if someone passes parameters of "incorrect" types), then there is very little chance that we would replace a Python module with a Cython-compiled one. The only exception would be cases where the Python semantics is murky (e.g. where Jython or so actually behaves differently for the same Python cod
