Re: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
Hi again PyPy has functions to parse C headers to get macros and constants so I could create C functions to wrap the macros and probably inline constants in the Python part of the wrapper. This doesn't solve the problem of ifdefs but this is a start. Cheers Romain 2011/4/7 Carl Witty > On Thu, Apr 7, 2011 at 9:06 AM, Dag Sverre Seljebotn > wrote: > > This seems similar to Carl Witty's port of Cython to .NET/IronPython. An > > important insight from that project is that Cython code does NOT specify > an > > ABI, only an API which requires a C compiler to make sense. That is; many > > wrapped C libraries have plenty of macros, we only require partial > > definition of struct, we only require approximate typedef's, and so on. > > > > In the .NET port, the consequence was that rather than the original idea > of > > generating C# code (with FFI specifications) was dropped, and one instead > > went with C++/CLR (which is a proper C++ compiler that really understands > > the C side on an API level, in addition to giving access to the .NET > > runtime). > > Let me just add that a way to deal with the API vs. ABI issue would be > useful for other potential Cython targets as well, such as IronPython > using C# and Jython. (A C# port for IronPython would be more valuable > than my C++/CLI port because it would work under Mono -- Mono doesn't > have a C++/CLI compiler and probably never will.) > > Carl > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] [GSoC] Python backend for Cython using PyPy's FFI
Hi I proposed the Summer of Code project regarding the Python backend for Cython. As I said in my proposal this would translate Cython code to Python + FFI code (I don't know yet if it will use ctypes or something specific to PyPy). PyPy's ctypes is now really fast and this will allow people to port their Cython code to PyPy. For the moment I've been mostly in touch with the PyPy people and they seem happy with my proposal. Of course I'm available for questions. Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [GSoC] Python backend for Cython using PyPy's FFI
Hi I investigated the code produced by Cython, and I see 3 cases which should be handled : * A pure python variable which has a value assigned (including None) * A pure python variable which has no value assigned * A C variable (we can't test if they are set of not) The first and second one seem relatively easy to implement it can reuse __pyx_string_tab. The code will call __Pyx_GetName on each element of the array, if it returns NULL, don't put the variable in the dictionary, else put the name and it's value in the dictionary. For the last one, it will probably need the C name, C type and Python name of each C variable, add the python name in the dictionary and wrap the C value into a Python object. However this will only provide read access to the globals. To offer read/write access, making a proxy dictionary is probably the most straightforward implementation (it can be rather inefficient though since we need to do an if/else to compare the value we want to set with every C globals). Do you think I'm on the right way ? Thanks Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Scope resolution in a Visitor
Hi I'm doing the PyPy backend for Cython Summer of Code project and I would like to know if there is a way of getting the AST Node responsible for the declaration of a variable. For example : def func(): def func(): pass func() In the situation where I'm currently traversing the function call, I would like to get the function declaration node. Is there already something to do that ? Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Scope resolution in a Visitor
On Tue, May 24, 2011 at 06:56:19AM +0200, Stefan Behnel wrote:
> Romain Guillebert, 23.05.2011 20:33:
> >I'm doing the PyPy backend for Cython Summer of Code project and I would
> >like to know if there is a way of getting the AST Node responsible for
> >the declaration of a variable.
> >
> >For example :
> >
> >def func():
> > def func():
> > pass
> >
> > func()
> >
> >In the situation where I'm currently traversing the function call, I
> >would like to get the function declaration node.
> >Is there already something to do that ?
>
> What for? What kind of information do you need from it?
>
> In Cython, the metadata about a name is stored in its symbol table
> entry. The call to "func" is a SimpleCallNode that holds a NameNode
> which has an "entry" attribute. The entry knows the name, type,
> signature, etc. of the function that is being called.
>
> Note, however, that you need to run both the "analyse declarations"
> step and the "analyse types/expression" step to figure out all
> information about the name. The declaration analysis will add the
> definition of "func" to the symbol table, so you can look it up
> ("env.lookup('func')") from that point on.
>
> Does that help?
>
> Stefan
I want to generate standard python function from "cdef extern from"
function, but I need to annotate these function as they may require a
special treatment (for passing value by reference), also, I may turn C
macros into a call to a C function returning the macro itself.
cdef extern from "stdio.h":
cdef int BUFSIZ
Will be turned into :
def get_BUFSIZ():
# whatever
So I need to turn every access to BUFSIZ into a function call.
Cheers
Romain
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Scope resolution in a Visitor
> Sounds to me like you should attach the necessary information to the > symbol table entry when analysing he "external" declaration. That's what I wanted to do that's why I asked how I could access the external declaration node > We currently store a "cname", so adding something like a property name > or accessor information should fit your use case. We may want to > reserve a separate namespace for Python output related information, > though. Maybe something like "entry.pybackend.*"? This would be a clean way to do it > Also, we considered having module level properties at some point. > That may fall into the same bucket. Writing a transformation for that should be relatively straightforward. > What should happen for cimported declarations? Will they be handled > in the same way as internally defined "external" declarations? I > guess so, even if they may be shared between modules. There's 2 possibilities I think : - For each file we generate the wrapping functions - For each pxd file, generate a file containing the wrapping functions Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] [GSoC] Blog post regarding the Cython backend aiming PyPy
Hi I've posted and article on my blog that explains what I've done during the community bonding period and the first week of the Google Summer of Code : http://rguillebert.blogspot.com/2011/05/cython-backend-aiming-pypy-week-1.html Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] [GSoC] Blog post regarding the Cython backend aiming PyPy
Hi I've posted and article on my blog that explains what I've done during the community bonding period and the first week of the Google Summer of Code : http://rguillebert.blogspot.com/2011/05/cython-backend-aiming-pypy-week-1.html Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] [GSoC] Blog post regarding the Cython backend aiming PyPy
Hi I've posted and article on my blog that explains what I've done during the community bonding period and the first week of the Google Summer of Code : http://rguillebert.blogspot.com/2011/05/cython-backend-aiming-pypy-week-1.html Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [GSoC] Blog post regarding the Cython backend aiming PyPy
On Tue, May 31, 2011 at 12:08:52PM +0200, mark florisson wrote:
> Cool. Would it be useful to always generate wrapper functions for
> extern functions with numeric argument types? E.g. this is valid
> Cython code:
>
> cdef extern from "foo.h":
> ctypedef unsigned int size_t
> size_t func_with_typedef_arg(size_t a)
>
> So here the ctypedef for size_t is unsigned int, which is only valid
> in C for some platforms/architectures. So perhaps a wrapper function
> could solve that issue:
>
> int __pyx_wrapper_func_with_typedef_arg(int a) {
> /* some bounds and sign checking code here ? */
> return func_with_typedef(a);
> }
>
> Because then you always know that calling it with argtypes = [c_int]
> and restype = c_int is valid.
> (BTW, it's also valid to declare something as a function which is
> actually a macro, so I suppose you always need wrappers for
> functions.)
>
> Do you already have an idea how to handle struct type arguments? Those
> are often also incomplete... but perhaps I'm geting too far ahead, I
> don't think we're quite there yet. I suppose you could also place this
> ABI burden on the user (for now), as does ctypes.
>
> As for the linking stuff, perhaps it's a good idea to look at
> http://wiki.cython.org/enhancements/distutils_preprocessing (currently
> down unfortunately, but in google cache). Then you can list libraries
> like this: '# distutils: libraries = spam eggs' at the top of the
> file.
>
> Cheers,
>
> Mark
Hi Mark
For the moment, I try to handle the standard case : call a function with
C basic types, then add structures and pass by pointer.
To deal with the ABI, fijal told me that there's a tool called
ctypes_configure, I will investigate this to see what I can get from it.
Cheers
Romain
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
[Cython] [GSoC] CTypes backend for Cython aiming PyPy - Week 2
Hi I summarized the second week of my Summer of Code project in this blog post: http://rguillebert.blogspot.com/2011/06/cython-backend-aiming-pypy-week-2.html Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython backend aiming PyPy Status
Hi I created a blog post summarizing what I've done the last few weeks on the Cython backend aiming PyPy. It's located at this URL : http://rguillebert.blogspot.com/2011/07/cython-backend-aiming-pypy-status.html Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Transformation of pxds
Hi I can now compile pxd files, but I have encountered something rather strange : AnalyseExpressionsTransform turns : cimport foo def test(): foo.printf() into : cimport foo def test(): printf() It is due to the method analyse_as_cimported_attribute of AttributeNode in the file Cython/Compiler/ExprNodes.py Is this useful (since I don't think it has anything to do with expression analysis) and do you have an idea on how I can solve this (my first idea is : extend the class to disable the effect of this method but it's more a hack than anything else) ? Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython bug ?
Hi I tried to compiled Demos/primes.pyx using the ctypes backend and I think I've found a bug : The Entry for the kmax parameter does not set is_arg to 1. However if I turn the def into a cdef, it works fine. Is this a bug or a known hack ? Thanks Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] CTypes backend for Cython Status
Hi The Google Summer of Code has ended and I didn't give the current status to anyone yet (I was very busy with a report I had to write for my university). There is still work to do on the project (there was more work than I expected, especially because of semantic differences between Cython and ctypes) so I'll talk about what needs to be done (even if it does not sound good compared to talking about what has been done) from the most important to the least important in my opinion : - Pointer vs Array, Cython mixes the two while ctypes does not, this can probably be fixed by using arrays everywhere (if we can convert pointers into arrays) - Take into account header files declared globally - Macros, this is probably the biggest part but it's doable, Cython has the types of the arguments and the return value so it's possible to generate a C function corresponding to a macro - Pointer to functions Some of them are trivial, others just require good ideas and macros demands a big amount of work. I'm still working on it and if someone wants to give a hand, I'll be happy to explain what I've done. Thanks Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CTypes backend for Cython Status
Hi Stefan On Fri, Sep 09, 2011 at 10:35:20AM +0200, Stefan Behnel wrote: > Hi Romain, > > thanks for the feedback. > > Romain Guillebert, 08.09.2011 06:18: > >The Google Summer of Code has ended and I didn't give the current status > >to anyone yet (I was very busy with a report I had to write for my > >university). > > > >There is still work to do on the project (there was more work than I > >expected, especially because of semantic differences between Cython and > >ctypes) so I'll talk about what needs to be done (even if it does not > >sound good compared to talking about what has been done) from the most > >important to the least important in my opinion : > > > >- Pointer vs Array, Cython mixes the two while ctypes does not, this > > can probably be fixed by using arrays everywhere (if we can convert > > pointers into arrays) > > Could you elaborate on this? Why can't you always use pointers? > You can't do array indexing on ctypes pointers (and no straightforward pointer arithmetic either) > > >- Take into account header files declared globally > > "globally" as in ... ? > > And what exactly do you mean by "take header files into account"? > Take into account : pass to ctypes_configure And I didn't want to say globally, but right now I only pass to ctypes_configure the header file given by "cdef extern from" and from my tests it's not enough > > >- Macros, this is probably the biggest part but it's doable, Cython has > > the types of the arguments and the return value so it's possible to > > generate a C function corresponding to a macro > > Agreed. I think that's the only viable solution. Won't work for all > macros, but at least for a rather large number of them. > > > >- Pointer to functions > > I take it this isn't implemented at all? Do you mean pointers to > Cython cdef functions or also to external C defined functions? > Pointer to cdefed functions and pointers to function returned by C functions (although I don't know for the later but it would surprise me) > > >Some of them are trivial, others just require good ideas and macros > >demands a big amount of work. > > Really? Apart from generating the rather straight forward wrapper > functions for them, what else needs to be done? > You have to include the right headers and take care of the types (if a type is ctypedefed there's probably something to do) > > >I'm still working on it and if someone wants to give a hand, I'll be > >happy to explain what I've done. > > Happy to hear that. > > Stefan Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython-ctypes branch
Hi I'll try to do that this week, I agree that it's better to get this branch merged. Rpython isn't suitable at all for this kind of use case because you have to recompile the entire PyPy executable each time you change a library (long compile time and big memory consumption), loading modules is not trivial, the entire program must be type-inferable (which probably isn't the case of most Cython programs), global variables are considered constants, and I think (don't quote me) that the JIT doesn't work on rpython code. I have no idea on the speedup/slowdown though. Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Merging of the ctypes backend branch
Hi everyone I rebased the ctypes backend branch to the last cython commit, and I wondered how the branch should be merged with the main cython repository. I see 3 options : a) upload the branch without merging it b) merge the branch but not run the test suite on ctypes by default or c) merge the branch and run the test suite. I think it's better to do c) but the test results shouldn't mix with the results of the other backends. What do you think ? Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
