Re: [Cython] CyFunction refactoring plan
On 30 September 2011 07:47, Vitja Makarov wrote: > 2011/9/30 Vitja Makarov : >> 2011/9/30 Robert Bradshaw : >>> On Thu, Sep 29, 2011 at 10:43 PM, Stefan Behnel wrote: Vitja Makarov, 30.09.2011 06:41: > > 2011/9/28 Vitja Makarov: >> >> I tried to build simple plan for ongoing cython function refactoring >> >> * Replace assignment synthesis with SingleAssignmentNode, where LHS is >> NameNode and RHS is PyCFunctionNode >> * Split function body into python wrapper and C function >> http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring >> >> Then we can implement some features and optimizations: >> >> * Reduce difference between cdef and def functions >> * Store runtime evaluated default values inside CyFunction, ticket #674 >> * Implement no-args super(), ticket #696 >> * Function call inlining > > If nobody don't mind I would start with first one. >>> >>> I would love to see this happen. >>> Please go ahead. :) Note that you will encounter some problems when enabling name assignments for all named functions. I tried that at least once and it "didn't work", but I didn't take the time yet to investigate them further. I assume you are going to work on this in your own repo? >>> >>> Please also coordinate with Mark's work on function dispatching for >>> fused types. >>> >> >> I assume that that fused type functions are cdef ones so I think that >> should be easy to merge. >> On the other hand it's better to have Mark's branch merged into master. >> >> Mark, what is the state of your fused types branch? >> Is it possible to break it into smaller parts to ease reviewing and merging? >> > > It seems I meant memview branch not fusedtypes. There are 2 pending branches, _memview_rebase, which has support for memoryviews, and fusedtypes. The former is ready for merge, it's waiting to be reviewed. The fused types branch needs to subclass CyFunction (it basically modified the old binding function). There was also some duplicate functionality there, so I thought it'd be easier and more convenient to use the utility code loading there. Since it's not a strict dependency and it will be blocking progress, I will try to find some time to get it merge-ready for master. But no, it does cdef, cpdef and def methods, and it has some changes to all function nodes (FuncdefNode, CFuncdefNode and DefNode). These changes shouldn't be major though, but the logic in FusedFuncdefNode does differentiate between all the different functions in order to support them. Feel free to ask me about specifics any time. > -- > vitja. > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: callback function pointer problem
On 29 September 2011 22:48, Robert Bradshaw
wrote:
> On Thu, Sep 29, 2011 at 5:29 AM, mark florisson
> wrote:
>> On 29 September 2011 13:13, Miguel Angel wrote:
Structs already coerce to python dicts. In the memoryview branch it
also does the reverse, i.e. coerce a dict to a struct (you have to be
careful with strings, making sure you hold a reference to the string
object as long as you use the pointer in the struct). You have to be
sure to declare every struct attribute though.
>>>
>>> Could i see an example usage of this in any repository or the
>>> documentation? I
>>> can't find any.
>>>
>>> Regards,
>>> Miguel Angel.
>>>
>>
>> It's not in the documentation yet, I updated that in a branch that
>> still needs to be merged (which hopefully will be in 0.16). The
>> functionality from struct -> object is there though:
>>
>> cdef extern from *:
>> ctypedef struct MyStruct:
>> int attrib
>>
>> cdef MyStruct s = {'attrib':10} # Note, this is NOT object->struct
>> conversion, it only eats compile-time expressions here
>
> or even
>
> s = MyStruct(attrib=10)
That looks very nice, a lot more intuitive than the dict assignment.
> It would be natural to support array -> list conversion for slices the
> baseclass is convertable, e.g.
>
> cdef double* data = ...
> print data[:10]
>
> Converting an int[10] to a Python object without slicing is a bit odd as
>
> cdef int a[10]
> [compute a]
> cdef object o = a
> a[0] = something else
> o[1] = another thing
>
> is non-initiative, as arrays can't be assigned in C and "assignment"
> of lists in Python are by reference, not by copy. "o = a[:]" is plenty
> clear that there's a copy, so that's better. The other way is a bit
> messier, but
>
> a[:n] = [python list]
>
> could certainly be supported via an implicit loop. Another
> consideration is that with primitive data types we're venturing into
> memory view territory, which might be a more useful representation
> than Python lists.
>
> - Robert
>
Indeed. Currently you can only cast pointers to memoryview slices, but
there's no reason to not support arrays as well (without also needing
to first cast the array to a pointer). So if arrays have shape
information, you could immediately coerce to memoryview (cython.array)
objects without needing a cast. This would also work for arrays in
structs, as they can only ever have a static size (known at C compile
time).
Shall we support that? I won't be hard as the infrastructure for
casting pointers is already there.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: callback function pointer problem
Note: The last field in a C struct can be of variable size, decided at
malloc-time. I can't think of a clear syntax for that, except perhaps a builtin
function cython.coercevariablesizestruct...
Dag Sverre
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
mark florisson wrote:
On 29 September 2011 22:48, Robert Bradshaw
wrote: > On Thu, Sep 29, 2011 at 5:29 AM, mark florisson >
wrote: >> On 29 September 2011 13:13, Miguel Angel
wrote: Structs already coerce to python dicts. In
the memoryview branch it also does the reverse, i.e. coerce a dict to a
struct (you have to be careful with strings, making sure you hold a
reference to the string object as long as you use the pointer in the
struct). You have to be sure to declare every struct attribute though. >>>
>>> Could i see an example usage of this in any repository or the
documentation? I >>> can't find any. >>> >>> Regards, >>> Miguel Angel. >>> >>
>> It's not in the documentation yet, I updated that in a branch that >> still
needs to be merged (which hopefully will be in 0.16). The >> functionality from
struct -> object is there though: >> >> cdef extern from *: >>ctypedef
struct MyStruc
t: >>
int attrib >> >> cdef MyStruct s = {'attrib':10} # Note, this is NOT
object->struct >> conversion, it only eats compile-time expressions here > > or
even > >s = MyStruct(attrib=10) That looks very nice, a lot more intuitive
than the dict assignment. > It would be natural to support array -> list
conversion for slices the > baseclass is convertable, e.g. > >cdef double*
data = ... >print data[:10] > > Converting an int[10] to a Python object
without slicing is a bit odd as > >cdef int a[10] >[compute a] >
cdef object o = a >a[0] = something else >o[1] = another thing > > is
non-initiative, as arrays can't be assigned in C and "assignment" > of lists in
Python are by reference, not by copy. "o = a[:]" is plenty > clear that there's
a copy, so that's better. The other way is a bit > messier, but > >a[:n] =
[python list] > > could certainly be supported via an implicit loop. Another >
consideration is that with primitive data types we're ve
nturing
into > memory view territory, which might be a more useful representation >
than Python lists. > > - Robert > Indeed. Currently you can only cast pointers
to memoryview slices, but there's no reason to not support arrays as well
(without also needing to first cast the array to a pointer). So if arrays have
shape information, you could immediately coerce to memoryview (cython.array)
objects without needing a cast. This would also work for arrays in structs, as
they can only ever have a static size (known at C compile time). Shall we
support that? I won't be hard as the infrastructure for casting pointers is
already there._
cython-devel mailing list cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: callback function pointer problem
Are you saying that when coercing a struct to an object, one would copy scalar
fields by value but reference array fields? -1, that would be confusing. Either
the whole struct through a view, or copy it all.
It bothers me that structs are passed by value in Cython, but it seems
impossible to change that now. (i.e, once upon a time one could have required
the use of a copy method to do a struct assignment and give a syntax error
otherwise, which would have worked nicer with Python semantics).
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
mark florisson wrote:
On 29 September 2011 22:48, Robert Bradshaw
wrote: > On Thu, Sep 29, 2011 at 5:29 AM, mark florisson >
wrote: >> On 29 September 2011 13:13, Miguel Angel
wrote: Structs already coerce to python dicts. In
the memoryview branch it also does the reverse, i.e. coerce a dict to a
struct (you have to be careful with strings, making sure you hold a
reference to the string object as long as you use the pointer in the
struct). You have to be sure to declare every struct attribute though. >>>
>>> Could i see an example usage of this in any repository or the
documentation? I >>> can't find any. >>> >>> Regards, >>> Miguel Angel. >>> >>
>> It's not in the documentation yet, I updated that in a branch that >> still
needs to be merged (which hopefully will be in 0.16). The >> functionality from
struct -> object is there though: >> >> cdef extern from *: >>ctypedef
struct MyStruc
t: >>
int attrib >> >> cdef MyStruct s = {'attrib':10} # Note, this is NOT
object->struct >> conversion, it only eats compile-time expressions here > > or
even > >s = MyStruct(attrib=10) That looks very nice, a lot more intuitive
than the dict assignment. > It would be natural to support array -> list
conversion for slices the > baseclass is convertable, e.g. > >cdef double*
data = ... >print data[:10] > > Converting an int[10] to a Python object
without slicing is a bit odd as > >cdef int a[10] >[compute a] >
cdef object o = a >a[0] = something else >o[1] = another thing > > is
non-initiative, as arrays can't be assigned in C and "assignment" > of lists in
Python are by reference, not by copy. "o = a[:]" is plenty > clear that there's
a copy, so that's better. The other way is a bit > messier, but > >a[:n] =
[python list] > > could certainly be supported via an implicit loop. Another >
consideration is that with primitive data types we're ve
nturing
into > memory view territory, which might be a more useful representation >
than Python lists. > > - Robert > Indeed. Currently you can only cast pointers
to memoryview slices, but there's no reason to not support arrays as well
(without also needing to first cast the array to a pointer). So if arrays have
shape information, you could immediately coerce to memoryview (cython.array)
objects without needing a cast. This would also work for arrays in structs, as
they can only ever have a static size (known at C compile time). Shall we
support that? I won't be hard as the infrastructure for casting pointers is
already there._
cython-devel mailing list cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: callback function pointer problem
On 30 September 2011 22:14, Dag Sverre Seljebotn
wrote:
> Are you saying that when coercing a struct to an object, one would copy
> scalar fields by value but reference array fields? -1, that would be
> confusing. Either the whole struct through a view, or copy it all.
Hmm, yeah it might be confusing. To me it makes sense though, as
scalars are immutable but arrays are not, although by that logic the
struct should be a view. So perhaps a list makes more sense. Making
arrays in structs lists means you likely want to do the same for the
normal case, i.e. arrays as variables. I'm not sure you'd want normal
arrays to object conversion to be implicit though, I'd rather see an
explicit cast for that.
> It bothers me that structs are passed by value in Cython, but it seems
> impossible to change that now. (i.e, once upon a time one could have
> required the use of a copy method to do a struct assignment and give a
> syntax error otherwise, which would have worked nicer with Python
> semantics).
>
>
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>
> mark florisson wrote:
>>
>> On 29 September 2011 22:48, Robert Bradshaw
>> wrote: > On Thu, Sep 29, 2011 at 5:29 AM, mark florisson >
>> wrote: >> On 29 September 2011 13:13, Miguel
>> Angel wrote: Structs already coerce to python
>> dicts. In the memoryview branch it also does the reverse, i.e. coerce a
>> dict to a struct (you have to be careful with strings, making sure you
>> hold a reference to the string object as long as you use the pointer in
>> the struct). You have to be sure to declare every struct attribute
>> though. >>> >>> Could i see an example usage of this in any repository or
>> the documentation? I >>> can't find any. >>> >>> Regards, >>> Miguel Angel.
>> >>> >> >> It's not in the documentation yet, I updated that in a branch that
>> >> still needs to be merged (which hopefully will be in 0.16). The >>
>> functionality from struct -> object is there though: >> >> cdef extern from
>> *: >> ctypedef struct MyStruct: >> int attrib >> >> cdef MyStruct
>> s = {'attrib':10} # Note, this is NOT object->struct >> conversion, it only
>> eats compile-time expressions here > > or even > > s =
>> MyStruct(attrib=10) That looks very nice, a lot more intuitive than the dict
>> assignment. > It would be natural to support array -> list conversion for
>> slices the > baseclass is convertable, e.g. > > cdef double* data = ... >
>> print data[:10] > > Converting an int[10] to a Python object without
>> slicing is a bit odd as > > cdef int a[10] > [compute a] > cdef
>> object o = a > a[0] = something else > o[1] = another thing > > is
>> non-initiative, as arrays can't be assigned in C and "assignment" > of lists
>> in Python are by reference, not by copy. "o = a[:]" is plenty > clear that
>> there's a copy, so that's better. The other way is a bit > messier, but > >
>> a[:n] = [python list] > > could certainly be supported via an implicit
>> loop. Another > consideration is that with primitive data types we're
>> venturing into > memory view territory, which might be a more useful
>> representation > than Python lists. > > - Robert > Indeed. Currently you can
>> only cast pointers to memoryview slices, but there's no reason to not
>> support arrays as well (without also needing to first cast the array to a
>> pointer). So if arrays have shape information, you could immediately coerce
>> to memoryview (cython.array) objects without needing a cast. This would also
>> work for arrays in structs, as they can only ever have a static size (known
>> at C compile time). Shall we support that? I won't be hard as the
>> infrastructure for casting pointers is already there.
>>
>> cython-devel mailing list cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: callback function pointer problem
On 30 September 2011 21:50, Dag Sverre Seljebotn
wrote:
> Note: The last field in a C struct can be of variable size, decided at
> malloc-time. I can't think of a clear syntax for that, except perhaps a
> builtin function cython.coercevariablesizestruct...
>
>
>
> Dag Sverre
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
That's really more of a hack right, not something that's technically
legal in C. If the user want to use hacked data structures, he/she
should not rely on built-in conversion, but convert manually, by
building a dict manually or whatever (and by casting like
mystruct.my_last_overallocated_field).
> mark florisson wrote:
>>
>> On 29 September 2011 22:48, Robert Bradshaw
>> wrote: > On Thu, Sep 29, 2011 at 5:29 AM, mark florisson >
>> wrote: >> On 29 September 2011 13:13, Miguel
>> Angel wrote: Structs already coerce to python
>> dicts. In the memoryview branch it also does the reverse, i.e. coerce a
>> dict to a struct (you have to be careful with strings, making sure you
>> hold a reference to the string object as long as you use the pointer in
>> the struct). You have to be sure to declare every struct attribute
>> though. >>> >>> Could i see an example usage of this in any repository or
>> the documentation? I >>> can't find any. >>> >>> Regards, >>> Miguel Angel.
>> >>> >> >> It's not in the documentation yet, I updated that in a branch that
>> >> still needs to be merged (which hopefully will be in 0.16). The >>
>> functionality from struct -> object is there though: >> >> cdef extern from
>> *: >> ctypedef struct MyStruct: >> int attrib >> >> cdef MyStruct
>> s = {'attrib':10} # Note, this is NOT object->struct >> conversion, it only
>> eats compile-time expressions here > > or even > > s =
>> MyStruct(attrib=10) That looks very nice, a lot more intuitive than the dict
>> assignment. > It would be natural to support array -> list conversion for
>> slices the > baseclass is convertable, e.g. > > cdef double* data = ... >
>> print data[:10] > > Converting an int[10] to a Python object without
>> slicing is a bit odd as > > cdef int a[10] > [compute a] > cdef
>> object o = a > a[0] = something else > o[1] = another thing > > is
>> non-initiative, as arrays can't be assigned in C and "assignment" > of lists
>> in Python are by reference, not by copy. "o = a[:]" is plenty > clear that
>> there's a copy, so that's better. The other way is a bit > messier, but > >
>> a[:n] = [python list] > > could certainly be supported via an implicit
>> loop. Another > consideration is that with primitive data types we're
>> venturing into > memory view territory, which might be a more useful
>> representation > than Python lists. > > - Robert > Indeed. Currently you can
>> only cast pointers to memoryview slices, but there's no reason to not
>> support arrays as well (without also needing to first cast the array to a
>> pointer). So if arrays have shape information, you could immediately coerce
>> to memoryview (cython.array) objects without needing a cast. This would also
>> work for arrays in structs, as they can only ever have a static size (known
>> at C compile time). Shall we support that? I won't be hard as the
>> infrastructure for casting pointers is already there.
>>
>> cython-devel mailing list cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Re: callback function pointer problem
On 30 September 2011 22:42, mark florisson wrote:
> On 30 September 2011 22:14, Dag Sverre Seljebotn
> wrote:
>> Are you saying that when coercing a struct to an object, one would copy
>> scalar fields by value but reference array fields? -1, that would be
>> confusing. Either the whole struct through a view, or copy it all.
>
> Hmm, yeah it might be confusing. To me it makes sense though, as
> scalars are immutable but arrays are not, although by that logic the
> struct should be a view. So perhaps a list makes more sense. Making
> arrays in structs lists means you likely want to do the same for the
> normal case, i.e. arrays as variables. I'm not sure you'd want normal
> arrays to object conversion to be implicit though, I'd rather see an
> explicit cast for that.
I suppose none of it makes me very happy, maybe I'll just forget about
it for now.
>> It bothers me that structs are passed by value in Cython, but it seems
>> impossible to change that now. (i.e, once upon a time one could have
>> required the use of a copy method to do a struct assignment and give a
>> syntax error otherwise, which would have worked nicer with Python
>> semantics).
>>
>>
>> --
>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>
>> mark florisson wrote:
>>>
>>> On 29 September 2011 22:48, Robert Bradshaw
>>> wrote: > On Thu, Sep 29, 2011 at 5:29 AM, mark florisson >
>>> wrote: >> On 29 September 2011 13:13, Miguel
>>> Angel wrote: Structs already coerce to python
>>> dicts. In the memoryview branch it also does the reverse, i.e. coerce a
>>> dict to a struct (you have to be careful with strings, making sure you
>>> hold a reference to the string object as long as you use the pointer in
>>> the struct). You have to be sure to declare every struct attribute
>>> though. >>> >>> Could i see an example usage of this in any repository or
>>> the documentation? I >>> can't find any. >>> >>> Regards, >>> Miguel Angel.
>>> >>> >> >> It's not in the documentation yet, I updated that in a branch that
>>> >> still needs to be merged (which hopefully will be in 0.16). The >>
>>> functionality from struct -> object is there though: >> >> cdef extern from
>>> *: >> ctypedef struct MyStruct: >> int attrib >> >> cdef MyStruct
>>> s = {'attrib':10} # Note, this is NOT object->struct >> conversion, it only
>>> eats compile-time expressions here > > or even > > s =
>>> MyStruct(attrib=10) That looks very nice, a lot more intuitive than the dict
>>> assignment. > It would be natural to support array -> list conversion for
>>> slices the > baseclass is convertable, e.g. > > cdef double* data = ... >
>>> print data[:10] > > Converting an int[10] to a Python object without
>>> slicing is a bit odd as > > cdef int a[10] > [compute a] > cdef
>>> object o = a > a[0] = something else > o[1] = another thing > > is
>>> non-initiative, as arrays can't be assigned in C and "assignment" > of lists
>>> in Python are by reference, not by copy. "o = a[:]" is plenty > clear that
>>> there's a copy, so that's better. The other way is a bit > messier, but > >
>>> a[:n] = [python list] > > could certainly be supported via an implicit
>>> loop. Another > consideration is that with primitive data types we're
>>> venturing into > memory view territory, which might be a more useful
>>> representation > than Python lists. > > - Robert > Indeed. Currently you can
>>> only cast pointers to memoryview slices, but there's no reason to not
>>> support arrays as well (without also needing to first cast the array to a
>>> pointer). So if arrays have shape information, you could immediately coerce
>>> to memoryview (cython.array) objects without needing a cast. This would also
>>> work for arrays in structs, as they can only ever have a static size (known
>>> at C compile time). Shall we support that? I won't be hard as the
>>> infrastructure for casting pointers is already there.
>>>
>>> cython-devel mailing list cython-devel@python.org
>>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
