[Cython] Automatic conversion with fixed-size C arrays
Hi devs, Being able to convert between simple C structs and Python dictionaries is great, but it doesn't work if there is a fixed-size array field in the struct. It seems like the following struct should be convertible safely: cdef struct state_t: int i, j, k float x[3] float v[3] Along with that, I'd expect that converting between fixed-sized C arrays and Python iterables should work as well: def auto_convert(list ll): cdef int arr[10] = ll # ... return arr If this is deemed something worth doing and if someone is willing to give pointers when I get stuck, I'm offering to put in the development time to implement this. Thoughts? Kurt ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Wed, Jul 16, 2014 at 1:02 PM, Robert Bradshaw wrote: > > > Yes, this'd be nice to have. One difficulty with arrays is that they > can't be returned by value, and so the ordinary from_py_function > mechanism (which gets called recursively) would need to be adapted. > (Allowing to_py_function to be optinally be called by reference > instead of by value could be nice as well from a performance > standpoint.) > OK, thanks for the pointers. I'll put some time on this over the weekend. Should I just make a PR when things are ready to review, or should I put up an issue first? > > - Robert > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Thu, Jul 24, 2014 at 2:07 PM, Robert Bradshaw wrote:
> On Fri, Jul 18, 2014 at 12:44 PM, Robert Bradshaw
> wrote:
> > On Fri, Jul 18, 2014 at 12:08 PM, Kurt Smith wrote:
> >> On Wed, Jul 16, 2014 at 1:02 PM, Robert Bradshaw
> wrote:
> >>>
> >>>
> >>> Yes, this'd be nice to have. One difficulty with arrays is that they
> >>> can't be returned by value, and so the ordinary from_py_function
> >>> mechanism (which gets called recursively) would need to be adapted.
> >>> (Allowing to_py_function to be optinally be called by reference
> >>> instead of by value could be nice as well from a performance
> >>> standpoint.)
> >>
> >>
> >> OK, thanks for the pointers. I'll put some time on this over the
> weekend.
> >> Should I just make a PR when things are ready to review, or should I
> put up
> >> an issue first?
> >
> > I think this thread is sufficient; looking forward to a pull request.
>
> Don't know if you had time to look at this yet,
Yes, I've put in some time. Initial focus is getting
cdef int a[10] = obj
working, and then using that for structs.
Took a little while to re-orient myself with the codebase.
I'm working on getting the `to_py_function` and `from_py_function`
infrastructure to take arguments by reference; right now I'm getting
something hacked into place, and I'd appreciate your review to point out
the right way (or at least a better way) to do it.
Will likely have a PR for you by this weekend.
> but another possible
> hack would be to use the fact that struct { int x[N]; } is bit
> compatible with int[N] and can be passed/assigned by value. E.g.
>
Yes, this is a good point, although if I can get the pass-by-reference
working for fixed-size arrays, ideally we can do the same for structs with
array members.
>
> struct wrapped_array {
> int x[3];
> };
>
> int main(int argc, char** argv) {
> int src[3];
> int dest[3];
> *((struct wrapped_array *) &dest) = *((struct wrapped_array *) &src);
> }
>
> OTOH, it might make sense to simply add array members as a special
> case in struct conversion.
>
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Thu, Jul 24, 2014 at 11:16 PM, Robert Bradshaw
wrote:
> On Thu, Jul 24, 2014 at 7:13 PM, Kurt Smith wrote:
> >
> > On Thu, Jul 24, 2014 at 2:07 PM, Robert Bradshaw
> wrote:
> >>
> >> On Fri, Jul 18, 2014 at 12:44 PM, Robert Bradshaw
> >> wrote:
> >> > On Fri, Jul 18, 2014 at 12:08 PM, Kurt Smith
> wrote:
> >> >> On Wed, Jul 16, 2014 at 1:02 PM, Robert Bradshaw >
> >> >> wrote:
> >> >>>
> >> >>>
> >> >>> Yes, this'd be nice to have. One difficulty with arrays is that they
> >> >>> can't be returned by value, and so the ordinary from_py_function
> >> >>> mechanism (which gets called recursively) would need to be adapted.
> >> >>> (Allowing to_py_function to be optinally be called by reference
> >> >>> instead of by value could be nice as well from a performance
> >> >>> standpoint.)
> >> >>
> >> >>
> >> >> OK, thanks for the pointers. I'll put some time on this over the
> >> >> weekend.
> >> >> Should I just make a PR when things are ready to review, or should I
> >> >> put up
> >> >> an issue first?
> >> >
> >> > I think this thread is sufficient; looking forward to a pull request.
> >>
> >> Don't know if you had time to look at this yet,
> >
> >
> > Yes, I've put in some time. Initial focus is getting
> >
> > cdef int a[10] = obj
> >
> > working, and then using that for structs.
>
> Sounds good.
>
> > Took a little while to re-orient myself with the codebase.
> >
> > I'm working on getting the `to_py_function` and `from_py_function`
> > infrastructure to take arguments by reference; right now I'm getting
> > something hacked into place, and I'd appreciate your review to point out
> the
> > right way (or at least a better way) to do it.
>
> from_py_function always takes its argument (a PyObject*) by reference.
> It's used as an rvalue, so might not make sense for arrays.
>
My bad, I worded my response poorly -- what I mean is that currently the
generated code is something like:
__pyx_v_t1 = xxx_from_py_function_xxx(temp_py_obj);
___pyx_v_a = __pyx_v_t1;
where __pyx_v_a and __pyx_v_t1 are declared as fixed-size C arrays.
This won't work, for reasons pointed out.
So I'm trying to generate instead:
err_code = xxx_from_py_function_xxx(temp_py_obj, &__pyx_v_a[0], 10);
if (err_code == -1) { ... }
Where the 10 is the array length. The function would initialize the array
internally. The python object is passed by reference as before, and the
array is also passed by reference. The array is assigned to just once,
minimizing copying. We can return a status code to propagate errors, etc.
This seems like the cleanest code to me. Thoughts?
That's what I mean about "taking arguments by reference".
> > Will likely have a PR for you by this weekend.
>
> Cool.
>
> >> but another possible
> >> hack would be to use the fact that struct { int x[N]; } is bit
> >> compatible with int[N] and can be passed/assigned by value. E.g.
> >
> >
> > Yes, this is a good point, although if I can get the pass-by-reference
> > working for fixed-size arrays, ideally we can do the same for structs
> with
> > array members.
>
> Agreed.
>
> - Robert
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Fri, Jul 25, 2014 at 1:16 PM, Robert Bradshaw <
rober...@math.washington.edu> wrote:
> On Fri, Jul 25, 2014 at 8:24 AM, Kurt Smith wrote:
>
>>
>> On Thu, Jul 24, 2014 at 11:16 PM, Robert Bradshaw
>> wrote:
>>
>>> On Thu, Jul 24, 2014 at 7:13 PM, Kurt Smith wrote:
>>> >
>>> > On Thu, Jul 24, 2014 at 2:07 PM, Robert Bradshaw
>>> wrote:
>>> >>
>>> >> On Fri, Jul 18, 2014 at 12:44 PM, Robert Bradshaw >> >
>>> >> wrote:
>>> >> > On Fri, Jul 18, 2014 at 12:08 PM, Kurt Smith
>>> wrote:
>>> >> >> On Wed, Jul 16, 2014 at 1:02 PM, Robert Bradshaw <
>>> rober...@gmail.com>
>>> >> >> wrote:
>>> >> >>>
>>> >> >>>
>>> >> >>> Yes, this'd be nice to have. One difficulty with arrays is that
>>> they
>>> >> >>> can't be returned by value, and so the ordinary from_py_function
>>> >> >>> mechanism (which gets called recursively) would need to be
>>> adapted.
>>> >> >>> (Allowing to_py_function to be optinally be called by reference
>>> >> >>> instead of by value could be nice as well from a performance
>>> >> >>> standpoint.)
>>> >> >>
>>> >> >>
>>> >> >> OK, thanks for the pointers. I'll put some time on this over the
>>> >> >> weekend.
>>> >> >> Should I just make a PR when things are ready to review, or should
>>> I
>>> >> >> put up
>>> >> >> an issue first?
>>> >> >
>>> >> > I think this thread is sufficient; looking forward to a pull
>>> request.
>>> >>
>>> >> Don't know if you had time to look at this yet,
>>> >
>>> >
>>> > Yes, I've put in some time. Initial focus is getting
>>> >
>>> > cdef int a[10] = obj
>>> >
>>> > working, and then using that for structs.
>>>
>>> Sounds good.
>>>
>>> > Took a little while to re-orient myself with the codebase.
>>> >
>>> > I'm working on getting the `to_py_function` and `from_py_function`
>>> > infrastructure to take arguments by reference; right now I'm getting
>>> > something hacked into place, and I'd appreciate your review to point
>>> out the
>>> > right way (or at least a better way) to do it.
>>>
>>> from_py_function always takes its argument (a PyObject*) by reference.
>>> It's used as an rvalue, so might not make sense for arrays.
>>>
>>
>> My bad, I worded my response poorly -- what I mean is that currently the
>> generated code is something like:
>>
>> __pyx_v_t1 = xxx_from_py_function_xxx(temp_py_obj);
>>
>> ___pyx_v_a = __pyx_v_t1;
>>
>> where __pyx_v_a and __pyx_v_t1 are declared as fixed-size C arrays.
>>
>> This won't work, for reasons pointed out.
>>
>> So I'm trying to generate instead:
>>
>> err_code = xxx_from_py_function_xxx(temp_py_obj, &__pyx_v_a[0], 10);
>> if (err_code == -1) { ... }
>>
>
> FWIW think you can just write &__pyx_v_a
>
Yes, thanks, this form will also generalize to multidimensional fixed-sized
arrays. For that matter, we can just pass in __pyx_v_a, which is
equivalent to either &__pyx_v_a or &__pyx_v_a[0] (K&R 5.3).
>
>
>> Where the 10 is the array length. The function would initialize the
>> array internally. The python object is passed by reference as before, and
>> the array is also passed by reference. The array is assigned to just once,
>> minimizing copying. We can return a status code to propagate errors, etc.
>> This seems like the cleanest code to me. Thoughts?
>>
>
> Yes, that makes sense.
>
> One of my concerns was whether xxx_from_py_function_xxx was always
> assigned to an lvalue, enabling this transformation, but I think with error
> checking it's always assigned to a temporary, so we're OK here.
>
>
Sorry, I didn't quite follow that. The lhs is an lvalue at the C level
already, right? And we'd want to use the array variable directly in the
xxx_from_py_function_xxx call, not a temporary, since the `__pyx_v_a =
__pyx_v_t1` temp assignment won't work. What am I missing here?
Are you thinking of transforming all xxx_from_py_function_xxx to be of this
> form?
>
No, I was thinking this would kick in for just fixed-size arrays, structs,
and perhaps others as makes sense. Everything else would remain as-is.
> It could make sense for structs, and make for cleaner error checking, but
> would there be more overhead for simple types like converting to a long
> (which are currently macros)?
>
Agreed. Simple types should remain as-is.
> If the length is an argument, would one have to check for whether to pass
> this at every use of the from_py_function?
>
No, my thinking is that this transformation (and passing in the array size)
would only apply for fixed-size arrays.
> (Maybe that could be encapsulated away into a method on the type, but I
> think it's used in quite a few places.)
>
> All that being said, it'll be really nice to get this working.
>
> - Robert
>
>
> ___
> cython-devel mailing list
> cython-devel@python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Thu, Jul 24, 2014 at 9:13 PM, Kurt Smith wrote: > > I'm working on getting the `to_py_function` and `from_py_function` > infrastructure to take arguments by reference; right now I'm getting > something hacked into place, and I'd appreciate your review to point out > the right way (or at least a better way) to do it. > > Will likely have a PR for you by this weekend. > Update: all the pieces are coming together. I have sorted out all the code generation pieces and nearly have a minimal example working, and will have something for you to review soon. From what I can tell, Cython has some baked-in assumptions when it comes to assignment statements. Telling Cython that in this case, no, don't assign to something at the C level, instead generate a function call and pass in the lvalue by reference takes some doing. I tried to fit things in with Cython's type system and pipelines, but I have no doubt that it can be improved to do things the right way. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Mon, Jul 28, 2014 at 11:02 AM, Robert Bradshaw wrote: > On Sun, Jul 27, 2014 at 9:34 PM, Kurt Smith wrote: > > > I have sorted out all the code > > generation pieces and nearly have a minimal example working, and will > have > > something for you to review soon. From what I can tell, Cython has some > > baked-in assumptions when it comes to assignment statements. Telling > Cython > > that in this case, no, don't assign to something at the C level, instead > > generate a function call and pass in the lvalue by reference takes some > > doing. > > Yeah, I'm not too surprised about that. > > I have an initial version working for assigning a Python object to a fixed-size array. It's special cased to C ints and pretty messy in parts, but it gets the job done. I have some ideas on how to generalize the type. Like I said, I'll need your input on the right way to do some things. Will clean it up and put up a PR as soon as I can. Once this is in place, assigning a dict-like object to a struct can build on it. There are a number of other little things that would be nice, like allowing slice assignment: c_arr[5:13] = obj and initializing a dynamic C array with a Python object, provided the extent is specified: cdef int *arr = malloc(N * sizeof(int)) arr[:N] = obj ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
On Mon, Jul 28, 2014 at 11:29 PM, Kurt Smith wrote: > On Mon, Jul 28, 2014 at 11:02 AM, Robert Bradshaw > wrote: > >> On Sun, Jul 27, 2014 at 9:34 PM, Kurt Smith wrote: >> >> > I have sorted out all the code >> > generation pieces and nearly have a minimal example working, and will >> have >> > something for you to review soon. From what I can tell, Cython has some >> > baked-in assumptions when it comes to assignment statements. Telling >> Cython >> > that in this case, no, don't assign to something at the C level, instead >> > generate a function call and pass in the lvalue by reference takes some >> > doing. >> >> Yeah, I'm not too surprised about that. >> >> > I have an initial version working for assigning a Python object to a > fixed-size array. > https://github.com/cython/cython/pull/308 Do you prefer that we continue discussion on this thread, or on the PR? ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Automatic conversion with fixed-size C arrays
Apologies for the radio silence: the original project that motivated this work has a deadline, so I've had to set this aside for now. I fully intend to return to this once that's done. On Thu, Jul 31, 2014 at 11:34 PM, Robert Bradshaw wrote: > We can move to the PR, makes it easier to do line commenting. > > On Tue, Jul 29, 2014 at 9:54 PM, Kurt Smith wrote: > > On Mon, Jul 28, 2014 at 11:29 PM, Kurt Smith wrote: > >> > >> On Mon, Jul 28, 2014 at 11:02 AM, Robert Bradshaw > >> wrote: > >>> > >>> On Sun, Jul 27, 2014 at 9:34 PM, Kurt Smith > wrote: > >>> > >>> > I have sorted out all the code > >>> > generation pieces and nearly have a minimal example working, and will > >>> > have > >>> > something for you to review soon. From what I can tell, Cython has > >>> > some > >>> > baked-in assumptions when it comes to assignment statements. Telling > >>> > Cython > >>> > that in this case, no, don't assign to something at the C level, > >>> > instead > >>> > generate a function call and pass in the lvalue by reference takes > some > >>> > doing. > >>> > >>> Yeah, I'm not too surprised about that. > >>> > >> > >> I have an initial version working for assigning a Python object to a > >> fixed-size array. > > > > > > https://github.com/cython/cython/pull/308 > > > > Do you prefer that we continue discussion on this thread, or on the PR? > > > > ___ > > cython-devel mailing list > > cython-devel@python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
