On Tue, Sep 25, 2012 at 5:41 AM, Ian Bell <ian.h.b...@gmail.com> wrote: > I agree that some of the more powerful elements of Cython would be/are > difficult or impossible to pickle in a general way. I have a counter > use-case where auto-pickling would be very useful. > > I have cdef classes that have many (think 50-100) members of double and long > types, as well as other cdef classes that are also only using > Cython-friendly data types. I have implemented pickling for these classes > and it sucks. Trying to maintain the list of parameters that need to be > pickled is a major hassle. The biggest problem is that if you add add a > parameter to the class but forget to add it to the pickling function, you > end up with the parameter not getting reloaded on unpickling. This is not > good! Yes it is up the programmer to do this right, but I could use some > help. > > What I have ended up doing for my other classes is define a __cdict__ > function that returns a dictionary with the values and parameters as defined > in the cdef class. In this way, pickling and unpickling is a little bit > more pleasant. But it would be fantastic if the __cdict__ could be baked > into the class automatically. It would be fine for me if only the things > that could be easily handled would be returned, and a second list of > attributes that require more work to be serialized so that the user/coder > could be sure that all the attributes are being handled and none get > forgotten.
Exactly. Even if we handled only cdef classes with only (transitively) picklable fields, this would go a long way. The only question to have is how to handle changes in the class struct between pickling and unpickling. Eventually we could come up with syntax/mechanisms for manually specifying serialization of more complicated members like double*, but that's the kind of class you might have to write a __reduce__ method on anyways. > Also, the docs on pickling of cdef classes is very sparse, a simple example > would be very beneficial. The is one in the examples in the source, but it > is not easy to find Agreed. > On Tue, Sep 25, 2012 at 6:44 AM, Sturla Molden <stu...@molden.no> wrote: >> >> On 20.09.2012 21:11, Ian Bell wrote: >> >>> Auto-pickling would be tremendously helpful as pickling and unpickling >>> is one of the most annoying features of working with threads and >>> processes in python. >> >> >> How should Cython interfere how to pickle a C pointer? >> >> cdef class foobar: >> cdef double *data >> >> A C object can be anything. Cython does not know anything about size, >> offset or strides, or even if it's safe to take a copy. >> >> Example: How to pickle a shared memory buffer? Surely we cannot take a >> copy, because that would defeat the purpose of "shared" memory. And even if >> could take a copy, how many bytes should be copied? Do you think an >> autopickler could have figured this out? >> >> >> https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_sysv.pyx >> >> >> https://github.com/sturlamolden/sharedmem-numpy/blob/master/sharedmem/sharedmemory_win.pyx >> >> On yes, the code is different on Unix and Windows, something the >> auto-pickler could not possibly know either. >> >> Auto-pickling cdef classes is not doable, IMHO. >> >> >> And by the way, implementing a __reduce__ method manually is not very >> difficult either. >> >> >> Sturla Molden >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel@python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > > > _______________________________________________ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel > _______________________________________________ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
