[issue14288] Make iterators pickleable

Changes by Kristján Valur Jónsson : -- resolution: -> fixed status: open -> closed ___ Python tracker ___ ___ Python-bugs-list mailin

[issue14288] Make iterators pickleable

Roundup Robot added the comment: New changeset 4ff234337e24 by Kristján Valur Jónsson in branch 'default': Issue #14288: Serialization support for builtin iterators. http://hg.python.org/cpython/rev/4ff234337e24 New changeset 51c88d51aa4a by Kristján Valur Jónsson in branch 'default': Issue #14

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: Good idea Antoine. So, I'll with your suggested fix to the unittests I'll commit this and then look on while Rome burns. -- ___ Python tracker ___

[issue14288] Make iterators pickleable

Antoine Pitrou added the comment: > Now, how important do you think type consistency is? when using > iterators, does one ever look at it and test its type? if this is > important, I _could_ take another look at dicts and seta and create > fresh iterators to the dicts and sets made out of the

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: I've incorporated antoine's comments and the proposed internal function name into a new patch. A lot of the changes concerned thecking the type() of the unpickled iterator. Now, it wasn't a specific design goal to get the exact same objects back, onl

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: Btw, regarding compatibility: The docs say " The pickle serialization format is guaranteed to be backwards compatible across Python releases." I take this to mean the serialization format itself. I don't think there is a broader guarantee that pickl

[issue14288] Make iterators pickleable

Antoine Pitrou added the comment: Well, please take a look at the review link. There are already some comments there. -- ___ Python tracker ___

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: Raymond had already reviewed it, and sbt. I wasn't aware of any more pending reviews, but I'll wait for yours, of course. -- ___ Python tracker _

[issue14288] Make iterators pickleable

Antoine Pitrou added the comment: > Okay, I'll go ahead, fix the 'iter()' trick api name and apply the > patch. Then we'll see what happens :). Please wait for reviews. -- ___ Python tracker

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: Okay, I'll go ahead, fix the 'iter()' trick api name and apply the patch. Then we'll see what happens :). Any suggestion towards what documentation changes are needed? I don't think the list of pickleable objects is made explicit anywhere, it is larg

[issue14288] Make iterators pickleable

Changes by Raymond Hettinger : -- assignee: rhettinger -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue14288] Make iterators pickleable

sbt added the comment: > If you look at the patch it isn't (or shouldn't be). Sorry. I misunderstood when Raymond said "running the iterator to completion". -- ___ Python tracker

[issue14288] Make iterators pickleable

Antoine Pitrou added the comment: > The thing to emphasise here is that pickling an iterator is > "destructive": afterwards the original iterator will be "empty". If you look at the patch it isn't (or shouldn't be). I agree with Raymond that accumulating dict and set iterators in a list is a

[issue14288] Make iterators pickleable

sbt added the comment: > ... and that pickling things like dict iterators entail running the > iterator to completion and storing all of the results in a list. The thing to emphasise here is that pickling an iterator is "destructive": afterwards the original iterator will be "empty". I can't

[issue14288] Make iterators pickleable

Raymond Hettinger added the comment: Michael, thanks for the link. The email was clearer about its rationale than was listed here. When this patch gets applied, any discussion of it in the docs should be clear that generators aren't included and that pickling things like dict iterators enta

[issue14288] Make iterators pickleable

Michael Foord added the comment: Yes there was a discussion on python-dev. Various people spoke in favour, no-one against: http://mail.python.org/pipermail/python-dev/2012-March/117566.html -- ___ Python tracker

[issue14288] Make iterators pickleable

Martin v. Löwis added the comment: I think the "worth doing" argument doesn't really hold, given that it's done. The question at hand really is a) is the patch correct? b) can we commit to maintaining it, even as things around it may change? I'm not bothered with the patch being potentially in

[issue14288] Make iterators pickleable

Raymond Hettinger added the comment: Has python-dev discussion been launched? It is far from clear that this is worth doing. Pickling runtime structures may be a normal use case for Stackless but isn't a normal use case for regular Python. Also, it seems pointless to start down this path

[issue14288] Make iterators pickleable

Changes by Antoine Pitrou : -- nosy: +pitrou ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: sbt, I will fix the api name. Any other objections then? Leave it as it is with the "iter()" trick? -- versions: +Python 3.3 -Python 3.4 ___ Python tracker

[issue14288] Make iterators pickleable

Changes by Raymond Hettinger : -- assignee: -> rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: another trick has been suggested. For hidden iterator objects, such as stringiter, to actually put them in the "types" module. in there, we could do something like: #types.py stringiter = iter('').__class__ and we would then change the name of the ite

[issue14288] Make iterators pickleable

sbt added the comment: I think PyAPI_FUNC(PyObject *) _PyIter_GetIter(const char *iter); has a confusing name for a convenience function which retrieves an attribute from the builtin module by name. Not sure what would be better. Maybe _PyIter_GetBuiltin(). -- nosy: +sbt _

[issue14288] Make iterators pickleable

Changes by Raymond Hettinger : -- Removed message: http://bugs.python.org/msg155759 ___ Python tracker ___ ___ Python-bugs-list mailin

[issue14288] Make iterators pickleable

Raymond Hettinger added the comment: The dict iterators depend on the order of the dict being the same when unpickled on another python (the order will vary depending on dummy entries, insertion order, 32 vs 64 bit builds, salted hashes, etc). Sets have the same issue -- it doesn't seem poss

[issue14288] Make iterators pickleable

Georg Brandl added the comment: The "review" link next to the the patch file entry should already work and provide a nice visual diff + commenting interface. -- nosy: +georg.brandl ___ Python tracker

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: Btw, is there some way I can make this patch easier to review? I haven't contributed much since the Hg switchover, can I make it so that people can do visual diff? -- ___ Python tracker

[issue14288] Make iterators pickleable

Raymond Hettinger added the comment: The patch looks fine. If python-dev thinks that making-iterators-picklable is worth doing, I would support this going into Python 3.3 (no extra benefit will come from waiting). -- ___ Python tracker

[issue14288] Make iterators pickleable

Kristján Valur Jónsson added the comment: Sure, I'll start a discussion there, but at least I've gotten the patch in. The patch is smaller than it looks, most of it is tests. -- ___ Python tracker __

[issue14288] Make iterators pickleable

Raymond Hettinger added the comment: ISTM that a discussion on python-dev would be of value here. Iterators are a protocol, not a class; hence, the effort to make them all picklable is potentially endless. The effort would also always be incomplete because some iterators are difficult or in

[issue14288] Make iterators pickleable

Changes by Jesús Cea Avión : -- nosy: +jcea ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue14288] Make iterators pickleable

New submission from Kristján Valur Jónsson : A common theme in many talks last year about cloud computing was the need to suspend execution, pickle state, and resume it on a different node. This patch is the result of last year's stackless sprint at pycon, finally completed and submitted for