[issue35034] Add closing and iteration to threading.Queue
New submission from Vladimir Filipović :
Code using threading.Queue often needs to coordinate a "work is finished as far
as far as I care" state between the producing and consuming side.
When going from the producer to the consumer ("No more items after this, so
don't bother waiting"), this is usually implemented with sentinel objects,
which is at best needlessly verbose and at worst tricky to get right (as with
multiple consumers, or communicating a non-trivial sentinel object).
When going the other direction ("I'm not interested in consuming any more, so
you can stop putting them on the queue"), or when a third component needs to
notify both sides ("You two start wrapping up, but don't drop any in-flight
items") there isn't even a clear usual solution.
Adding a close() method to the Queue (with accompanying exception treatment
etc.) would solve all of this in a very clean way. It would not change anything
for code that doesn't want to use it. It would simplify a lot of everyday uses
of Queue. Many simple producers could reduce their coordination code to a `with
closing(queue)` idiom. A further __iter__() method would enable many simple
consumers to safely cut all their coordination boilerplate down to just `for
item in queue`.
I've got a sample implementation ready for Queue and its children, and I'm
about to submit it as a PR.
I'd be happy to contribute an equivalent (within limits of its API promises)
implementation for SimpleQueue, but I don't have it written, and I'm unsure if
I should make a separate issue for that.
--
components: Library (Lib)
messages: 328181
nosy: hemflit
priority: normal
severity: normal
status: open
title: Add closing and iteration to threading.Queue
type: enhancement
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue35034>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35034] Add closing and iteration to threading.Queue
Change by Vladimir Filipović : -- keywords: +patch pull_requests: +9354 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35034] Add closing and iteration to threading.Queue
Change by Vladimir Filipović : -- keywords: +patch, patch pull_requests: +9354, 9355 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35034] Add closing and iteration to threading.Queue
Change by Vladimir Filipović : -- keywords: +patch, patch, patch pull_requests: +9354, 9355, 9357 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35034] Add closing and iteration to threading.Queue
Change by Vladimir Filipović : -- keywords: +patch, patch, patch, patch pull_requests: +9354, 9355, 9356, 9357 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue35034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35034] Add closing and iteration to threading.Queue
Vladimir Filipović added the comment: Hi Raymond! I've posted to python-ideas: https://mail.python.org/pipermail/python-ideas/2018-October/054238.html The amount of attention it got was modest, so I couldn't exactly say the community has thoroughly vetted and enthusiastically endorsed this proposal :) But all responses were in the narrow range of neutral to MILDLY positive. There weren't any objections at all to either the idea or the code. Nor did anybody question whether a feature of this kind should be added at all. I don't think I could realistically expect much more support than that for a change this minor. (Well, it would have been better if at least one person'd had an (uncontested) strongly positive response.) Some problems with other kinds of implementations were brought up, but this one isn't vulnerable to them. I believe I've also addressed all your comments from this BPO issue in the opening post there. An older discussion was linked, which had also shown tacit consensus that this type of feature in general would be welcome. I suppose it's time to decide if that level of response is enough. -- resolution: later -> status: closed -> open ___ Python tracker <https://bugs.python.org/issue35034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35034] Add closing and iteration to threading.Queue
Vladimir Filipović added the comment: Hi Raymond! Thanks for the attention you've given this and for the words of encouragement. I'll try to make something like this for PyPI. I do feel I should critique your metaphor of Queues as email. (Please don't take this as a request to re-evaluate anything about this proposal or re-hash the "farewell messages are an inferior solution" from python-ideas. It's really only about that "basis for mental model" in general, which I think is doing you a disservice.) Email is principally bidirectional; with Python's Queues, only one side has the freedom to send any kind of coordination instructions via the same transport that's used for workload messages. Email is principally one-to-one. The *-to-many Queue case is specifically the one where propagation of farewell messages to all recipients is error-prone, with recipients suddenly needing to worry about the existence of other recipients. (And if you feel that's not valid because you view multi-recipient as part of the core idea of email, consider that such email is basically "broadcast", with every message reaching every recipient - even further removed from the idea of Queues than single-recipient email is.) The email metaphor tends to imply FIFO consumption, while Queues support other orderings. These other orderings further complicate the code for dealing with farewell messages. With email, participants generally have handles on each other, and treat the transport as very abstract. With Queues it's exactly the opposite: they have a handle on the transport mechanism and abstract away the other participants; the transport explicitly takes over some burden of coordination and dispatch; and thanks to that it's very easy for participants to enter or leave an ongoing "conversation" without having to introduce themselves to the others. (I'm realizing now that email is a pretty good metaphor for Erlang's IPC model though.) -- ___ Python tracker <https://bugs.python.org/issue35034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
