[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

STINNER Victor added the comment: We started to get *random* failures of test_queue_feeder_donot_stop_onexc() of test_multiprocessing_spawn since a few days. I may be related to this change. Can you please take a look at bpo-30595? -- nosy: +haypo _

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Thomas Moreau added the comment: I think this is a good solution as it let the user define easily the behavior it needs in other situation too. I would recommend adding the object responsible for the failure to the _on_queue_thread_error callback. This would simplify the error handling. @@ -

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: Thomas, thanks for the heads up. I would suggest something like the following patch to multiprocessing.Pool: $ git diff diff --git a/Lib/multiprocessing/queues.py b/Lib/multiprocessing/queues.py index 7f77837..ebbb360 100644 --- a/Lib/multiprocessing/queues.py

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Thomas Moreau : -- pull_requests: +2001 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Thomas Moreau added the comment: This fix, while preventing the Queue to crash, does not give any way to programatically detect that the message was dropped. This is a problem as we can no longer assume that the Queue will not drop messages. For instance, we can no longer detect deadlocks in

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: This is committed and pushed, thank you! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: New changeset bdd964710deffe8593063dcb63157e5b55a82c61 by Antoine Pitrou in branch '2.7': [2.7] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc (GH-1683) (#1817) https://github.com/python/cpython/commit/bdd964710deffe8593063dcb63157e5

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: New changeset 89004d761361332314beb08b443bff5b092ec36e by Antoine Pitrou in branch '3.5': [3.5] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc (GH-1683) (#1816) https://github.com/python/cpython/commit/89004d761361332314beb08b443bff5

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: New changeset 2783cc42629b9445ea848ce36bbf213ef7789271 by Antoine Pitrou in branch '3.6': [3.6] bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc (GH-1683) (#1815) https://github.com/python/cpython/commit/2783cc42629b9445ea848ce36bbf213

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Antoine Pitrou : -- pull_requests: +1902 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Antoine Pitrou : -- pull_requests: +1901 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Antoine Pitrou : -- pull_requests: +1900 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: New changeset bc50f03db4f58c869b78e98468e374d7e61f1227 by Antoine Pitrou (grzgrzgrz3) in branch 'master': bpo-30414: multiprocessing.Queue._feed do not break from main loop on exc (#1683) https://github.com/python/cpython/commit/bc50f03db4f58c869b78e98468e374d

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: Nevermind, I saw the PR and the test case. -- ___ Python tracker ___ ___ Python-bugs-list mailing li

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Antoine Pitrou : -- stage: -> patch review type: -> behavior ___ Python tracker ___ ___ Python-bugs-list mailing list Uns

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Antoine Pitrou added the comment: Can you expand on which exceptions you are getting in the feeder thread? -- nosy: +pitrou ___ Python tracker ___ ___

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Xiang Zhang : -- nosy: +davin, xiang.zhang ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://m

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

Changes by Grzegorz Grzywacz : -- pull_requests: +1778 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://

[issue30414] multiprocesing.Queue silently ignore messages after exc in _feeder

New submission from Grzegorz Grzywacz: multiprocessing.Queue is running background thread feeder. Feeder serialize and sends buffered data to pipe. The issue is with exception handling, feeder is catching all exceptions but out of main loop, so after exception is handled feeder is not going b