[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: The idea is to pickle the object immediately (i.e. when added) instead of when requested (dequeued). This means that any operations (even deletion) performed on the original object do not make changes to the item in the queue, before it has been dequeued. Whether this is an acceptable solution I don't know.. it's just my proposal. The attached queue_example.py script illustrates what I mean. Regards, VT -- ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Please find attached an updated patch (including unit test) for the release27-maint branch. I've run the test_multiprocessing suite against svn revision 86129 of said branch. -- Added file: http://bugs.python.org/file19472/patch_27maint.diff ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
New submission from Vilnis Termanis : If an object, which as been put() in the multiprocessing.Queue is changed immediately after the put() call then changed version may be added to the queue which I assume is not the expected behaviour: >>> from multiprocessing import Queue >>> q = Queue() >>> obj = [[i for i in xrange(j * 10, (j * 10) + 10)] for j in xrange(0,10)] >>> q.put(obj); obj[-1][-1] = None >>> obj2 = q.get() >>> print obj2[-1][-1] None Note: This also happens if the queue is called form a child process like in the attached example. -- components: Library (Lib) files: queue_example.py messages: 100278 nosy: vilnis.termanis severity: normal status: open title: multiprocessing.Queue's put() not atomic thread wise type: behavior versions: Python 2.6 Added file: http://bugs.python.org/file16409/queue_example.py ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: More notes: - Reason for non-atmoic behaviour is due to queue _feeder thread sending the object after the put() method has already exited. Hence changing the object straight after put() can result in the modified object being sent instead. - Reproduced under Win32 and Ubuntu-64 with v2.6.4 - Not tried with v3.* but looking at the ToT code it might also be affected - The attached patch makes put() atomic but I am completely unsure about whether this is the correct way to address the problem (if it's seen as one). Regards, VT -- keywords: +patch Added file: http://bugs.python.org/file16410/queues.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Quick though (last one I promise): Maybe put() could have an argument to indicate whether to be thread-blocking (i.e. guaranteeing the object to be sent in its current state) or not. The (current) non-thread-blocking mode will have better performance for the put()-calling process if there are a lot of processes trying to write to the queue at once since it can continue regardless (while the _feed thread waits for the internal Pipe to become available). -- ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Hi Meador, I apologise: I really shouldn't have called it a patch. It's just to show one way that I've found fixes the problem (but presumably reduces performance in other cases, hence the suggestion & wait for feedback on the issue). I'm not really experienced with Python (nor am I an experienced / particularly good developer in general) so it's probably safer to treat me just as a regular user who stumbled upon a problem. :) (I'm definitely not shying away from "ticking all the boxes" but I don't see that there is any point in me trying to yet.) Regards, VT PS: I've re-attached the example "fix" with the diff the right way round (sorry). -- Added file: http://bugs.python.org/file16412/lock_suggestion.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: I couldn't see the wood for the trees: If put() is thread-blocking, multiprocessing.Queue is reduced to the same functionality as multiprocessing.queues.SimpleQueue, if I'm not mistaken. So maybe there should be a warning in the documentation that, for multiprocessing.[Joinable]Queue, modifying obj straight after calling put(obj) might en-queue the modified version. To me at least this wasn't obvious until I looked at the multiprocessing.queue code. I've modified the example for clarity and retracted the (unworthy) patch attempt. Regards, VT -- ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16412/lock_suggestion.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16409/queue_example.py ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Added file: http://bugs.python.org/file16413/queue_example.py ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Alternative suggestion (since SimpleQueue doesn't provide buffering): Allow option to force immediate pickling of the object to be en-queued, i.e. pickling when adding to internal buffer instead of as part of Connection.send() in _feed thread. Does this sound like a reasonable solution? (I'll do my best to write unit tests etc. as per submission guidelines, if so.) Regards, VT -- Added file: http://bugs.python.org/file16414/pickle_suggestion.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Performance comparison before/after suggested change (adding/removing 1000-item list 1 times). Inline 1.926 1.919 1.907 (bufferPickled = True) 2.138 2.379 2.304 (bufferPickled = True) 1.158 1.151 1.141 (bufferPickled = True) -- Added file: http://bugs.python.org/file16415/pickle_suggestion_v2.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16414/pickle_suggestion.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Added file: http://bugs.python.org/file16416/queue_perf.py ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8056] Piped parent's multiprocessing.Process children cannot write to stdout
New submission from Vilnis Termanis : Affects Win32 only (tested under Ubuntu 9.10-64 and XP-32 with v2.6.4). If script output is piped, child processes created via multiprocessing.Process cannot write to stdout. Also, trying to call stdout.flush() in child processes causes IOError. Normal behaviour C:\>stdout.py [Parent] stdout: ', mode 'w' at 0x00A54070> [Parent] message via stdout [Child] stdout: ', mode 'w' at 0x00A54070> [Child] message via stdout Piped behaviour (same result if redirecting to file) --- C:\>stdout.py | cat [Parent] stdout: ', mode 'w' at 0x00A54070> [Parent] message via stdout [Child] stdout: ', mode 'w' at 0x00A54070> Process Process-1: Traceback (most recent call last): File "C:\Python26\lib\multiprocessing\process.py", line 232, in _bootstrap self.run() File "C:\Python26\lib\multiprocessing\process.py", line 88, in run self._target(*self._args, **self._kwargs) File "C:\stdout.py", line 18, in child stdout.flush() IOError: [Errno 9] Bad file descriptor -- components: Library (Lib), Windows files: stdout.py messages: 100390 nosy: vilnis.termanis severity: normal status: open title: Piped parent's multiprocessing.Process children cannot write to stdout type: behavior versions: Python 2.6 Added file: http://bugs.python.org/file16432/stdout.py ___ Python tracker <http://bugs.python.org/issue8056> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16473/queues.diff ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Updated patch to include new test (now against release26-maint branch). Verified test_multiprocessing suite passes before (apart from new test) & after change. (Tested under Ubuntu 9.10 64-bit) -- Added file: http://bugs.python.org/file16473/queues.diff ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16415/pickle_suggestion_v2.patch ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Vilnis Termanis added the comment: Updated patch to include new test (now against release26-maint branch). Verified test_multiprocessing suite passes before (apart from new test) & after change. (Tested under Ubuntu 9.10 64-bit) -- Added file: http://bugs.python.org/file16472/queues.diff ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Removed file: http://bugs.python.org/file16472/queues.diff ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : Added file: http://bugs.python.org/file16475/queues.diff ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8056] Piped parent's multiprocessing.Process children cannot write to stdout
Vilnis Termanis added the comment: I tried to reproduce / narrow-down the cause of this with a debug build in a VM but couldn't reproduce the behaviour (neither with debug nor with standard 2.6.4 binary). I have to conclude that there is something perculiar with my native Windows installation (since in both VM and natively Python is using exactly the same DLLs and ). Apologies for wasting your time. -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue8056> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue8037] multiprocessing.Queue's put() not atomic thread wise
Changes by Vilnis Termanis : -- versions: +Python 3.1 ___ Python tracker <http://bugs.python.org/issue8037> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28422] multiprocessing Manager mutable type member access failure
Vilnis Termanis added the comment: If you un-comment the print_exc() call, you'll see that it still fails - for queue.Queue: Manager failure (for Queue) Traceback (most recent call last): File "fish.py", line 74, in main add_type_in_own_process(mgr, type_name) File "fish.py", line 61, in add_type_in_own_process use_manager(obj, type_name, mgr=mgr) File "fish.py", line 35, in use_manager obj.append(getattr(mgr, type_name)()) File "", line 2, in append File "/usr/lib/python3.8/multiprocessing/managers.py", line 850, in _callmethod raise convert_to_error(kind, result) multiprocessing.managers.RemoteError: --- Traceback (most recent call last): File "/usr/lib/python3.8/multiprocessing/managers.py", line 243, in serve_client request = recv() File "/usr/lib/python3.8/multiprocessing/connection.py", line 251, in recv return _ForkingPickler.loads(buf.getbuffer()) File "/usr/lib/python3.8/multiprocessing/managers.py", line 959, in RebuildProxy return func(token, serializer, incref=incref, **kwds) TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned' --- -- resolution: works for me -> status: pending -> open ___ Python tracker <https://bugs.python.org/issue28422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28422] multiprocessing Manager mutable type member access failure
Vilnis Termanis added the comment: Apologies, my mistake - it does indeed work with 3.9 & 3.10. Feel free to close. -- resolution: -> works for me status: open -> pending versions: +Python 3.8 ___ Python tracker <https://bugs.python.org/issue28422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25469] multiprocessing .Condition.notify(_all) function has O(N) time complexity where N is the number of wait() calls with a timeout since the last notify(_all) call
Vilnis Termanis added the comment: A few additional notes: 1) On an AMD E-450 under Ubuntu 14.04 x64 with a current in-development build, when running condition_test.py, Condition.notify_all() takes (measure using time.clock) 0.0003 with the patch and 0.3219 without it. Additionally the in-script timeit calls seem to suggest that the patch does not adversely affect Condition.wait(). 2) Unit tests have also been run against test_multiprocessing_forkserver (typo in original message). 3) A scenario where this might become apparent would be to have many worker processes which have a loop along the lines of: while not event.is_set(): do_more_work() # Pause because the task is only supposed to happen every X seconds # or a wait is in effect due to an error. Do not want to use # time.sleep so loop is exited promptly rather than blindly sleeping. event.wait(some_amount) other_work_or_shutdown() If many processes call event.wait() with a timeout and the event is not set often (e.g. since it's a shutdown flag), a long-running program with many worker processes can end up having event.set() take a considerable amount of time. -- ___ Python tracker <http://bugs.python.org/issue25469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25469] multiprocessing .Condition.notify(_all) function has O(N) time complexity where N is the number of wait() calls with a timeout since the last notify(_all) call
Changes by Vilnis Termanis : Added file: http://bugs.python.org/file40854/condition_test.py ___ Python tracker <http://bugs.python.org/issue25469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25469] multiprocessing .Condition.notify(_all) function has O(N) time complexity where N is the number of wait() calls with a timeout since the last notify(_all) call
New submission from Vilnis Termanis: multiprocessing's Condition uses a couple of semaphores to keep track of processes which are sleeping and have woken up whilst waiting for the condition. These counters however are only ever decremented in the notify(_all) functions, via a loop which results in somewhat unexpected behaviour where said functions take longer to run than expected. The proposed patch simplifies notify(_all) functions such that time complexity is still O(N) but crucially N is the number of currently sleeping processes only (rather than number of wait() calls since last notify(_all) call). Note: This also affects Event.wait() & Event.set() in the same fashion since a Condition is used under the hood. I've attached mp_sync_condition.patch based on "in-development" branch as of 98840:2028aed246c0. I have run unit tests on said commit (via debug build) using: ./python -bb -E -Wd -m test -r -v -uall\ test_multiprocessing_fork\ test_multiprocessing_fork\ server test_multiprocessing_spawn Additionally I've run condition_test.py before & after to illustrate performance of the proposed change as well as demonstrate the issue. -- components: Library (Lib) files: mp_sync_condition.patch keywords: patch messages: 253403 nosy: vilnis.termanis priority: normal severity: normal status: open title: multiprocessing .Condition.notify(_all) function has O(N) time complexity where N is the number of wait() calls with a timeout since the last notify(_all) call type: performance versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file40853/mp_sync_condition.patch ___ Python tracker <http://bugs.python.org/issue25469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25469] multiprocessing .Condition.notify(_all) function has O(N) time complexity where N is the number of wait() calls with a timeout since the last notify(_all) call
Vilnis Termanis added the comment: I realise that having type bug type of "performance" means it's less likely to be looked at than if I had marked is as "behaviour" or "resource usage" (which would not be completely unfair, I think). Also, the patch should be applicable to 2.7 and 3.2 through to 3.5 (despite versions having been removed from the bug report). -- ___ Python tracker <http://bugs.python.org/issue25469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25469] multiprocessing .Condition.notify(_all) function has O(N) time complexity where N is the number of wait() calls with a timeout since the last notify(_all) call
Vilnis Termanis added the comment: I've added a regression test for the proposed patch along the lines of the example script (i.e. fails before and passes with patch). Apologies if the test is a bit clumsy - maybe there is a more elegant way? -- Added file: http://bugs.python.org/file40958/mp_sync_condition_with_test.patch ___ Python tracker <http://bugs.python.org/issue25469> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28422] multiprocessing Manager mutable type member access failure
New submission from Vilnis Termanis:
Accessing some Manager types (e.g. Lock) through a list, dict or Namespace
proxy is not possible if both the mutable container (e.g. list) and contained
type instance (e.g. Lock) have been created in the same process.
Symptoms:
In accessing process:
multiprocessing.managers.RemoteError on access, e.g.:
Unserializable message: ('#RETURN', )
.. and in underlying manager:
_pickle.PicklingError: Can't pickle : attribute lookup
lock on _thread failed
The provided test script performs:
0) Initialise SyncManager (via multiprocessing.Manager())
1) Create list proxy through manager
2) Insert another proxied type into the list (e.g. Lock)
3) Try to access type instance from (2) via container created in (1)
Note: When step (2) is run in a child process, everything work as expected.
When all steps execute in the same process, one gets the aforementioned
exception.
See also:
https://mail.python.org/pipermail/python-list/2009-September/552988.html
--
components: Library (Lib)
files: manager_pickling.py
messages: 278529
nosy: vilnis.termanis
priority: normal
severity: normal
status: open
title: multiprocessing Manager mutable type member access failure
versions: Python 2.7, Python 3.5
Added file: http://bugs.python.org/file45066/manager_pickling.py
___
Python tracker
<http://bugs.python.org/issue28422>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28422] multiprocessing Manager mutable type member access failure
Changes by Vilnis Termanis : -- type: -> crash ___ Python tracker <http://bugs.python.org/issue28422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28422] multiprocessing Manager mutable type member access failure
Changes by Vilnis Termanis : -- type: crash -> behavior ___ Python tracker <http://bugs.python.org/issue28422> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
