[issue42752] multiprocessing Queue leaks a file descriptor associated with the pipe writer (#33081 still a problem)
New submission from Alex Orange : Didn't feel like necroing #33081, but this is basically that problem. The trouble is the cleanup that appeared to fix #33081 only kicks in once something has been put in the queue. So if for instance a Process function puts something in the queue and the parent gets it, then calls q.close() the writer on the parent side doesn't get culled until the object does. This is particularly a problem for PyPy and isn't exactly great for any weird corner cases if anyone holds onto Queue objects after they're closed for any reason (horders!). Attached file test_queue.py is an example of how to trigger this problem. Run it without a command line argument "python test_queue.py" and it won't crash (though it will take a very long time to complete). Run with an argument "python test_queue.py fail" and it will fail once you run out of file descriptors (one leaked per queue). My suggestion on how to handle this is to set self._close to something that will close self._writer. Then, when _start_thread is called, instead of directly passing the self._writer.close object, pass a small function that will switch out self._close to the Finalize method used later on and return self._writer. Finally, inside _feed, use this method to get the _writer object and wrap the outer while 1 with a contextlib.closer on this object. This is a fair bit of stitching things together here and there so let me know if anyone has any suggestions on this before I get started. -- components: Library (Lib) files: test_queue.py messages: 383832 nosy: Alex.Orange priority: normal severity: normal status: open title: multiprocessing Queue leaks a file descriptor associated with the pipe writer (#33081 still a problem) type: resource usage versions: Python 3.10, Python 3.6, Python 3.7, Python 3.8, Python 3.9 Added file: https://bugs.python.org/file49701/test_queue.py ___ Python tracker <https://bugs.python.org/issue42752> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42752] multiprocessing Queue leaks a file descriptor associated with the pipe writer (#33081 still a problem)
Alex Orange added the comment: Well, having not heard anything I decided to just make a patch and throw it up. Here it is. This includes a test that will fail with the old version and passes once patched as well as the patch to the queue code itself. Worth noting, the CleanExchange class is used because simpler things like using a closure to pass the exchange mechanism hold a reference to the Queue one way or another that is difficult/impossible to kill. This is because the intermediate thread mechanisms hold a reference to all the arguments that are passed to the run function. CleanExchange allows an indirect reference to be passed and for the reference to Queue to be None'd out. -- keywords: +patch Added file: https://bugs.python.org/file49713/queue_close_write.patch ___ Python tracker <https://bugs.python.org/issue42752> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17387] Error in C API documentation of PySequenceMethods
New submission from Alex Orange: The documentation at http://docs.python.org/2/c-api/typeobj.html#PySequenceMethods is missing sq_slice between sq_item and sq_ass_item. This will mess up anyone trying to use anything after sq_item (that isn't using designated initializers). -- assignee: docs@python components: Documentation messages: 183779 nosy: Alex.Orange, docs@python priority: normal severity: normal status: open title: Error in C API documentation of PySequenceMethods versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue17387> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17387] Error in C API documentation of PySequenceMethods
Alex Orange added the comment: If you look at the 2.7.3 version of that file: http://hg.python.org/cpython/file/70274d53c1dd/Include/object.h it has more information. It is a ssizessizeargfunc. I assume it passes the lower and upper bound and expects back a subsequence. -- ___ Python tracker <http://bugs.python.org/issue17387> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17387] Error in C API documentation of PySequenceMethods
Alex Orange added the comment: Just to clarify though, that is entirely an assumption as to how it's supposed to be used. -- ___ Python tracker <http://bugs.python.org/issue17387> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17387] Error in C API documentation of PySequenceMethods
Alex Orange added the comment: I must admit I'm a little new to the development side of things. Can someone point me at a repo or something that the documentation files are in? I'm sort of guessing that the html is the processed output of something. -- ___ Python tracker <http://bugs.python.org/issue17387> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
