[issue10482] subprocess and deadlock avoidance

Change by Mark Lawrence : -- nosy: -BreamoreBoy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail

[issue10482] subprocess and deadlock avoidance

Martin Panter added the comment: Related: Issue 1260171, essentially proposing streaming readers and writers for communicate() instead of fixed buffers, but without using OS threads. -- ___ Python tracker

[issue10482] subprocess and deadlock avoidance

Changes by Akira Li <4kir4...@gmail.com>: -- nosy: +akira ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https

[issue10482] subprocess and deadlock avoidance

Changes by Martin Panter : -- nosy: +vadmium ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue10482] subprocess and deadlock avoidance

Mark Lawrence added the comment: @Glenn can you provide a formal patch so we can take this forward? -- nosy: +BreamoreBoy versions: +Python 3.5 -Python 3.4 ___ Python tracker ___

[issue10482] subprocess and deadlock avoidance

Changes by Christian Heimes : -- stage: -> needs patch versions: +Python 3.4 -Python 3.3 ___ Python tracker ___ ___ Python-bugs-list

[issue10482] subprocess and deadlock avoidance

Changes by Jon Brandvein : -- nosy: +brandj ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue10482] subprocess and deadlock avoidance

Changes by Ross Lagerwall : -- nosy: +rosslagerwall ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue10482] subprocess and deadlock avoidance

Nick Coghlan added the comment: Or various incarnations of functools.partial applied to subprocess.Popen. -- ___ Python tracker ___ _

[issue10482] subprocess and deadlock avoidance

Glenn Linderman added the comment: Looking at the code the way I've used it in my modified server.py: stderr = [] stderr_thread = threading.Thread(target=self._readerthread, args=(p.stderr, stderr)) stderr_thread.d

[issue10482] subprocess and deadlock avoidance

Nick Coghlan added the comment: The general idea is sound. My work colleagues have certainly had to implement their own reader/writer thread equivalents to keep subprocess from blocking. It makes sense to provide more robust public support for such techniques in process itself. -- no

[issue10482] subprocess and deadlock avoidance

Glenn Linderman added the comment: Sorry, left some extraneous code in the last message, here is the right code: def _writerthread(self, fhr, fhw, length=None): if length is None: flag = True while flag: buf = fhr.read( 512 ) f

[issue10482] subprocess and deadlock avoidance

Glenn Linderman added the comment: Here's an updated _writerthread idea that handles more cases: def _writerthread(self, fhr, fhw, length=None): if length is None: flag = True while flag: buf = fhr.read( 512 ) fhw.write( buf )

[issue10482] subprocess and deadlock avoidance

Glenn Linderman added the comment: So I've experimented a bit, and it looks like simply exposing ._readerthread as an external API would handle the buffered case for stdout or stderr. For http.server CGI scripts, I think it is fine to buffer stderr, as it should not be a high-volume channel.

[issue10482] subprocess and deadlock avoidance

New submission from Glenn Linderman : .communicate is a nice API for programs that produce little output, and can be buffered. While that may cover a wide range of uses, it doesn't cover launching CGI programs, such as is done in http.server. Now there are nice warnings about that issue in t