[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file26963/readall-combined.patch ___ Python tracker ___ ___ Python-bugs-list

[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file26959/readall-resize.patch ___ Python tracker ___ ___ Python-bugs-list m

[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file26960/readall-chunks.patch ___ Python tracker ___ ___ Python-bugs-list m

[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by Richard Oudkerk : Removed file: http://bugs.python.org/file26953/popen_communicate.patch ___ Python tracker ___ ___ Python-bugs-lis

[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by Ralf Schmitt : -- nosy: +schmir ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue15758] 500x speed up for Popen.communicate() on Windows

STINNER Victor added the comment: I'm lost, there are too much patches. Could you please remove old/useless patches? Patching the subprocess module is not a good solution: if there is an issue in the io module, the io module should be fixed, and so this patch can be removed. -- _

[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by STINNER Victor : -- nosy: +haypo ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue15758] 500x speed up for Popen.communicate() on Windows

Antoine Pitrou added the comment: If new_buffersize() is only used with HAVE_FSTAT, then its code can probably be simplified. -- ___ Python tracker ___ _

[issue15758] 500x speed up for Popen.communicate() on Windows

Richard Oudkerk added the comment: > It seems we could do both (fix the resizing logic, and fallback on > chunks if the size is unknown). Combined patch attached. -- Added file: http://bugs.python.org/file26963/readall-combined.patch ___ Python track

[issue15758] 500x speed up for Popen.communicate() on Windows

Antoine Pitrou added the comment: > Alternative patch (readall-chunks.patch) which delegates to > RawIOBase.readall() if the file cannot be stat-ed or appears to have > size zero. It seems we could do both (fix the resizing logic, and fallback on chunks if the size is unknown). --

[issue15758] 500x speed up for Popen.communicate() on Windows

Richard Oudkerk added the comment: Alternative patch (readall-chunks.patch) which delegates to RawIOBase.readall() if the file cannot be stat-ed or appears to have size zero. amount = 1 MB; time taken = 0.02 secs; rate = 64.10 MB/s amount = 2 MB; time taken = 0.02 secs; rate = 128.21 MB/s amoun

[issue15758] 500x speed up for Popen.communicate() on Windows

Richard Oudkerk added the comment: The attached patch (readall-resize.patch) makes the resizes only happen when the buffer is full. It produces amount = 1 MB; time taken = 0.02 secs; rate = 64.10 MB/s amount = 2 MB; time taken = 0.03 secs; rate = 64.10 MB/s amount = 4 MB; time taken = 0.06 sec

[issue15758] 500x speed up for Popen.communicate() on Windows

Richard Oudkerk added the comment: In each loop before calling read() the buffer size is recalculated based on the amount of space used, causing a realloc *regardless* of how much empty space is left in the buffer. And each read is only producing a smallish chunk (5120 bytes). So assuming re

[issue15758] 500x speed up for Popen.communicate() on Windows

Richard Oudkerk added the comment: > FileIO.readall() already has an overallocation mechanism which should > yield linear complexity. Perhaps it needs to be tweaked a bit? > (look at new_buffersize in Modules/_io/fileio.c) I think it needs a bit more than a tweak;-) Looks like it increases by

[issue15758] 500x speed up for Popen.communicate() on Windows

Antoine Pitrou added the comment: FileIO.readall() already has an overallocation mechanism which should yield linear complexity. Perhaps it needs to be tweaked a bit? (look at new_buffersize in Modules/_io/fileio.c) By the way, your results are bit weird. Why does the data rate increase with t

[issue15758] 500x speed up for Popen.communicate() on Windows

Richard Oudkerk added the comment: RawIOBase.readall() does the sensible thing already. Maybe FileIO should be allowed to inherit it. The alternative patch (which probably only works for raw unbuffered case) diff -r ca54c27a9045 Lib/subprocess.py --- a/Lib/subprocess.py Tue Aug 21 14:54:22 20

[issue15758] 500x speed up for Popen.communicate() on Windows

Antoine Pitrou added the comment: Yes, I think FileIO.readall() should be fixed rather than avoided. -- nosy: +pitrou versions: +Python 3.4 ___ Python tracker ___ ___

[issue15758] 500x speed up for Popen.communicate() on Windows

Changes by Richard Oudkerk : -- keywords: +patch Added file: http://bugs.python.org/file26953/popen_communicate.patch ___ Python tracker ___ _

[issue15758] 500x speed up for Popen.communicate() on Windows

New submission from Richard Oudkerk: Piping significant amounts of data through a subprocess using Popen.communicate() is crazily slow on Windows. The attached program just pushes data through mingw's cat.exe. Python 3.3: amount = 1 MB; time taken = 0.07 secs; rate = 13.51 MB/s amount = 2 MB;