New submission from Vyacheslav Grigoryev:
I'm creating a master stand-alone module on Python which should run some
children via subprocess module. Working with children is done in separate
worker threads. Additionally I need to receive real-time output from a child so
in a worker thread I also create reader helper thread which reads on a pipe
from the child.
All is working correctly while I have only one worker thread and run only one
child. When the child finishes the reader helper thread gets EOF and exits. But
when I have two worker threads and run two children, a pipe from early child
doesn't see EOF until the second child is finished. Though they are completely
unrelated.
Let's take a look on simplest example for reproducing the problem. There is a
simplest child:
--
import time, sys
# first arg is an ID. Second arg is how long to work in seconds
sys.stdout.write("start slave %s\n" % sys.argv[1])
sys.stdout.flush()
time.sleep(int(sys.argv[2]))
sys.stdout.write("finish slave %s\n" % sys.argv[1])
sys.stdout.flush()
--
And there is a master module:
--
import subprocess, sys, os, threading, time
g_logLock = threading.Lock()
def log(msg):
with g_logLock:
t = time.time()
print "%s.%03d %-5s %s" % \
(time.strftime('%H:%M:%S', time.localtime(t)), int((t - t // 1) *
1000), threading.currentThread().name, msg)
def thread1Proc():
def reader(stdout):
while True:
line = stdout.readline()
if not line:
break
log('slave said: %s' % line.strip())
log('finish slave reader thread')
log('thread 1 started')
timeToWork = '1'
util = subprocess.Popen((sys.executable, 'slave.py', '1', timeToWork),
stdout=subprocess.PIPE)
readerT = threading.Thread(target=reader, args=(util.stdout,), name='t1-r')
readerT.start()
log('slave 1 returned %d' % util.wait())
readerT.join()
log('thread 1 finished')
def thread2Proc():
log('thread 2 started')
timeToWork = '3'
util = subprocess.Popen((sys.executable, 'slave.py', '2', timeToWork))
log('slave 2 returned %d' % util.wait())
log('thread 2 finished')
#---
log('starting test')
threads = (threading.Thread(target=thread1Proc, name='t1'),
threading.Thread(target=thread2Proc, name='t2'))
for t in threads:
t.start()
for t in threads:
t.join()
log('finished test')
--
Here is what I see on the output (note - slave 1 outputs to the master via
pipe, while slave 2 outputs to a console because its output is not redirected):
>master.py
08:57:31.342 MainThread starting test
08:57:31.342 t1thread 1 started
08:57:31.342 t2thread 2 started
08:57:31.405 t1-r slave said: start slave 1
start slave 2
08:57:32.420 t1-r slave said: finish slave 1
08:57:32.420 t1slave 1 returned 0
finish slave 2
08:57:34.415 t1-r finish slave reader thread
08:57:34.415 t2slave 2 returned 0
08:57:34.415 t1thread 1 finished
08:57:34.431 t2thread 2 finished
08:57:34.431 MainThread finished test
Here you can see that even if the slave 1 finishes at 32.420, its reader thread
receives EOF and exits only when the slave 2 finishes also - at 34.415 (slave 1
works 1 second, slave 2 - 3 seconds).
Why the reader thread doesn't see EOF just in time?
The issue is reproduced in Python 2.7.12 x86 on Windows 7. On Linux Ubuntu
16.04 with system Python 2.7 all works as expected.
--
components: Library (Lib)
messages: 278809
nosy: Vyacheslav Grigoryev
priority: normal
severity: normal
status: open
title: subprocess pipe can't see EOF from a child in case of a few children run
with subprocess
type: behavior
versions: Python 2.7
___
Python tracker
<http://bugs.python.org/issue28462>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
