[issue1731717] race condition in subprocess module
Tom Culliton added the comment: This or some variant also shows up with scons (http://scons.tigris.org/issues/show_bug.cgi?id=1839) leading to some nasty intermittent build failures. Neal may remember how we addressed this for a Process class in a past life. Basically it's OK to collect all the child exit codes if you record the results and return them when requested. This would mean that waitpid and the like would have to check a cached list of PIDs and exit statuses before actually waiting. Don't know if that's possible/any help in this case or not... Traceback (most recent call last): File "/usr/lib/scons-0.97.0d20070918/SCons/Taskmaster.py", line 194, in execute self.targets[0].build() File "/usr/lib/scons-0.97.0d20070918/SCons/Node/__init__.py", line 365, in build stat = apply(executor, (self,), kw) File "/usr/lib/scons-0.97.0d20070918/SCons/Executor.py", line 139, in __call__ return self.do_execute(target, kw) File "/usr/lib/scons-0.97.0d20070918/SCons/Executor.py", line 129, in do_execute status = apply(act, (self.targets, self.sources, env), kw) File "/usr/lib/scons-0.97.0d20070918/SCons/Action.py", line 332, in __call__ stat = self.execute(target, source, env) File "/usr/lib/scons-0.97.0d20070918/SCons/Action.py", line 479, in execute result = spawn(shell, escape, cmd_line[0], cmd_line, ENV) File "/usr/lib/scons-0.97.0d20070918/SCons/Platform/posix.py", line 104, in spawnvpe_spawn return exec_spawnvpe([sh, '-c', string.join(args)], env) File "/usr/lib/scons-0.97.0d20070918/SCons/Platform/posix.py", line 68, in exec_spawnvpe stat = os.spawnvpe(os.P_WAIT, l[0], l, env) File "/sw/pkgs/python-2.3.5_00/lib/python2.3/os.py", line 553, in spawnvpe return _spawnvef(mode, file, args, env, execvpe) File "/sw/pkgs/python-2.3.5_00/lib/python2.3/os.py", line 504, in _spawnvef wpid, sts = waitpid(pid, 0) OSError: [Errno 10] No child processes scons: building terminated because of errors. -- nosy: +tom_culliton _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1731717> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1731717] race condition in subprocess module
Tom Culliton added the comment: Looking at the subprocess.py code it occurred to me that it never checks if the value of self.pid returned by os.fork is -1, this would mean that later it runs waitpid with -1 as the first argument, putting it into promiscuous (wait for any process in the group) mode. This seems like a bad idea... _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1731717> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1731717] race condition in subprocess module
Tom Culliton added the comment: Good question. The documentation I was reading was mute on the subject so I made a "reasonable" guess. Does it throw an exception instead? Guido van Rossum wrote: > Guido van Rossum added the comment: > > >> Looking at the subprocess.py code it occurred to me that it never >> checks if the value of self.pid returned by os.fork is -1 >> > > What makes you think os.fork(0 can return -1? It's not C you know... > > _ > Tracker <[EMAIL PROTECTED]> > <http://bugs.python.org/issue1731717> > _ > _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1731717> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1731717] race condition in subprocess module
Tom Culliton <[EMAIL PROTECTED]> added the comment: I'm not sure what the POSIX standards say on this (and MS-Windows may go it's own contrary way), but for most real systems the PID is a running count (generally 32 bit or larger today) which would have to cycle all the way around to repeat. It's actually done this way on purpose to provide the longest possible time between IDs getting reused. I suspect that having it cycle and reuse an ID isn't an issue in practice, and keeping a list of results leaves you with the problem of figuring out which PID 55367 they're talking about... Not to mention that if you're leaving child process results unchecked for long enough for the PID counter to cycle, there are other problems with the application. ;-) Gregory P. Smith wrote: > Gregory P. Smith <[EMAIL PROTECTED]> added the comment: > > """Basically it's OK to collect > all the child exit codes if you record the results and return them when > requested. This would mean that waitpid and the like would have to check > a cached list of PIDs and exit statuses before actually waiting.""" > > note that this would still have problems. PIDs are recycled by the OS > so as soon as you've waited on one and the process dies, the OS is free > to launch a new process using it. If the new process happens to be > another one of ours launched by subprocess that would be a problem. > Make the cached map of pids -> exit codes be a map of pids -> [list of > exit codes] instead? > > _ > Tracker <[EMAIL PROTECTED]> > <http://bugs.python.org/issue1731717> > _ > _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1731717> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1731717] race condition in subprocess module
Tom Culliton <[EMAIL PROTECTED]> added the comment: AIX, HP-UX, Solaris, 64 bit Linux, ... Even in the Linux x86 header files there's a mix of int and short. The last time I had to do the math on how long it would take the PID to cycle was probably on an AIX box and it was a very long time. Jean-Paul Calderone wrote: > Jean-Paul Calderone <[EMAIL PROTECTED]> added the comment: > > Which system uses a 32 bit PID? Not Linux, or FreeBSD, or OS X - they > all use 16 bit PIDs. > > -- > nosy: +exarkun > > _ > Tracker <[EMAIL PROTECTED]> > <http://bugs.python.org/issue1731717> > _ > _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1731717> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
