[ python-Bugs-1760423 ] No docs for list comparison
Bugs item #1760423, was opened at 2007-07-25 15:39 Message generated for change (Comment added) made by m_n_summerfield You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1760423&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Kent Johnson (kjohnson) Assigned to: Nobody/Anonymous (nobody) Summary: No docs for list comparison Initial Comment: AFAICT there is no documentation for the behaviour of <, > or == for sequences. == is perhaps obvious but it would be worth noting that < and > for lists and strings does a lexicographical comparison on the list elements or string characters. -- Comment By: Mark Summerfield (m_n_summerfield) Date: 2007-08-03 11:19 Message: Logged In: YES user_id=1856376 Originator: NO I've added a note plus a cross-reference to the comparisons coverage in the language reference. This won't appear until Python 2.6 though. -- Comment By: Kent Johnson (kjohnson) Date: 2007-07-27 12:52 Message: Logged In: YES user_id=49695 Originator: YES Yes, that is what I was looking for. It seems a strange place for the information. Could this be cross-referenced from the Library Reference? I never even thought to look for it in the Language Ref. Other list operations are documented in http://docs.python.org/lib/typesseq.html and this page discusses comparison of library types in general: http://docs.python.org/lib/comparisons.html I think it would be worth either repeating or linking to the information from the language ref on either of these pages. Thanks, Kent -- Comment By: Carsten Haese (chaese) Date: 2007-07-27 12:29 Message: Logged In: YES user_id=99003 Originator: NO http://docs.python.org/ref/comparisons.html works for me. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1760423&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1760423 ] No docs for list comparison
Bugs item #1760423, was opened at 2007-07-25 15:39 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1760423&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: Python 2.5 >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Kent Johnson (kjohnson) Assigned to: Nobody/Anonymous (nobody) Summary: No docs for list comparison Initial Comment: AFAICT there is no documentation for the behaviour of <, > or == for sequences. == is perhaps obvious but it would be worth noting that < and > for lists and strings does a lexicographical comparison on the list elements or string characters. -- >Comment By: Georg Brandl (gbrandl) Date: 2007-08-03 11:36 Message: Logged In: YES user_id=849994 Originator: NO Fixed in rev. 56698. -- Comment By: Mark Summerfield (m_n_summerfield) Date: 2007-08-03 11:19 Message: Logged In: YES user_id=1856376 Originator: NO I've added a note plus a cross-reference to the comparisons coverage in the language reference. This won't appear until Python 2.6 though. -- Comment By: Kent Johnson (kjohnson) Date: 2007-07-27 12:52 Message: Logged In: YES user_id=49695 Originator: YES Yes, that is what I was looking for. It seems a strange place for the information. Could this be cross-referenced from the Library Reference? I never even thought to look for it in the Language Ref. Other list operations are documented in http://docs.python.org/lib/typesseq.html and this page discusses comparison of library types in general: http://docs.python.org/lib/comparisons.html I think it would be worth either repeating or linking to the information from the language ref on either of these pages. Thanks, Kent -- Comment By: Carsten Haese (chaese) Date: 2007-07-27 12:29 Message: Logged In: YES user_id=99003 Originator: NO http://docs.python.org/ref/comparisons.html works for me. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1760423&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1767242 ] os.chmod failure
Bugs item #1767242, was opened at 2007-08-03 14:02
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767242&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Richard Heck (rgheck)
Assigned to: Nobody/Anonymous (nobody)
Summary: os.chmod failure
Initial Comment:
When running on Linux and accessing a FAT partition, os.chmod fails with
otherwise sensible partitions:
>>> os.chmod("/media/IHP-100/Test.lyx", 400)
>>> os.chmod("/media/IHP-100/Test.lyx", 600)
Traceback (most recent call last):
File "", line 1, in ?
OSError: [Errno 1] Operation not permitted: '/media/IHP-100/Test.lyx'
The only thing that seems to be allowed is `4'.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767242&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1731717 ] race condition in subprocess module
Bugs item #1731717, was opened at 2007-06-06 08:19
Message generated for change (Comment added) made by abo
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1731717&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: dsagal (dsagal)
Assigned to: Peter Åstrand (astrand)
Summary: race condition in subprocess module
Initial Comment:
Python's subprocess module has a race condition: Popen() constructor has a call
to global "_cleanup()" function on whenever a Popen object gets created, and
that call causes a check for all pending Popen objects whether their subprocess
has exited - i.e. the poll() method is called for every active Popen object.
Now, if I create Popen object "foo" in one thread, and then a.wait(), and
meanwhile I create another Popen object "bar" in another thread, then a.poll()
might get called by _clean() right at the time when my first thread is running
a.wait(). But those are not synchronized - each calls os.waitpid() if
returncode is None, but the section which checks returncode and calls
os.waitpid() is not protected as a critical section should be.
The following code illustrates the problem, and is known to break reasonably
consistenly with Python2.4. Changes to subprocess in Python2.5 seems to address
a somewhat related problem, but not this one.
import sys, os, threading, subprocess, time
class X(threading.Thread):
def __init__(self, *args, **kwargs):
super(X, self).__init__(*args, **kwargs)
self.start()
def tt():
s = subprocess.Popen(("/bin/ls", "-a", "/tmp"), stdout=subprocess.PIPE,
universal_newlines=True)
# time.sleep(1)
s.communicate()[0]
for i in xrange(1000):
X(target = tt)
This typically gives a few dozen errors like these:
Exception in thread Thread-795:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "z.py", line 21, in tt
s.communicate()[0]
File "/usr/lib/python2.4/subprocess.py", line 1083, in communicate
self.wait()
File "/usr/lib/python2.4/subprocess.py", line 1007, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes
Note that uncommenting time.sleep(1) fixes the problem. So does wrapping
subprocess.poll() and wait() with a lock. So does removing the call to global
_cleanup() in Popen constructor.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-04 05:14
Message:
Logged In: YES
user_id=10273
Originator: NO
I just looked at popen2 and it has the same bug. I don't think __del__()
related changes have anything to do with this. Popen.poll() and _cleanup()
are non-threadsafe. When __init__() is called to create subprocesses
simultaniously in two different threads, they both call _cleanup() and
trample all over each other.
Removing _cleanup() will not leave zombies for popened processes that are
correctly handled to completion. Using methods like communicate() and
wait() will handle the process to completion and reap the child.
Unfortunately, I think a fairly common use-case is people using popen to
fire-and-forget subprocesses without using communicate() or wait(). These
will not get reaped, and will end up as zombies.
I cannot think of an easy way to reap abandoned popen instances that is
thread safe without using locks. At times like this I wish that the GIL was
exposed as a recursive lock... we could have __cleanup__() acquire/release
the GIL and make it atomic... no more race condition :-)
--
Comment By: Peter Åstrand (astrand)
Date: 2007-08-02 19:15
Message:
Logged In: YES
user_id=344921
Originator: NO
Suddenly starting to leave Zombies is NOT an option for me. To prevent
zombies, all applications using the subprocess module would need to be
changed. This is a major breakage of backwards compatibility, IMHO.
subprocess (as well as the popen2 module) has prevented zombies
automatically from the beginning and I believe they should continue to do
so (at least by default).
A little bit of history: When I wrote the subprocess module, I stole the
idea of calling _cleanup whenever a new instance is created from the popen2
module, since I thought it was nice, lightweight and avoided having a
__del__ method (which I have some bad experience with, GC-related). This
worked great for a long time. At 2006-04-10, however, martin.v.loewis
committed patch 1467770 (revision r45234), to solve bug 1460493. It
introduces a __del__ method and some more complexity. I must admit that I
didn't review thi
[ python-Bugs-1731717 ] race condition in subprocess module
Bugs item #1731717, was opened at 2007-06-06 08:19
Message generated for change (Comment added) made by abo
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1731717&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: dsagal (dsagal)
Assigned to: Peter Åstrand (astrand)
Summary: race condition in subprocess module
Initial Comment:
Python's subprocess module has a race condition: Popen() constructor has a call
to global "_cleanup()" function on whenever a Popen object gets created, and
that call causes a check for all pending Popen objects whether their subprocess
has exited - i.e. the poll() method is called for every active Popen object.
Now, if I create Popen object "foo" in one thread, and then a.wait(), and
meanwhile I create another Popen object "bar" in another thread, then a.poll()
might get called by _clean() right at the time when my first thread is running
a.wait(). But those are not synchronized - each calls os.waitpid() if
returncode is None, but the section which checks returncode and calls
os.waitpid() is not protected as a critical section should be.
The following code illustrates the problem, and is known to break reasonably
consistenly with Python2.4. Changes to subprocess in Python2.5 seems to address
a somewhat related problem, but not this one.
import sys, os, threading, subprocess, time
class X(threading.Thread):
def __init__(self, *args, **kwargs):
super(X, self).__init__(*args, **kwargs)
self.start()
def tt():
s = subprocess.Popen(("/bin/ls", "-a", "/tmp"), stdout=subprocess.PIPE,
universal_newlines=True)
# time.sleep(1)
s.communicate()[0]
for i in xrange(1000):
X(target = tt)
This typically gives a few dozen errors like these:
Exception in thread Thread-795:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "z.py", line 21, in tt
s.communicate()[0]
File "/usr/lib/python2.4/subprocess.py", line 1083, in communicate
self.wait()
File "/usr/lib/python2.4/subprocess.py", line 1007, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes
Note that uncommenting time.sleep(1) fixes the problem. So does wrapping
subprocess.poll() and wait() with a lock. So does removing the call to global
_cleanup() in Popen constructor.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-04 05:26
Message:
Logged In: YES
user_id=10273
Originator: NO
Actually, after thinking about it, there may be a way to make _cleanup()
threadsafe without using explicit locks by leveraging off the GIL. If
_active.pop() and _active.append() are atomic because of the GIL, then
_cleanup() can be made threadsafe. To be truely threadsafe it also needs to
make sure that _active contains only abandoned popen instances so that
_cleanup() doesn't have it's inst.poll() calls collide with other threads
that are still using those popen instances. This can be done by havin the
popen instances added to _active only by Popen.__del__(), and only removed
by _cleanup() when they are done.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-04 05:14
Message:
Logged In: YES
user_id=10273
Originator: NO
I just looked at popen2 and it has the same bug. I don't think __del__()
related changes have anything to do with this. Popen.poll() and _cleanup()
are non-threadsafe. When __init__() is called to create subprocesses
simultaniously in two different threads, they both call _cleanup() and
trample all over each other.
Removing _cleanup() will not leave zombies for popened processes that are
correctly handled to completion. Using methods like communicate() and
wait() will handle the process to completion and reap the child.
Unfortunately, I think a fairly common use-case is people using popen to
fire-and-forget subprocesses without using communicate() or wait(). These
will not get reaped, and will end up as zombies.
I cannot think of an easy way to reap abandoned popen instances that is
thread safe without using locks. At times like this I wish that the GIL was
exposed as a recursive lock... we could have __cleanup__() acquire/release
the GIL and make it atomic... no more race condition :-)
--
Comment By: Peter Åstrand (astrand)
Date: 2007-08-02 19:15
Message:
Logged In: YES
user_id=344921
Originator: NO
Suddenly starting to leave Zombies is NOT an option for me. To prevent
zombies, all applicatio
[ python-Bugs-1754642 ] subprocess.Popen.wait fails sporadically with threads
Bugs item #1754642, was opened at 2007-07-16 19:26 Message generated for change (Comment added) made by abo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1754642&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Geoffrey Bache (gjb1002) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess.Popen.wait fails sporadically with threads Initial Comment: When several threads are using the subprocess module I occasionally get stack traces that look like out, err = process.communicate() File "/usr/lib/python2.4/subprocess.py", line 1083, in communicate self.wait() File "/usr/lib/python2.4/subprocess.py", line 1007, in wait pid, sts = os.waitpid(self.pid, 0) OSError: [Errno 10] No child processes See the fixed bug at https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1183780&group_id=5470 The problem I am observing seems to match the one described there in the "related modules" part. After discovering that bug, I downloaded the test program attached to it and discovered that I got the errors described from time to time. I am using Python 2.4.3 on Red Hat EL4 and Python 2.4.4 on Red Hat EL3, and both have this error. I reattach my tweaked version of that test program. The only basic difference is that testing subprocess is the default option and the default number of threads increased to 50 as this seemed to mean it occurred more often. I have reproduced the original popen2 problem once on Python 2.4.4 but it's much more infrequent, and not a problem in practice for me. See the comment at the top of that script for details of expected behaviour etc. -- Comment By: Donovan Baarda (abo) Date: 2007-08-04 05:37 Message: Logged In: YES user_id=10273 Originator: NO Bugs 1754642 and 1753891 both look like duplicates of bug 1731717 to me. I suggest marking them both as dups of 1731717 because that one has info on the race-condition that causes this and discussions on how to fix it. -- Comment By: Geoffrey Bache (gjb1002) Date: 2007-07-16 19:32 Message: Logged In: YES user_id=769182 Originator: YES Hmm, seems someone else reported this during the weekend. Looks rather similar to bug 1753891... I let someone else mark as duplicate if they want. Can't harm to have two different test programs for an indeterministic problem :) -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1754642&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1753891 ] subprocess raising "No Child Process" OSError
Bugs item #1753891, was opened at 2007-07-14 13:06 Message generated for change (Comment added) made by abo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753891&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: slowfood (slowfood) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess raising "No Child Process" OSError Initial Comment: The program below demostrates a "No Child Process" OSError on a multi-cpu systems. This is extracted from a large system where we are trying to manage many sub-processes, some of which end up having little/no real work to do so they return very fast, here emulated by having the sub-process be an invocation of: Executable="/bin/sleep 0" Seems like some race condition, since if you make the child process take some time (sleep 0.1) the frequency of errors decreeses. Error only shows up when there are more threads than have real CPU's by at least a factor of two, on dual core machines up NumThreads to 18 to get the failures. Same error on both Python 2.4.3 with: Linux 2.6.18-gentoo-r3 Gentoo and Python 2.4.4 with: Linux 2.6.20-gentoo-r8 Any help appreciated - = = = = Start code example === % python subprocess_noChildErr.py Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "testCaseA.py", line 14, in run subprocess.call(Executable.split()) File "/usr/lib64/python2.4/subprocess.py", line 413, in call return Popen(*args, **kwargs).wait() File "/usr/lib64/python2.4/subprocess.py", line 1007, in wait pid, sts = os.waitpid(self.pid, 0) OSError: [Errno 10] No child processes Test finished % cat subprocess_noChildErr.py import subprocess, threading # Params Executable="/bin/sleep 0" NumThreads = 18 NumIterations = 10 class TestClass(threading.Thread): def __init__(self, threadNum): self.threadNum = threadNum threading.Thread.__init__(self) def run(self): for i in range(NumIterations): subprocess.call(Executable.split()) def test(): allThreads = [] for i in range(NumThreads): allThreads.append(TestClass(i)) for i in range(NumThreads): allThreads[i].start() for i in range(NumThreads): allThreads[i].join() print "Test finished" if __name__ == '__main__': test() % python -V Python 2.4.4 % uname -a Linux 2.6.20-gentoo-r8 #2 SMP PREEMPT Sun Jul 1 13:22:56 PDT 2007 x86_64 Dual-Core AMD Opteron(tm) Processor 2212 AuthenticAMD GNU/Linux % % date Fri Jul 13 19:26:44 PDT 2007 % = = = = End code example === -- Comment By: Donovan Baarda (abo) Date: 2007-08-04 05:37 Message: Logged In: YES user_id=10273 Originator: NO Bugs 1754642 and 1753891 both look like duplicates of bug 1731717 to me. I suggest marking them both as dups of 1731717 because that one has info on the race-condition that causes this and discussions on how to fix it. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753891&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1753891 ] subprocess raising "No Child Process" OSError
Bugs item #1753891, was opened at 2007-07-13 20:06 Message generated for change (Comment added) made by slowfood You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753891&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Threads Group: Python 2.4 Status: Open >Resolution: Duplicate Priority: 5 Private: No Submitted By: slowfood (slowfood) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess raising "No Child Process" OSError Initial Comment: The program below demostrates a "No Child Process" OSError on a multi-cpu systems. This is extracted from a large system where we are trying to manage many sub-processes, some of which end up having little/no real work to do so they return very fast, here emulated by having the sub-process be an invocation of: Executable="/bin/sleep 0" Seems like some race condition, since if you make the child process take some time (sleep 0.1) the frequency of errors decreeses. Error only shows up when there are more threads than have real CPU's by at least a factor of two, on dual core machines up NumThreads to 18 to get the failures. Same error on both Python 2.4.3 with: Linux 2.6.18-gentoo-r3 Gentoo and Python 2.4.4 with: Linux 2.6.20-gentoo-r8 Any help appreciated - = = = = Start code example === % python subprocess_noChildErr.py Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "testCaseA.py", line 14, in run subprocess.call(Executable.split()) File "/usr/lib64/python2.4/subprocess.py", line 413, in call return Popen(*args, **kwargs).wait() File "/usr/lib64/python2.4/subprocess.py", line 1007, in wait pid, sts = os.waitpid(self.pid, 0) OSError: [Errno 10] No child processes Test finished % cat subprocess_noChildErr.py import subprocess, threading # Params Executable="/bin/sleep 0" NumThreads = 18 NumIterations = 10 class TestClass(threading.Thread): def __init__(self, threadNum): self.threadNum = threadNum threading.Thread.__init__(self) def run(self): for i in range(NumIterations): subprocess.call(Executable.split()) def test(): allThreads = [] for i in range(NumThreads): allThreads.append(TestClass(i)) for i in range(NumThreads): allThreads[i].start() for i in range(NumThreads): allThreads[i].join() print "Test finished" if __name__ == '__main__': test() % python -V Python 2.4.4 % uname -a Linux 2.6.20-gentoo-r8 #2 SMP PREEMPT Sun Jul 1 13:22:56 PDT 2007 x86_64 Dual-Core AMD Opteron(tm) Processor 2212 AuthenticAMD GNU/Linux % % date Fri Jul 13 19:26:44 PDT 2007 % = = = = End code example === -- >Comment By: slowfood (slowfood) Date: 2007-08-03 13:01 Message: Logged In: YES user_id=1844537 Originator: YES I agree with abo that this looks to be a dup of 1731717, will try to mark it as such. -- Comment By: Donovan Baarda (abo) Date: 2007-08-03 12:37 Message: Logged In: YES user_id=10273 Originator: NO Bugs 1754642 and 1753891 both look like duplicates of bug 1731717 to me. I suggest marking them both as dups of 1731717 because that one has info on the race-condition that causes this and discussions on how to fix it. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753891&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1753891 ] subprocess raising "No Child Process" OSError
Bugs item #1753891, was opened at 2007-07-13 20:06 Message generated for change (Comment added) made by slowfood You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753891&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Threads Group: Python 2.4 Status: Open Resolution: Duplicate Priority: 5 Private: No Submitted By: slowfood (slowfood) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess raising "No Child Process" OSError Initial Comment: The program below demostrates a "No Child Process" OSError on a multi-cpu systems. This is extracted from a large system where we are trying to manage many sub-processes, some of which end up having little/no real work to do so they return very fast, here emulated by having the sub-process be an invocation of: Executable="/bin/sleep 0" Seems like some race condition, since if you make the child process take some time (sleep 0.1) the frequency of errors decreeses. Error only shows up when there are more threads than have real CPU's by at least a factor of two, on dual core machines up NumThreads to 18 to get the failures. Same error on both Python 2.4.3 with: Linux 2.6.18-gentoo-r3 Gentoo and Python 2.4.4 with: Linux 2.6.20-gentoo-r8 Any help appreciated - = = = = Start code example === % python subprocess_noChildErr.py Exception in thread Thread-3: Traceback (most recent call last): File "/usr/lib64/python2.4/threading.py", line 442, in __bootstrap self.run() File "testCaseA.py", line 14, in run subprocess.call(Executable.split()) File "/usr/lib64/python2.4/subprocess.py", line 413, in call return Popen(*args, **kwargs).wait() File "/usr/lib64/python2.4/subprocess.py", line 1007, in wait pid, sts = os.waitpid(self.pid, 0) OSError: [Errno 10] No child processes Test finished % cat subprocess_noChildErr.py import subprocess, threading # Params Executable="/bin/sleep 0" NumThreads = 18 NumIterations = 10 class TestClass(threading.Thread): def __init__(self, threadNum): self.threadNum = threadNum threading.Thread.__init__(self) def run(self): for i in range(NumIterations): subprocess.call(Executable.split()) def test(): allThreads = [] for i in range(NumThreads): allThreads.append(TestClass(i)) for i in range(NumThreads): allThreads[i].start() for i in range(NumThreads): allThreads[i].join() print "Test finished" if __name__ == '__main__': test() % python -V Python 2.4.4 % uname -a Linux 2.6.20-gentoo-r8 #2 SMP PREEMPT Sun Jul 1 13:22:56 PDT 2007 x86_64 Dual-Core AMD Opteron(tm) Processor 2212 AuthenticAMD GNU/Linux % % date Fri Jul 13 19:26:44 PDT 2007 % = = = = End code example === -- >Comment By: slowfood (slowfood) Date: 2007-08-03 13:07 Message: Logged In: YES user_id=1844537 Originator: YES Managed to mark it as a duplicate, but not able to tie it to 1731717. Sorry - ;;slowfood -- Comment By: slowfood (slowfood) Date: 2007-08-03 13:01 Message: Logged In: YES user_id=1844537 Originator: YES I agree with abo that this looks to be a dup of 1731717, will try to mark it as such. -- Comment By: Donovan Baarda (abo) Date: 2007-08-03 12:37 Message: Logged In: YES user_id=10273 Originator: NO Bugs 1754642 and 1753891 both look like duplicates of bug 1731717 to me. I suggest marking them both as dups of 1731717 because that one has info on the race-condition that causes this and discussions on how to fix it. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753891&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1731717 ] race condition in subprocess module
Bugs item #1731717, was opened at 2007-06-05 18:19
Message generated for change (Comment added) made by gvanrossum
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1731717&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: dsagal (dsagal)
Assigned to: Peter Åstrand (astrand)
Summary: race condition in subprocess module
Initial Comment:
Python's subprocess module has a race condition: Popen() constructor has a call
to global "_cleanup()" function on whenever a Popen object gets created, and
that call causes a check for all pending Popen objects whether their subprocess
has exited - i.e. the poll() method is called for every active Popen object.
Now, if I create Popen object "foo" in one thread, and then a.wait(), and
meanwhile I create another Popen object "bar" in another thread, then a.poll()
might get called by _clean() right at the time when my first thread is running
a.wait(). But those are not synchronized - each calls os.waitpid() if
returncode is None, but the section which checks returncode and calls
os.waitpid() is not protected as a critical section should be.
The following code illustrates the problem, and is known to break reasonably
consistenly with Python2.4. Changes to subprocess in Python2.5 seems to address
a somewhat related problem, but not this one.
import sys, os, threading, subprocess, time
class X(threading.Thread):
def __init__(self, *args, **kwargs):
super(X, self).__init__(*args, **kwargs)
self.start()
def tt():
s = subprocess.Popen(("/bin/ls", "-a", "/tmp"), stdout=subprocess.PIPE,
universal_newlines=True)
# time.sleep(1)
s.communicate()[0]
for i in xrange(1000):
X(target = tt)
This typically gives a few dozen errors like these:
Exception in thread Thread-795:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "z.py", line 21, in tt
s.communicate()[0]
File "/usr/lib/python2.4/subprocess.py", line 1083, in communicate
self.wait()
File "/usr/lib/python2.4/subprocess.py", line 1007, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes
Note that uncommenting time.sleep(1) fixes the problem. So does wrapping
subprocess.poll() and wait() with a lock. So does removing the call to global
_cleanup() in Popen constructor.
--
>Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 16:11
Message:
Logged In: YES
user_id=6380
Originator: NO
If you want this fixed in 2.5.x, a new release is just around the corner,
and time is very tight. Otherwise, 2.6a1 is half a year off.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-03 15:26
Message:
Logged In: YES
user_id=10273
Originator: NO
Actually, after thinking about it, there may be a way to make _cleanup()
threadsafe without using explicit locks by leveraging off the GIL. If
_active.pop() and _active.append() are atomic because of the GIL, then
_cleanup() can be made threadsafe. To be truely threadsafe it also needs to
make sure that _active contains only abandoned popen instances so that
_cleanup() doesn't have it's inst.poll() calls collide with other threads
that are still using those popen instances. This can be done by havin the
popen instances added to _active only by Popen.__del__(), and only removed
by _cleanup() when they are done.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-03 15:14
Message:
Logged In: YES
user_id=10273
Originator: NO
I just looked at popen2 and it has the same bug. I don't think __del__()
related changes have anything to do with this. Popen.poll() and _cleanup()
are non-threadsafe. When __init__() is called to create subprocesses
simultaniously in two different threads, they both call _cleanup() and
trample all over each other.
Removing _cleanup() will not leave zombies for popened processes that are
correctly handled to completion. Using methods like communicate() and
wait() will handle the process to completion and reap the child.
Unfortunately, I think a fairly common use-case is people using popen to
fire-and-forget subprocesses without using communicate() or wait(). These
will not get reaped, and will end up as zombies.
I cannot think of an easy way to reap abandoned popen instances that is
thread safe without using locks. At times like this I wish that the GIL was
exposed as a recursive lock... we could have __cleanup__() acquire/release
the
[ python-Bugs-1767242 ] os.chmod failure
Bugs item #1767242, was opened at 2007-08-03 18:02
Message generated for change (Comment added) made by gbrandl
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767242&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Pending
Resolution: None
Priority: 5
Private: No
Submitted By: Richard Heck (rgheck)
Assigned to: Nobody/Anonymous (nobody)
Summary: os.chmod failure
Initial Comment:
When running on Linux and accessing a FAT partition, os.chmod fails with
otherwise sensible partitions:
>>> os.chmod("/media/IHP-100/Test.lyx", 400)
>>> os.chmod("/media/IHP-100/Test.lyx", 600)
Traceback (most recent call last):
File "", line 1, in ?
OSError: [Errno 1] Operation not permitted: '/media/IHP-100/Test.lyx'
The only thing that seems to be allowed is `4'.
--
>Comment By: Georg Brandl (gbrandl)
Date: 2007-08-03 20:25
Message:
Logged In: YES
user_id=849994
Originator: NO
Why do you think this is Python's fault? os.chmod() is only a very thin
wrapper around the OS' chmod().
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767242&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1762972 ] 'exec' does not accept what 'open' returns
Bugs item #1762972, was opened at 2007-07-28 23:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Interpreter Core Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Brett Cannon (bcannon) >Assigned to: Guido van Rossum (gvanrossum) Summary: 'exec' does not accept what 'open' returns Initial Comment: Since the move over to io.py 'exec' no longer accepts what 'open' returns. Looks like there is a type check in 'exec' for strings, files, or code object, but 'open' returns a TextIOWrapper that fails that typecheck. -- >Comment By: Guido van Rossum (gvanrossum) Date: 2007-08-03 16:26 Message: Logged In: YES user_id=6380 Originator: NO Arguably we don't need this feature any more. The only reasonable way to implement it would be to slurp the entire contents of the file into a string and then exec that. The caller might as well do this. Alternatively one could use execfile() (which takes a filename instead of a stream). The only advantage of using exec() might be that it would deal with encoding declarations transparantly; but we should probably have a way to handle those somewhere in the library anyway, as it's also necessary for other tools (e.g. modulefinder.py, pyclbr.py and linecache.py). Alternatively I'm not so sure that we will need to support encoding declarations forever -- I'm hoping that at some point UTF-8 and BOM+UTF-16 will take over the world. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1753395 ] struni: assertion in Windows debug build
Bugs item #1753395, was opened at 2007-07-13 07:35 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753395&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Thomas Heller (theller) >Assigned to: Thomas Heller (theller) Summary: struni: assertion in Windows debug build Initial Comment: When running Lib/test/test_descr.py with a Windows debug build, I get an assertion in the MS runtime lib, in the isalpha(*p) call, file typeobject.c, line 1582. The value of '*p' at the time of the call is 4660. The assertion reported is: File: istype.c Line: 68 Expression; (unsigned)(c + 1) <= 256 -- >Comment By: Guido van Rossum (gvanrossum) Date: 2007-08-03 16:37 Message: Logged In: YES user_id=6380 Originator: NO I wonder if the test "i > 255" wasn't meant to be "*p > 255"? Please try that, and submit if it works. Looks like Walter Doerwald introduced this bug in r55892. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1753395&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1767242 ] os.chmod failure
Bugs item #1767242, was opened at 2007-08-03 14:02
Message generated for change (Comment added) made by rgheck
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767242&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
>Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Richard Heck (rgheck)
Assigned to: Nobody/Anonymous (nobody)
Summary: os.chmod failure
Initial Comment:
When running on Linux and accessing a FAT partition, os.chmod fails with
otherwise sensible partitions:
>>> os.chmod("/media/IHP-100/Test.lyx", 400)
>>> os.chmod("/media/IHP-100/Test.lyx", 600)
Traceback (most recent call last):
File "", line 1, in ?
OSError: [Errno 1] Operation not permitted: '/media/IHP-100/Test.lyx'
The only thing that seems to be allowed is `4'.
--
>Comment By: Richard Heck (rgheck)
Date: 2007-08-03 16:39
Message:
Logged In: YES
user_id=1072146
Originator: YES
Well, the shell's chmod doesn't return an error.
[EMAIL PROTECTED] scripts]$ chmod 600 /media/IHP-100/Test.lyx
[EMAIL PROTECTED] scripts]$ echo $?
0
[EMAIL PROTECTED] scripts]$ chmod 400 /media/IHP-100/Test.lyx
[EMAIL PROTECTED] scripts]$ echo $?
0
[EMAIL PROTECTED] scripts]$ chmod 700 /media/IHP-100/Test.lyx
[EMAIL PROTECTED] scripts]$ echo $?
0
[EMAIL PROTECTED] scripts]$ python
Python 2.4.4 (#1, Oct 23 2006, 13:58:00)
[GCC 4.1.1 20061011 (Red Hat 4.1.1-30)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.chmod("/media/IHP-100/Test.lyx", 400)
>>> os.chmod("/media/IHP-100/Test.lyx", 600)
Traceback (most recent call last):
File "", line 1, in ?
OSError: [Errno 1] Operation not permitted: '/media/IHP-100/Test.lyx'
I don't know what would happen in C, say.
--
Comment By: Georg Brandl (gbrandl)
Date: 2007-08-03 16:25
Message:
Logged In: YES
user_id=849994
Originator: NO
Why do you think this is Python's fault? os.chmod() is only a very thin
wrapper around the OS' chmod().
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767242&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1762972 ] 'exec' does not accept what 'open' returns
Bugs item #1762972, was opened at 2007-07-28 20:24 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Interpreter Core Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Brett Cannon (bcannon) Assigned to: Guido van Rossum (gvanrossum) Summary: 'exec' does not accept what 'open' returns Initial Comment: Since the move over to io.py 'exec' no longer accepts what 'open' returns. Looks like there is a type check in 'exec' for strings, files, or code object, but 'open' returns a TextIOWrapper that fails that typecheck. -- >Comment By: Brett Cannon (bcannon) Date: 2007-08-03 14:09 Message: Logged In: YES user_id=357491 Originator: YES I am fine with ditching the feature and forcing people to deal with the encoding issues externally to 'exec', although as you pointed out dealing with any -*- encoding marker might be a pain without some library help (although doesn't the parser pick up on this and handle those nasty details?). As for execfile, PEP 3100 has it listed for removal in favour of moving people over to using 'exec'. The main thing is trying to come up with an easy solution for replacing 'reload' with a reasonable one-liner when debugging at the interpreter. And yes, I hope encoding declarations can go away and the world either lives with just UTF-8 or BOM+UTF-16 (I really wish we could just require this for all Python source files and start towards this, but I know some people would flip out over that). -- Comment By: Guido van Rossum (gvanrossum) Date: 2007-08-03 13:26 Message: Logged In: YES user_id=6380 Originator: NO Arguably we don't need this feature any more. The only reasonable way to implement it would be to slurp the entire contents of the file into a string and then exec that. The caller might as well do this. Alternatively one could use execfile() (which takes a filename instead of a stream). The only advantage of using exec() might be that it would deal with encoding declarations transparantly; but we should probably have a way to handle those somewhere in the library anyway, as it's also necessary for other tools (e.g. modulefinder.py, pyclbr.py and linecache.py). Alternatively I'm not so sure that we will need to support encoding declarations forever -- I'm hoping that at some point UTF-8 and BOM+UTF-16 will take over the world. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1762972 ] 'exec' does not accept what 'open' returns
Bugs item #1762972, was opened at 2007-07-28 23:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Interpreter Core Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Brett Cannon (bcannon) Assigned to: Guido van Rossum (gvanrossum) Summary: 'exec' does not accept what 'open' returns Initial Comment: Since the move over to io.py 'exec' no longer accepts what 'open' returns. Looks like there is a type check in 'exec' for strings, files, or code object, but 'open' returns a TextIOWrapper that fails that typecheck. -- >Comment By: Guido van Rossum (gvanrossum) Date: 2007-08-03 17:14 Message: Logged In: YES user_id=6380 Originator: NO OK, let's ditch it (i.e. status quo) and also ditch execfile. The one-liner you're looking for is *almost* exec(open(M.__file__).read(), M.__dict__) (where M stands for any already imported module), except for the nastiness that if m.__file__ ends with .pyc or .pyo you'd have to strip the last character. (And to do this right you'd need help from the imp module because those extensions may vary by platform -- e.g. I know of a company that patches their Pythons to use .pyc24.) -- Comment By: Brett Cannon (bcannon) Date: 2007-08-03 17:09 Message: Logged In: YES user_id=357491 Originator: YES I am fine with ditching the feature and forcing people to deal with the encoding issues externally to 'exec', although as you pointed out dealing with any -*- encoding marker might be a pain without some library help (although doesn't the parser pick up on this and handle those nasty details?). As for execfile, PEP 3100 has it listed for removal in favour of moving people over to using 'exec'. The main thing is trying to come up with an easy solution for replacing 'reload' with a reasonable one-liner when debugging at the interpreter. And yes, I hope encoding declarations can go away and the world either lives with just UTF-8 or BOM+UTF-16 (I really wish we could just require this for all Python source files and start towards this, but I know some people would flip out over that). -- Comment By: Guido van Rossum (gvanrossum) Date: 2007-08-03 16:26 Message: Logged In: YES user_id=6380 Originator: NO Arguably we don't need this feature any more. The only reasonable way to implement it would be to slurp the entire contents of the file into a string and then exec that. The caller might as well do this. Alternatively one could use execfile() (which takes a filename instead of a stream). The only advantage of using exec() might be that it would deal with encoding declarations transparantly; but we should probably have a way to handle those somewhere in the library anyway, as it's also necessary for other tools (e.g. modulefinder.py, pyclbr.py and linecache.py). Alternatively I'm not so sure that we will need to support encoding declarations forever -- I'm hoping that at some point UTF-8 and BOM+UTF-16 will take over the world. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1762972 ] 'exec' does not accept what 'open' returns
Bugs item #1762972, was opened at 2007-07-28 20:24
Message generated for change (Comment added) made by bcannon
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 3000
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brett Cannon (bcannon)
Assigned to: Guido van Rossum (gvanrossum)
Summary: 'exec' does not accept what 'open' returns
Initial Comment:
Since the move over to io.py 'exec' no longer accepts what 'open' returns.
Looks like there is a type check in 'exec' for strings, files, or code object,
but 'open' returns a TextIOWrapper that fails that typecheck.
--
>Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 14:42
Message:
Logged In: YES
user_id=357491
Originator: YES
Hrm. Well then we can either add something like __source_file__ to
modules, put 'reload' into the imp module (or on module objects as a
method), or have a function in imp (or as a method on modules) that returns
the source path (if it exists).
But having to do::
M.__file__.rsplit('.', 1) + filter((lambda x : x[2] == imp.PY_SOURCE),
imp.get_suffixes())[0]
seems like a lot to memorize (let alone type in), especially since there
is even no error checking that the path is even to a source or bytecode
file to start with or that you end up with.
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 14:14
Message:
Logged In: YES
user_id=6380
Originator: NO
OK, let's ditch it (i.e. status quo) and also ditch execfile.
The one-liner you're looking for is *almost*
exec(open(M.__file__).read(), M.__dict__)
(where M stands for any already imported module), except for the nastiness
that if m.__file__ ends with .pyc or .pyo you'd have to strip the last
character. (And to do this right you'd need help from the imp module
because those extensions may vary by platform -- e.g. I know of a company
that patches their Pythons to use .pyc24.)
--
Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 14:09
Message:
Logged In: YES
user_id=357491
Originator: YES
I am fine with ditching the feature and forcing people to deal with the
encoding issues externally to 'exec', although as you pointed out dealing
with any -*- encoding marker might be a pain without some library help
(although doesn't the parser pick up on this and handle those nasty
details?).
As for execfile, PEP 3100 has it listed for removal in favour of moving
people over to using 'exec'.
The main thing is trying to come up with an easy solution for replacing
'reload' with a reasonable one-liner when debugging at the interpreter.
And yes, I hope encoding declarations can go away and the world either
lives with just UTF-8 or BOM+UTF-16 (I really wish we could just require
this for all Python source files and start towards this, but I know some
people would flip out over that).
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 13:26
Message:
Logged In: YES
user_id=6380
Originator: NO
Arguably we don't need this feature any more. The only reasonable way to
implement it would be to slurp the entire contents of the file into a
string and then exec that. The caller might as well do this.
Alternatively one could use execfile() (which takes a filename instead of a
stream). The only advantage of using exec() might be that it would
deal with encoding declarations transparantly; but we should probably have
a way to handle those somewhere in the library anyway, as it's also
necessary for other tools (e.g. modulefinder.py, pyclbr.py and
linecache.py). Alternatively I'm not so sure that we will need to support
encoding declarations forever -- I'm hoping that at some point UTF-8 and
BOM+UTF-16 will take over the world.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1731717 ] race condition in subprocess module
Bugs item #1731717, was opened at 2007-06-06 08:19
Message generated for change (Comment added) made by abo
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1731717&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: dsagal (dsagal)
Assigned to: Peter Åstrand (astrand)
Summary: race condition in subprocess module
Initial Comment:
Python's subprocess module has a race condition: Popen() constructor has a call
to global "_cleanup()" function on whenever a Popen object gets created, and
that call causes a check for all pending Popen objects whether their subprocess
has exited - i.e. the poll() method is called for every active Popen object.
Now, if I create Popen object "foo" in one thread, and then a.wait(), and
meanwhile I create another Popen object "bar" in another thread, then a.poll()
might get called by _clean() right at the time when my first thread is running
a.wait(). But those are not synchronized - each calls os.waitpid() if
returncode is None, but the section which checks returncode and calls
os.waitpid() is not protected as a critical section should be.
The following code illustrates the problem, and is known to break reasonably
consistenly with Python2.4. Changes to subprocess in Python2.5 seems to address
a somewhat related problem, but not this one.
import sys, os, threading, subprocess, time
class X(threading.Thread):
def __init__(self, *args, **kwargs):
super(X, self).__init__(*args, **kwargs)
self.start()
def tt():
s = subprocess.Popen(("/bin/ls", "-a", "/tmp"), stdout=subprocess.PIPE,
universal_newlines=True)
# time.sleep(1)
s.communicate()[0]
for i in xrange(1000):
X(target = tt)
This typically gives a few dozen errors like these:
Exception in thread Thread-795:
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
self.run()
File "/usr/lib/python2.4/threading.py", line 422, in run
self.__target(*self.__args, **self.__kwargs)
File "z.py", line 21, in tt
s.communicate()[0]
File "/usr/lib/python2.4/subprocess.py", line 1083, in communicate
self.wait()
File "/usr/lib/python2.4/subprocess.py", line 1007, in wait
pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes
Note that uncommenting time.sleep(1) fixes the problem. So does wrapping
subprocess.poll() and wait() with a lock. So does removing the call to global
_cleanup() in Popen constructor.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-04 07:29
Message:
Logged In: YES
user_id=10273
Originator: NO
I just looked at subprocess in svs trunk and it looks like it might
already be fixed in both subprocess.py and popen2.py. The part where
__del__() adds abandoned Popen instances to _active and _cleanup() removes
them is already there. _cleanup() has been made more thread resistant by
catching and ignoring exceptions that arise when two _cleanups() try to
remove the same instances. Popen.poll() has been made more robust by having
it set self.returncode to an optional _deadstate argument value when poll
gets an os.error from os.pidwait() on a child that gets cleaned up by
another thread. I think there are still a few minor race conditions around
Popen.poll(), but it will only affect what non-zero returncode you get...
it's not so much "thread safe" as "thread robust".
I think the _deadstate argument approach to try and make Popen.poll()
thread-robust is a bit hacky. I would rather see _cleanup() be properly
threadsafe by having it remove the inst from _active before processing it
and re-adding it back if it is still not finished. However, as it is now it
looks like it is fixed...
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-04 06:11
Message:
Logged In: YES
user_id=6380
Originator: NO
If you want this fixed in 2.5.x, a new release is just around the corner,
and time is very tight. Otherwise, 2.6a1 is half a year off.
--
Comment By: Donovan Baarda (abo)
Date: 2007-08-04 05:26
Message:
Logged In: YES
user_id=10273
Originator: NO
Actually, after thinking about it, there may be a way to make _cleanup()
threadsafe without using explicit locks by leveraging off the GIL. If
_active.pop() and _active.append() are atomic because of the GIL, then
_cleanup() can be made threadsafe. To be truely threadsafe it also needs to
make sure that _active contains only abandoned popen instances so that
_cleanup() doesn't have it's inst.poll() calls collide with other threads
that are still using those popen instances. Thi
[ python-Bugs-1762972 ] 'exec' does not accept what 'open' returns
Bugs item #1762972, was opened at 2007-07-28 23:24
Message generated for change (Comment added) made by gvanrossum
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 3000
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brett Cannon (bcannon)
Assigned to: Guido van Rossum (gvanrossum)
Summary: 'exec' does not accept what 'open' returns
Initial Comment:
Since the move over to io.py 'exec' no longer accepts what 'open' returns.
Looks like there is a type check in 'exec' for strings, files, or code object,
but 'open' returns a TextIOWrapper that fails that typecheck.
--
>Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 17:53
Message:
Logged In: YES
user_id=6380
Originator: NO
Or we can change the semantics of __file__ to always point to the source
file. I think that's acceptable behavior -- the current behavior seems
more of a pain than necessary.
I also think that adding imp.reload() makes sense; I was just saying
yesterday to a few fellow-Googler-Python-developers that I was already
missing reload().
--
Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 17:42
Message:
Logged In: YES
user_id=357491
Originator: YES
Hrm. Well then we can either add something like __source_file__ to
modules, put 'reload' into the imp module (or on module objects as a
method), or have a function in imp (or as a method on modules) that returns
the source path (if it exists).
But having to do::
M.__file__.rsplit('.', 1) + filter((lambda x : x[2] == imp.PY_SOURCE),
imp.get_suffixes())[0]
seems like a lot to memorize (let alone type in), especially since there
is even no error checking that the path is even to a source or bytecode
file to start with or that you end up with.
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 17:14
Message:
Logged In: YES
user_id=6380
Originator: NO
OK, let's ditch it (i.e. status quo) and also ditch execfile.
The one-liner you're looking for is *almost*
exec(open(M.__file__).read(), M.__dict__)
(where M stands for any already imported module), except for the nastiness
that if m.__file__ ends with .pyc or .pyo you'd have to strip the last
character. (And to do this right you'd need help from the imp module
because those extensions may vary by platform -- e.g. I know of a company
that patches their Pythons to use .pyc24.)
--
Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 17:09
Message:
Logged In: YES
user_id=357491
Originator: YES
I am fine with ditching the feature and forcing people to deal with the
encoding issues externally to 'exec', although as you pointed out dealing
with any -*- encoding marker might be a pain without some library help
(although doesn't the parser pick up on this and handle those nasty
details?).
As for execfile, PEP 3100 has it listed for removal in favour of moving
people over to using 'exec'.
The main thing is trying to come up with an easy solution for replacing
'reload' with a reasonable one-liner when debugging at the interpreter.
And yes, I hope encoding declarations can go away and the world either
lives with just UTF-8 or BOM+UTF-16 (I really wish we could just require
this for all Python source files and start towards this, but I know some
people would flip out over that).
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 16:26
Message:
Logged In: YES
user_id=6380
Originator: NO
Arguably we don't need this feature any more. The only reasonable way to
implement it would be to slurp the entire contents of the file into a
string and then exec that. The caller might as well do this.
Alternatively one could use execfile() (which takes a filename instead of a
stream). The only advantage of using exec() might be that it would
deal with encoding declarations transparantly; but we should probably have
a way to handle those somewhere in the library anyway, as it's also
necessary for other tools (e.g. modulefinder.py, pyclbr.py and
linecache.py). Alternatively I'm not so sure that we will need to support
encoding declarations forever -- I'm hoping that at some point UTF-8 and
BOM+UTF-16 will take over the world.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470
___
Python-bugs-list mailing list
[ python-Bugs-1767363 ] String.capwords() does not capitalize first word
Bugs item #1767363, was opened at 2007-08-03 15:48 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767363&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Saatvik Agarwal (saatvik) Assigned to: Nobody/Anonymous (nobody) Summary: String.capwords() does not capitalize first word Initial Comment: Perhaps not the right category...nothing major but I noticed that string.capwords does not capitalize the first character of the first word of the string. Email: [EMAIL PROTECTED] -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767363&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1762972 ] 'exec' does not accept what 'open' returns
Bugs item #1762972, was opened at 2007-07-28 20:24
Message generated for change (Comment added) made by bcannon
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1762972&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: Python 3000
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Brett Cannon (bcannon)
Assigned to: Guido van Rossum (gvanrossum)
Summary: 'exec' does not accept what 'open' returns
Initial Comment:
Since the move over to io.py 'exec' no longer accepts what 'open' returns.
Looks like there is a type check in 'exec' for strings, files, or code object,
but 'open' returns a TextIOWrapper that fails that typecheck.
--
>Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 15:50
Message:
Logged In: YES
user_id=357491
Originator: YES
Agreed on both, although what to do when there is no source but only
bytecode? Just say that __file__ will point to the source if available,
otherwise to the bytecode? That seems reasonable to me.
And imp.reload sounds good. This whole bug report started because
'reload' was gone and I was trying to figure out the best way to replace
its loss.
I know I am short on time and thus have no clue if I can help on these.
Should we add these to PEP 3100 so we don't forget, or do you think you
might get to this?
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 14:53
Message:
Logged In: YES
user_id=6380
Originator: NO
Or we can change the semantics of __file__ to always point to the source
file. I think that's acceptable behavior -- the current behavior seems
more of a pain than necessary.
I also think that adding imp.reload() makes sense; I was just saying
yesterday to a few fellow-Googler-Python-developers that I was already
missing reload().
--
Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 14:42
Message:
Logged In: YES
user_id=357491
Originator: YES
Hrm. Well then we can either add something like __source_file__ to
modules, put 'reload' into the imp module (or on module objects as a
method), or have a function in imp (or as a method on modules) that returns
the source path (if it exists).
But having to do::
M.__file__.rsplit('.', 1) + filter((lambda x : x[2] == imp.PY_SOURCE),
imp.get_suffixes())[0]
seems like a lot to memorize (let alone type in), especially since there
is even no error checking that the path is even to a source or bytecode
file to start with or that you end up with.
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 14:14
Message:
Logged In: YES
user_id=6380
Originator: NO
OK, let's ditch it (i.e. status quo) and also ditch execfile.
The one-liner you're looking for is *almost*
exec(open(M.__file__).read(), M.__dict__)
(where M stands for any already imported module), except for the nastiness
that if m.__file__ ends with .pyc or .pyo you'd have to strip the last
character. (And to do this right you'd need help from the imp module
because those extensions may vary by platform -- e.g. I know of a company
that patches their Pythons to use .pyc24.)
--
Comment By: Brett Cannon (bcannon)
Date: 2007-08-03 14:09
Message:
Logged In: YES
user_id=357491
Originator: YES
I am fine with ditching the feature and forcing people to deal with the
encoding issues externally to 'exec', although as you pointed out dealing
with any -*- encoding marker might be a pain without some library help
(although doesn't the parser pick up on this and handle those nasty
details?).
As for execfile, PEP 3100 has it listed for removal in favour of moving
people over to using 'exec'.
The main thing is trying to come up with an easy solution for replacing
'reload' with a reasonable one-liner when debugging at the interpreter.
And yes, I hope encoding declarations can go away and the world either
lives with just UTF-8 or BOM+UTF-16 (I really wish we could just require
this for all Python source files and start towards this, but I know some
people would flip out over that).
--
Comment By: Guido van Rossum (gvanrossum)
Date: 2007-08-03 13:26
Message:
Logged In: YES
user_id=6380
Originator: NO
Arguably we don't need this feature any more. The only reasonable way to
implement it would be to slurp the entire contents of the file into a
string and then exec that. The caller might as well do this.
Alternatively one could use execfile() (which takes a filename instead of a
stream). The only ad
[ python-Bugs-1767363 ] String.capwords() does not capitalize first word
Bugs item #1767363, was opened at 2007-08-03 15:48 Message generated for change (Settings changed) made by saatvik You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767363&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.6 Status: Open >Resolution: Invalid Priority: 5 Private: No Submitted By: Saatvik Agarwal (saatvik) Assigned to: Nobody/Anonymous (nobody) Summary: String.capwords() does not capitalize first word Initial Comment: Perhaps not the right category...nothing major but I noticed that string.capwords does not capitalize the first character of the first word of the string. Email: [EMAIL PROTECTED] -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767363&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1767363 ] String.capwords() does not capitalize first word
Bugs item #1767363, was opened at 2007-08-03 22:48 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767363&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.6 >Status: Closed Resolution: Invalid Priority: 5 Private: No Submitted By: Saatvik Agarwal (saatvik) Assigned to: Nobody/Anonymous (nobody) Summary: String.capwords() does not capitalize first word Initial Comment: Perhaps not the right category...nothing major but I noticed that string.capwords does not capitalize the first character of the first word of the string. Email: [EMAIL PROTECTED] -- >Comment By: Georg Brandl (gbrandl) Date: 2007-08-04 06:50 Message: Logged In: YES user_id=849994 Originator: NO Note that for byte strings, only letters in [a-z] will be correctly upcased. If you want it to handle arbitrary letters correctly, use unicode strings. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1767363&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
