strange subprocess behavior when calling ps
Hi all, I have encountered a strange problem with some code I am writing to search the system process list for certain running processes. I am using subprocess.Popen() to call '/bin/ps -e'. When I save my code to the file pid.py (whose first line is #!/usr/bin/python) and run it with the command % ./pid.py it works perfectly fine, retrieving lines from my pipe to the /bin/ps output which look exactly as if I had typed the command '/bin/ps -e' myself into a shell window. Here is a sample line from that output: 1891 ttys0000:00.12 -tcsh Now for the weird part -- when I run this code using the command % python pid.py I get entirely different output. It only prints out a very few processes instead of the entire table, and each line also has lots of environment variable values displayed. Here is the line from that output which corresponds to the line immediately above: 1891 s000 S+ 0:00.12 -tcsh PATH=/usr/bin:/bin:/usr/sbin:/sbin TMPDIR=/var/folders/3e/3e-TyTQIG-aOa4x37pbTbk++-H6/-Tmp-/ SHELL=/bin/ tcsh HOME=/Users/hmrgsoft USER=hmrgsoft LOGNAME=hmrgsoft DISPLAY=/tmp/ launch-c1YZNr/org.x:0 SSH_AUTH_SOCK=/tmp/launch-AJ9xbl/Listeners Apple_PubSub_Socket_Render=/tmp/launch-BsRx5Y/Render COMMAND_MODE=unix2003 __CF_USER_TEXT_ENCODING=0x532:0:0 TERM_PROGRAM=Apple_Terminal TERM_PROGRAM_VERSION=273 LANG=en_US.UTF-8 TERM=xterm-color It's like it's calling up an entirely different ps, or passing it different command arguments. In both cases, however, I am explicitly calling /bin/ps with the same -e argument, and there appear to be no other ps commands on my system, neither do I appear to have any ps builtin command in any shell. I am running 2.6.6 under MacOS 10.6.4 on a MacBook Pro Intel. I have appended the code below. I am running both commands directly in a Terminal window running tcsh. Can anyone explain this? Thanks! Roger Davis # code follows #!/usr/bin/python import sys import subprocess def main(): psargs= ["/bin/ps", "-e"] try: ps= subprocess.Popen(psargs, stdout=subprocess.PIPE, close_fds=True) psout= ps.communicate()[0] pslines= psout.splitlines() for line in pslines: print "%s" % line except KeyboardInterrupt: print "Keyboard interrupt received -- terminating." sys.stdout.flush() sys.exit(-1) except: print "%s: unexpected error in generation of system process list" % prognm sys.stdout.flush() sys.exit(-1) main() -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
Thanks, Chris, you're at least on the right track. I did upgrade from python.org and the python in my shell PATH is /Library/Frameworks/ Python.framework/Versions/2.6/bin/python: % python Python 2.6.6 (r266:84374, Aug 31 2010, 11:00:51) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin /usr/bin/python is the Apple-distributed 2.6.1: % /usr/bin/python Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin This still doesn't explain the behavior to me, however. In either case python is supposed to be forking a child process with a specific executable (/bin/ps) and a specific argument list (-e) and reading that command's output lines. Why should those output lines be different whether I use 2.6.1, 2.6.6 or 8.9.10 for that matter? In fact, this makes the problem that much worse -- the newer python 2.6.6 is the one producing the incorrect output. Changing the first line of the script to read #!/Library/Frameworks/Python.framework/Versions/2.6/bin/python does not help, it still prints out the wrong output whether I do % ./pid.py or % /Library/Frameworks/Python.framework/Versions/2.6/bin/python ./ pid.py Any ideas? Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
Hi James, Thanks for the pointer to psutil. I actually did look around on python.org before coding this up to see if there was such a package available but there is not, at least not where I'm looking -- on the other hand, I can't find my car keys most of the time. I would really like to restrict my code dependencies to the standard Python distribution (if there actually is such a thing, I'm new to the language). Of course, that's assuming that I can actually get my code to work in that fashion. If I can't get to the bottom of this issue and figure out why my existing code does not work I will investigate psutil further. Thanks! Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
Thanks, Ned! That really helps to explain what is going on. Now, just
a couple more questions and I think I will know all I need to know.
First, I *still* don't quite understand why this happens with my 2.6.6
interpreter but not my 2.6.1, and why another of the respondents to
this thread (Chris) could not duplicate this problem with his own
2.6.6 environment. Is there something defective with my Python
environment having to do with the 2.6.6 upgrade I installed directly
from python.org? (Chris used Fink, apparently.) If so, how can I clean
it up? I do have an easier time believing that this is a mangled
installation issue now that the problem has essentially been reduced
to the handling of an environment variable which I can well imagine
could be easily garbled somewhere in a version mismatch scenario.
Second, I do need to make this work on multiple platforms, primarily
Linux in addition to the Mac but also with a potential variety of
Python releases on both MacOS and Linux. I'm not sure if Linux will
object to this COMMAND_MODE business, I guess I'll just have to try
and see what happens. Any thoughts on that?
Finally, to split hairs a bit concerning your suggested solution, I
have a question about the additional env={'COMMAND_MODE': 'unix2003'}
argument. This works for me, but my reading of the subprocess
documentation leads me to believe that this will actually wipe out the
entire environment that would otherwise be inherited from the parent
and replace it with that single setting. I don't think this has any
ill effects here with regard to ps, at least none I can detect at the
moment, but wouldn't a perhaps safer solution be to do
os.environ['COMMAND_MODE']= 'unix2003'
prior to the Popen() to instead augment/correct the existing
environment which will then later be inherited by the child, assuming
no explicit env= optional argument is used? This also works for me,
and I think I'm more inclined to go this route unless you can think of
a good reason not to do so.
Thanks very much for your help!
--
http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
> Completely off topic but I think the try clause could be rewritten that way: > ... > Don't use bare except clause, you're masking syntax errors for instance, > which will be flagged as 'unexpected error in generation ...". > In a more general manner, if something unexpected happens it's better to > just let the exception raise uncought. If you want to handle some > errors, meaning you're kindof expecting them then add a explicit clause > (like you did with KeyboardInterrupt). > > JM > > PS : "except Exception :" will catch most of the exceptions (all > inheriting from that class). It's better than using a bare "except :" > clause. (Exception won't catch SyntaxError) Thanks for the suggestion JM, it is off-topic and, although I will first just say that the exception mechanism is *not* one of the reasons I use Python (and stop there with regard to the whole exception mechanism and various usage strategies in general), I do have a few specific questions about a couple of your statements if you don't mind following up. First, inserting a syntax error into my existing code does not hide a SyntaxError exception as you have stated: % cat pid.py #!/usr/bin/python import os import sys import subprocess def main(): psargs= ["/bin/ps", "-e"] try: ps= subprocess.Popen(psargs, stdout=subprocess.PIPE, close_fds=True) psout= ps.communicate()[0] pslines= psout.splitlines() if pslines not good Python talky-talk for line in pslines: print "%s" % line except KeyboardInterrupt: print "Keyboard interrupt received -- terminating." sys.stdout.flush() sys.exit(-1) except: print "%s: unexpected error in generation of system process list" % prognm sys.stdout.flush() sys.exit(-1) main() % ./pid.py File "./pid.py", line 14 if pslines not good Python talky-talk ^ SyntaxError: invalid syntax It appears that the interpreter is catching the syntax error before the code is even executed. Second, python.org's exception hierarchy documentation (Section 6.1 at http://docs.python.org/library/exceptions.html) shows all exception types except SystemExit, KeyboardInterrupt and GeneratorExit as being descendants of Exception. This includes SyntaxError, a child of StandardError which is itself a child of Exception. So, if I say 'except Exception:' then isn't that clause going to process any child exception type of Exception (including SyntaxError) that I haven't already covered in a separate except clause? (Except of course that my interpreter doesn't seem to treat a syntax error as any kind of exception at all!) Finally, and this does not apply to your comments in particular, in researching around about exception handling I often see the usage except Exception, e: suggested, but can't for the life of me figure out what the heck the ', e' part does. Can anybody explain what this means and why it might be desirable (or not)? Thanks! -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
On Nov 16, 11:19 pm, Ned Deily wrote: > Interesting. It appears that OS X 10.6 takes into account the ... Thanks very much for your thorough explanation, Ned! I think I've got what I need now. Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
Thanks for the clarification on exceptions, Chris! Roger -- http://mail.python.org/mailman/listinfo/python-list
Re: strange subprocess behavior when calling ps
Hi JM, Thank you very much for your followup explanation! Roger -- http://mail.python.org/mailman/listinfo/python-list
what happens to Popen()'s parent-side file descriptors?
Hi, I am new to this group, please forgive me if this is a repeat question. I am a new Python programmer but experienced in C/Unix. I am converting a shell script to Python which essentially loops infinitely, each pass through the loop running commands like set output = `cat this | grep that | whatever ...` My understanding is that this functionality is best coded via subprocess.Popen(). I need to read output from these spawned children via a pipe from their stdout, hence something like p= subprocess.Popen(args, stdout=subprocess.PIPE) This means that somewhere a pipe file descriptor is opened on the parent side to read from the child's stdout. When, if ever, is that descriptor closed? Per-process FDs are limited and I am looping infinitely so I need to be very careful about not running out of them. Are there any other FDs related to this operation that also need to be closed? Testing with the interpreter (2.6, MacOSX) it appears that p.stdout is being closed somehow by someone other than me: import subprocess args= ["echo", "This is a mystery!"] i= 0 while True: p= subprocess.Popen(args, stdout=subprocess.PIPE) for line in p.stdout: print "[%5d] %s" % (i, line.strip()) i+= 1 The above code closes nothing but appears to run indefinitely without running the parent out of FDs. WTF is going on here? Popen.communicate() similarly appears to be closing the parent's pipe FDs, although that seems more understandable as it appears to be designed to encapsulate a lot of cleanup activity. In either case (code snippet above or communicate() an attempt to manually close p,stdout goes as follows: >>> print p.stdout.close() None Attempts to close anything else fail spectacularly as one might expect: >>> print p.stdin.close() Traceback (most recent call last): File "", line 1, in AttributeError: 'NoneType' object has no attribute 'close' Can anyone explain the treatment of the pipe FDs opened in the parent by Popen() to me or point me to some documentation? The python.org docs say absolutely nothing about this as far as I can tell, a glaring deficiency IMO (hint to all you underworked volunteer developers ;- > ). Also, does Popen.returncode contain only the child's exit code or is does it also contain signal info like the return of os.wait()? Documentation on this is also unclear to me. Thanks very much! -- http://mail.python.org/mailman/listinfo/python-list
Re: what happens to Popen()'s parent-side file descriptors?
Many thanks to all who responded to my question! It's nice to know, as someone new to Python, that there are lots of well-informed people out there willing to help with such issues. Thanks, Mike, for your pipes suggestion, I will keep that in mind for future projects. Seebs, you are of course correct that the example I quoted (`cat | grep | whatever`) is best done internally with the re module and built- in language features, and in fact that has already been done wherever possible. I should have picked a better example, there are numerous cases where I am calling external programs whose functionality is not duplicated by Python features. 'Nobody' (clearly a misnomer!) and Chris, thanks for your excellent explanations about garbage collection. (Chris, I believe you must have spent more time looking at the subprocess source and writing your response than I have spent writing my code.) GC is clearly at the heart of my lack of understanding on this point. It sounds like, from what Chris said, that *any* file descriptor would be closed when GC occurs if it is no longer referenced, subprocess-related or not. BTW, and this comment is not at all intended for any of you who have already very generously and patiently explained this stuff to me, it does seem like it might be a good idea to provide documentation on some of these more important GC details for pretty much any class, especially ones which have lots of murky OS interaction. I have to admit that in this case it makes perfect sense to close parent pipe descriptors there as I can't think of any reason why you might want to keep one open after your object is no longer referenced or your child exits. It sounds to me that, although my code might be safe now as is, I probably need to do an explicit p.stdXXX.close() myself for any pipes which I open via Popen() as soon as I am done with them. Documentation on python.org states that GC can be postponed or omitted altogether, a possibility that Chris mentions in his comments. Other documentation states that there is no harm in doing multiple close()es on the same file, so I assume that neither my code nor the subprocess GC code will break if the other does the deed first. If anybody thinks this is a bad idea, please comment. On a related point here, I have one case where I need to replace the shell construct externalprog otherfile I suppose I could just use os.system() here but I'd rather keep the Unix shell completely out of the picture (which is why I am moving things to Python to begin with!), so I'm just doing a simple open() on somefile and otherfile and then passing those file handles into Popen() for stdin and stdout. I am already closing those open()ed file handles after the child completes, but I suppose that I probably should also explicitly close Popen's p.stdin and p.stdout, too. (I'm guessing they might be dup()ed from the original file handles?) Thanks again to all! -- http://mail.python.org/mailman/listinfo/python-list
IDLE debugger questions
Hi, I have some questions about the IDLE debugger. I am using the 2.6.6 bundle downloaded from python.org. First, how do I debug a Python program that requires command-line args? I usually run it with a command like % test.py arg1 arg2 arg3 There seems to be practically no detailed IDLE debugger documentation and I have been unable to figure out how to pass these arguments into the debugger. Second, some folk say you can set a breakpoint by right-clicking on the desired line in the source window to get a popup menu. This does not work for me on my MacBook (external 3-button USB mouse), it only adds a line-break into the source code. Any suggestions? Thanks! Roger Davis -- http://mail.python.org/mailman/listinfo/python-list
Re: IDLE debugger questions
Thanks for that info, Ned, I can now get the sys.argv[] list I need, that's a big help! However, is there any other way to set a breakpoint in idle that will work on Mac OS X, maybe entering a manual command somewhere with a specified line number? Inability to set a breakpoint is an absolute showstopper for me, the code I am trying to debug loops way too much to be manually steppable in a human time frame. I also spent a couple of hours trying to get ddd to work, and oddly enough encountered a similar problem there. After downloading and installing pydb I was able to get things to start up OK. You can set a breakpoint, meaning that you can click on the source line and the little stop sign icon pops up right there as expected, but when you run the code the debugger just blows right by it like it's not even there. Does anyone out there have a fix for this? It sounds like ddd Python support is highly problematic at the moment, supposedly it works only with pydb but pydb's author has apparently moved on to a different debugger project (pydbgr?) that I suspect has little chance of working with ddd just yet. If anyone has evidence to the contrary please let me know! Are there any Python debuggers with a decent GUI out there at all that will work on a Mac with the following features: (i) ability to pass in a sys.srgv[] list that the program would otherwise see without the debugger, (ii) display local variables, (iii) single-step through the source code, and (iv) set a breakpoint? I think I can live without anything else for the moment. Thanks! -- http://mail.python.org/mailman/listinfo/python-list
