Re: [Python-Dev] Status of 2.7 and 3.2

[Eric Pruitt]

I am in the process of implementing a number of often requested features and
proposed patches in the subprocess module for my Google Summer of Code 2009
project. For information on my progress, check out my blog located at *
http://subdev.blogspot.com/* . Any comments and
suggestions are greatly appreciated.

> Just my 0.02 cents, but struggling with all warts of 2.5 subprocessing
> > in Windows I would vote for more time for stabilizating things - not
> > adding new features. Long awaited subprocess as replacement for
> > os.popen() AFAIK is still incapable to asynchronously communicate with
> > spawned processes on Windows. I would call this feature as critical
> > even on 2.6  As a release testcase - try porting pyexpect module to
> > this platform. Absence of native curses/console/readline module also
> > makes Python one-way unix shell language while many users expect it to
> > be crossplatform.
>
> I am not quite sure whether you are for new features or not. Your
> first sentence ("vote for ... not adding new features") seems to
> suggest that you would not like to see new features, and your last
> sentence ("absence of native curses/console/readline module")
> suggests that you *do* want to see new features (namely, a native
> curses module, and a native readline module).
>
> Which one is it?
>
> Regards,
> Martin
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Popen asynchronous input for Windows

[Eric Pruitt]

Hello,

I am looking for alternatives to Josiah Carlson's asynchronous I/O patch for
subprocess.Popen. While his patch seems to work well, it relies on pywin32
which is not part of the standard Python library. If I cannot find an
alternative, I will be using cTypes with the parts of Mark Hammond's code
that I need, license permitting. Any suggestions are greatly appreciated.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Popen asynchronous input for Windows

[Eric Pruitt]

Thanks for the lead. I have the pywin32 source code and have found the files
that appear to contain the code I need inside of some "*.i" files. After a
bit of Googling and paying attention to the blatantly obvious *.cpp files, I
realized Hammond's code is written in C++ whereas Python uses C. I am
familiar with neither language very well but if I can't find a work-around
for Windows asynchronous I/O, I will be learning a bit of both of them.

Please feel free to suggest any other ideas that don't rely on my sparse
knowledge of C and C++.

Eric

On Thu, Jun 18, 2009 at 13:32, Christian Heimes  wrote:

> Eric Pruitt schrieb:
> > Hello,
> >
> > I am looking for alternatives to Josiah Carlson's asynchronous I/O patch
> for
> > subprocess.Popen. While his patch seems to work well, it relies on
> pywin32
> > which is not part of the standard Python library. If I cannot find an
> > alternative, I will be using cTypes with the parts of Mark Hammond's code
> > that I need, license permitting. Any suggestions are greatly appreciated.
>
> The subprocess module several wrappers for win32 APIs in
> Modules/_subprocess.c. You could add the necessary functions.
>
> Christian
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Bytes, Strings, Encoding

[Eric Pruitt]

Hello,

I am working on the subprocess.Popen module for Python 2.7 and am now moving
my changes over to Python 3.1 however I am having trouble with the whole
byte situation and I can't quite seem to understand how to go back and forth
between bytes and strings. I am also looking for the Python 3k equivalent
for the Python 2.X built-in buffer class.

One version of the file with my modifications can be found here  <
http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9>.
Lines 1 - 15 are me attempting to get around certain changes between
Python 3.0 and Python 2.7. Further down on line 916, we have the function
"send" and "recv" in which I am having the most trouble with bytes and
strings.

Any help is appreciated. Feel free to comment on my blog
http://subdev.blogspot.com/ or reply to the last.

Thanks in advance,
Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Windows Toolchain

[Eric Pruitt]

Hello,

I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
Visual C++ Express Edition in order to test some modifications I made to the
_subprocess.c file as part of my Google Summer of Code proposal. I cannot
seem to figure out how to compile Python on Windows and could use some
guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
work when I attempt to open and no error message is given when clicked. When
opened from the Visual C++, it says "Solution folders are not supported in
this version of the application."

Any advice is greatly appreciated,
Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

Sorry, forgot to include my build log:

Command Lines:
http://pastebin.com/f25614b01

Output Window:
> Compiling...
> python.c
> Compiling resources...
> Microsoft (R) Windows (R) Resource Compiler Version 6.1.6723.1
> Copyright (C) Microsoft Corporation.  All rights reserved.
> Linking...
> LINK : fatal error LNK1181: cannot open input file '.\python31_d.lib'
> Creating browse information file...
> Microsoft Browse Information Maintenance Utility Version 9.00.30729
> Copyright (C) Microsoft Corporation. All rights reserved.

On Mon, Jul 13, 2009 at 10:46, Eric Pruitt  wrote:

> Hello,
>
> I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft
> Visual C++ Express Edition in order to test some modifications I made to the
> _subprocess.c file as part of my Google Summer of Code proposal. I cannot
> seem to figure out how to compile Python on Windows and could use some
> guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
> work when I attempt to open and no error message is given when clicked. When
> opened from the Visual C++, it says "Solution folders are not supported in
> this version of the application."
>
> Any advice is greatly appreciated,
> Eric
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

I opened the solution, hit Ctrl+F5 and it began compiling but it fails with
this error:
LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib'

How do I specify that "pythoncore" be built? When I follow your
instructions, right clicking on the "python" project in the "Solution
Explorer" panel, I get the exact same link error as above. If I hit F7, the
build log changes completely. The last lines of output with F7 are as
follows:

> 21>Build log was saved at "file://C:\Documents and
Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm"
> 21>select - 1 error(s), 0 warning(s)
> == Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped
==


On Mon, Jul 13, 2009 at 11:03, Amaury Forgeot d'Arc wrote:

> Hi,
>
> 2009/7/13 Eric Pruitt :
> > Hello,
> >
> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> Microsoft
> > Visual C++ Express Edition in order to test some modifications I made to
> the
> > _subprocess.c file as part of my Google Summer of Code proposal. I cannot
> > seem to figure out how to compile Python on Windows and could use some
> > guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not
> > work when I attempt to open and no error message is given when clicked.
>
> Is Visual C++ correctly installed?
>
> > When
> > opened from the Visual C++, it says "Solution folders are not supported
> in
> > this version of the application."
>
> This is a warning from the Express Edition, which has absolutely no
> impact on the build. You can safely ignore it and continue.
>
> What steps did you perform exactly?  From your build log the
> "pythoncore" project was not built. Why?
>
> The following has always worked for me: In the "Solution explorer"
> panel, right-click on the "python" project, and click "Build". This
> should build the project and all its dependencies.
> Or just hit the F7 key and watch the whole solution build.
>
> --
> Amaury Forgeot d'Arc
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be
Unix style but after fixing them, I still have the same problem.

On Mon, Jul 13, 2009 at 11:43, Amaury Forgeot d'Arc wrote:

> 2009/7/13 Eric Pruitt :
> > I opened the solution, hit Ctrl+F5 and it began compiling but it fails
> with
> > this error:
> > LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib'
> >
> > How do I specify that "pythoncore" be built? When I follow your
> > instructions, right clicking on the "python" project in the "Solution
> > Explorer" panel, I get the exact same link error as above. If I hit F7,
> the
> > build log changes completely. The last lines of output with F7 are as
> > follows:
> >
> >> 21>Build log was saved at "file://C:\Documents and
> >>
> Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm"
> >> 21>select - 1 error(s), 0 warning(s)
> >> == Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped
> >> ==
> >
>
> Did you open the sub-project "python.vcproj" by mistake?
> You must open the solution file: "pcbuild.sln".
>
> And if pcbuild.sln does not load correctly, check that it has correct
> line endings: it is supposed to have DOS line endings.
>
> --
> Amaury Forgeot d'Arc
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

The code I downloaded was the official Python 3.1 release tarball. I only
checked the line endings in the pcbuild.sln file so I am not sure about the
other files. I will attempt to build the Python 3000 SVN trunk shortly and
report back.

On Mon, Jul 13, 2009 at 13:09, Paul Moore  wrote:

> 2009/7/13 Eric Pruitt :
> > It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to
> be
> > Unix style but after fixing them, I still have the same problem.
>
> Where did you get your copy of the Python source from? If it's from
> Subversion, I'm surprised the line endings are wrong. If it's
> Mercurial, then this may be indicative of the still somewhat open
> question of how line endings will be handled in Mercurial (which
> doesn't have a server-side way of saying that a file should use
> platform-native line endings).
>
> If you still have the same problem after fixing the file, this may
> indicate further damage, or maybe the fix isn't doing the right thing
> as the file has already been opened (and mangled) by Visual Studio,
>
> I'd suggest trying to reproduce your issue with a clean checkout from
> Subversion.
>
> It would also help if you could describe precisely the steps you took
> to download the sources and build them (I appreciate that this may not
> be easy now, after you've tried various things).
>
> Paul.
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

Well, I narrowed the culprit down to Windows DEP. I had been having trouble
with it for some time now, with it raising an error for nearly every program
I ran so I just disabled it completely and Python now compiles fine. Thank
you all for the assistance.

On Mon, Jul 13, 2009 at 13:42, Paul Moore  wrote:

> 2009/7/13 Paul Moore :
> > 2009/7/13 Eric Pruitt :
> >> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to
> be
> >> Unix style but after fixing them, I still have the same problem.
> [...]
> > I'd suggest trying to reproduce your issue with a clean checkout from
> > Subversion.
>
> FWIW, I did:
>
> svn co http://svn.python.org/projects/python/tags/r31/ py31
>
> Open pcbuild in Visual C++ 2008 Express. OK on the warning about
> solution folders.
>
> Right click "python" and choose "Build".
>
> Works OK here. Also works if I download Python-3.1.tgz from python.org
> and unpack it using (command line) bsdtar. I wonder - did you use
> WinZip to unpack the archive? That has some sort of brain-damaged
> "Smart" CRLF conversion for tar files, which may have screwed up your
> files.
>
> Paul.
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

I will keep a grace period in mind when I am posting. In my defense,
however, I had been working on the problem for some time before posting this
and my messages in quick succession were in response to the suggestions
people offered. The problem has arisen again but I will work on it some more
and will also post a detailed post on my GSoC blog of the process I went
through when I encountered the issue.

On Mon, Jul 13, 2009 at 16:24, "Martin v. Löwis"  wrote:

> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> > Microsoft Visual C++ Express Edition in order to test some modifications
> > I made to the _subprocess.c file as part of my Google Summer of Code
> > proposal.
>
> As a posting guideline, please be careful to not post too many messages
> in quick succession. If you run into a problem, try to solve it for
> yourself. If you fail, consult your mentor and/or wait a couple of
> hours. Then, when posting, summarize your findings and the alternatives
> that you tried unsuccessfully.
>
> If people learn that you post while being in the process of working
> on a problem, they will learn that it is better to wait a day or two
> before responding, in case you figure it out on your own.
>
> Regards,
> Martin
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Toolchain

[Eric Pruitt]

The problem appears resolved again and I have two posts on the issue on my
blog located at http://subdev.blogspot.com/. I was missing an MSVC++
run-time DLL and re-installing Visual C++ Express fixed the problem. A bit
of a somewhat random note -- looking through some of the buildbot output for
Python 3.1, I noticed that at one point, the bot compilations had the same
issue; Google 
Cache
.

On Mon, Jul 13, 2009 at 16:24, "Martin v. Löwis"  wrote:

> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using
> > Microsoft Visual C++ Express Edition in order to test some modifications
> > I made to the _subprocess.c file as part of my Google Summer of Code
> > proposal.
>
> As a posting guideline, please be careful to not post too many messages
> in quick succession. If you run into a problem, try to solve it for
> yourself. If you fail, consult your mentor and/or wait a couple of
> hours. Then, when posting, summarize your findings and the alternatives
> that you tried unsuccessfully.
>
> If people learn that you post while being in the process of working
> on a problem, they will learn that it is better to wait a day or two
> before responding, in case you figure it out on your own.
>
> Regards,
> Martin
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile

[Eric Pruitt]

Hello,

I have written replacements for a few of Mark Hammond's PyWin32 functions
using ctypes to call upon the Windows kernel API. Code can be found here;
http://pastebin.com/m46de01f . When calling ReadFile, it appears that the
kernel API converts '/n' newlines '/r/n' newlines. I have not been able to
find any information about this but as far as I can tell, there is nothing
in my code that is causing the problem. Initially I suspected it related to
newline translation but the function in subprocess.Popen for translating
newlines only converts to '/n' newlines. When I replaced my ReadFile and
WriteFile functions with the PyWin32 functions I was imitating, the problem
still arose. I was hoping someone could confirm this problem for me from
experience or by testing out my code. If you would like the see the
functions used in full context, I have a Mercurial Google Code project that
can be checked out at
http://code.google.com/p/subprocdev/source/list(branch "python3k"). My
work entails modifications to subprocess.py so when
running the code, please update your PYTHONPATH variable so that my
subprocess.py will be imported.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Implementing File Modes

[Eric Pruitt]

Hello,

Since there was a bit of confusion last time, I'll start by saying I am
working on the subprocess.Popen module for Google Summer of Code. One of the
features I am implementing is a class so that a running process can stand in
in place of a file. For examples, instead of open( "filelist", mode = 'r')
one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide
if I should fully implement the mode argument. Right now, it essentially
ignores everything but a 'U' indicated universal newlines in the mode
argument. Should I leave that as is or make it so that things like "r+",
"w", "a" are handled the way they would be for an actual file?

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

I am implementing the file wrapper using changes to subprocess.Popen that
also make it asynchronous and non-blocking so implementing "r+" should be
trivial to do. How about handling stderr? I have the following ideas: leave
out support for reading from stderr, make it so that there is an optional
additional argument like "outputstderr = False", create another function
that toggles / sets whether stderr or stdout is returned or mix the two
outputs.

Thanks for the input,
Eric

On Mon, Jul 27, 2009 at 10:46, Paul Moore  wrote:

> 2009/7/27 Eric Pruitt :
> > Hello,
> >
> > Since there was a bit of confusion last time, I'll start by saying I am
> > working on the subprocess.Popen module for Google Summer of Code. One of
> the
> > features I am implementing is a class so that a running process can stand
> in
> > in place of a file. For examples, instead of open( "filelist", mode =
> 'r')
> > one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to
> decide
> > if I should fully implement the mode argument. Right now, it essentially
> > ignores everything but a 'U' indicated universal newlines in the mode
> > argument. Should I leave that as is or make it so that things like "r+",
> > "w", "a" are handled the way they would be for an actual file?
>
> I would expect "r" to produce a pipe that reads from stdout of the
> subprocess, and "w" to produce a pipe that writes to stdin of the
> subprocess. "a" would be the same as "w", and arguably "r+" would be a
> bidirectional pipe - read from the subprocess stdout and write to its
> stdin.
>
> I'd be OK with "r+" not being implemented (if it's too hard to avoid
> deadlocks) but "r" and "w" should be present.
>
> Paul.
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

My motivation came from an instance when I was using subprocess.Popen for a
Linux / Windows cross platform program. In part of the program, I was
writing and reading to a cron like object. On Windows, it was a text file
and on Linux it would be the crontab executable. Had I been able to
substitute the "open()" function with my wrapper, it would have been the
only change I had to make for cross platform compatibility; instead of
having to change numerous lines because Linux would need Popen and Windows
would need a regular file open(), I could simply make it so that if the
platform was Linux, my wrapper is used in place of that. Just another
example would be having an external program decrypt a file that can be in
plain text or encrypted that might go something like this:

if encryptionEnabled:
fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
supersecret.html.gpg")
else:
fileobj = open("notsosecret.html")

>From there, the functions would not have to be modified despite using a
running process versus a file object.

On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith  wrote:

> On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz
> wrote:
> > On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore  wrote:
> >>
> >> I like MRAB's idea of using a (non-standard) "e" flag to include
> >> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
> >>
> >> Anything more complicated probably should just use "raw" Popen
> >> objects. Don't overcomplicate the interface.
> >
> > In my opinion, mangling stderr and stdout together is already an
> > overcomplication.  It shouldn't be implemented.
> >
> > It seems like a good idea, until you realize that subtle changes to your
> OS,
> > environment, or buffering behavior may result in arbitrary, unparseable
> > output.
>
> Agreed.  Leave stderr support out of this.  People who need stderr
> should use the full subprocess.Popen interface.  Quick hack unixy
> types will just run their process using a shell (which already seems
> to be the default based on the "ls -l" example) and add 2>&1.  This
> functionality is basically the equivalent of adding the | symbol on
> either or both ends of a filename given to open() in perl.  (but
> without being so gross).
>
> I do wonder why you're trying to make it behave exactly like open()
> including the mode= syntax.
>
> Why not just define several names based on the behavior?
>
>  ProcessReadWrapper()
>  ProcessWriteWrapper()
>  ProcessReadWriteWrapper()
>
> -gps
>
> >
> > For example, let's say you've got a program whose output is a list of
> lines,
> > each one containing a number.  Sometimes it tries to import gtk, and
> fails
> > to open its display.
> >
> > That's fine, and you can still deal with it, as long as the interleaved
> > output looks like this:
> >
> > 100
> > 200
> > Gtk-WARNING **: cannot open display:
> > 300
> > 400
> >
> > but of course the output might (although unlikely with such small chunks
> of
> > output) end up looking like this, instead:
> >
> > 100
> > 2Gtk-WAR0NING0 **:
> > can30not 0open display:
> >
> > 400
> >
> > this is the sort of thing which is much more likely to happen once you
> start
> > dealing with large volumes of data, where there are more page-boundaries
> for
> > your buffers to get confused around, and you are playing with buffering
> > options to improve performance.  In other words, it's something that
> fails
> > only at scale or under load, and is therefore extremely difficult to
> debug.
> >
> > This option might be okay if it were allowed only on subprocesses opened
> in
> > a text mode, and if the buffering logic involved forced stderr and stdout
> to
> > be line-delimited, and interleave only lines, rather than arbitrary
> chunks
> > of bytes.  Of course then if you use this flag with a program that
> outputs
> > binary data with no newlines it will buffer forever and crash your
> program
> > with a MemoryError, but at least that's easy to debug when it happens.
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

Well, with a few changes to your code, that would indeed work (you are using
stdin as your pipe. Correct me if I'm wrong but if you intend to read from
it, you need to change it to "stdout = subprocess.PIPE" and the other lines
as well to reflect this change). My Google Summer of Code modifications to
subprocess.Popen provide non-blocking, asynchronous I/O support and my file
wrapper is built upon that augmented functionality. If I remember correctly,
when I was working on the program where I first thought a file wrapper for
subprocess.Popen would be rather handy, I also ran into blocking I/O as
well.

On Wed, Jul 29, 2009 at 20:20, Devin Cook  wrote:

> Hmm... can't you do this?
>
> if encryptionEnabled:
>p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"],
> stdin = subprocess.PIPE)
>fileobj = p.stdin
> else:
>fileobj = open("notsosecret.html")
>
> I think that works. Is there something this way won't work for? You
> can also do the same thing to get stdout and stderr file objects. I
> guess a wrapper would simplify this process.
>
> -Devin
>
> On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt wrote:
> > My motivation came from an instance when I was using subprocess.Popen for
> a
> > Linux / Windows cross platform program. In part of the program, I was
> > writing and reading to a cron like object. On Windows, it was a text file
> > and on Linux it would be the crontab executable. Had I been able to
> > substitute the "open()" function with my wrapper, it would have been the
> > only change I had to make for cross platform compatibility; instead of
> > having to change numerous lines because Linux would need Popen and
> Windows
> > would need a regular file open(), I could simply make it so that if the
> > platform was Linux, my wrapper is used in place of that. Just another
> > example would be having an external program decrypt a file that can be in
> > plain text or encrypted that might go something like this:
> >
> > if encryptionEnabled:
> > fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
> > supersecret.html.gpg")
> > else:
> > fileobj = open("notsosecret.html")
> >
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Implementing File Modes

[Eric Pruitt]

> On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote:
> > I am implementing the file wrapper using changes to subprocess.Popen
> > that also make it asynchronous and non-blocking so implementing "r+"
> > should be trivial to do. How about handling stderr? I have the
> > following ideas: leave out support for reading from stderr, make it
> > so that there is an optional additional argument like "outputstderr =
> > False", create another function that toggles / sets whether stderr or
> > stdout is returned or mix the two outputs.
>
> Leaving it out is always an option.
>
> As I see it, fundamentally you can either read from stdout and sterr as
> two different streams, or you can interleave (mix) them. To me, that
> suggests three functions:
>
> ProcessIOWrapper()  # read from stdout (or write to stdin etc.)
> ProcessIOWrapperStdErr()  # read/write from stderr
> ProcessIOWrapper2()  # read from mixed stdout and stderr
>
> I don't like a function to toggle between one and the other: that smacks
> of relying on a global setting in a bad way. I suppose you could add an
> optional argument to ProcessIOWrapper() to select between stdout,
> stderr, or both together.
>
>
>
> --
> Steven D'Aprano

How would having a toggle function rely on a global setting? Each class would
simply have its own member variable like "self.readsstderr." It's a moot point
though as I've decided to use separate functions as you suggested. With
separate functions, the user doesn't have to worry about modifying the mode
keyword if stderr is needed and as an added bonus, it also makes the code
a lot more readable.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Functionality in subprocess.Popen.terminate()

[Eric Pruitt]

In my GSoC project, I have implemented asnychronous I/O in subprocess.Popen.
Since the read/write operations are asynchronous, the program may have
already exited by the time one calls the asyncread function I have
implemented. While it returns the data just fine, I have come across an
issue with the TerminateProcess function in Windows: if the program has
already exited, when subprocess.Popen.Terminate calls the Windows built-in
"TerminateProcess" function, an "access denied" error will occur. Should I
just make it so that this exception is simply ignored or perform some kind
of check to see if the process exists beforehand? If the latter, I have been
unable to find a way to do so, to my liking at least. The solutions I saw
would require code that seems a bit excessive to me.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Functionality in subprocess.Popen.terminate()

[Eric Pruitt]

On Tue, Aug 4, 2009 at 04:27, Nick Coghlan wrote:
> Eric Pruitt wrote:
>> In my GSoC project, I have implemented asnychronous I/O in
>> subprocess.Popen. Since the read/write operations are asynchronous, the
>> program may have already exited by the time one calls the asyncread
>> function I have implemented. While it returns the data just fine, I have
>> come across an issue with the TerminateProcess function in Windows: if
>> the program has already exited, when subprocess.Popen.Terminate calls
>> the Windows built-in "TerminateProcess" function, an "access denied"
>> error will occur. Should I just make it so that this exception is simply
>> ignored or perform some kind of check to see if the process exists
>> beforehand? If the latter, I have been unable to find a way to do so, to
>> my liking at least. The solutions I saw would require code that seems a
>> bit excessive to me.
>
> I'm pretty sure we already ignore some spurious error messages in cases
> like calling flush() in file.close(). I would suggest checking what the
> io module does in such cases and see what kind of precedent it sets.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ---
>

Sounds good enough to me but I was wondering if it might be a good
idea to add a function like "pidinuse" to subprocess as a whole that
would determine if a process ID was being used and return a simple
boolean value. I came across a number of people searching for a way to
determine if a PID was running (Google "python check if pid exists")
so it seems like the implemented functionality would be of use to the
community as a whole, not just my wrapper class.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (try-except) conditional expression similar to (if-else) conditional (PEP 308)

[Eric Pruitt]

What about catching specific error numbers? Maybe an option so that
the dictionary elements can also be dictionaries with integers as the
keys:

filedata = try_3(open, randomfile, except = { IOError,  {2: None} } )

If it isn't found in the dictionary, then we raise the error.

On Thu, Aug 6, 2009 at 07:03, ilya wrote:
> I took a look at the options 1 and 2:
>
>    x = float(string) except float('nan') if ValueError
>    y = float(string) except ValueError: float('nan')
>
> and I think this can be done just as easily with existing syntax:
>
>    x = try_1(float, string, except_ = float('nan'), if_ = ValueError)
>    y = try_2(float, string, { ValueError: float('nan') })
>
> Here's the full example:
>
> - example starts -
>
> def try_1(func, *args, except_ = None, if_ = None):
>    try:
>        return func(*args)
>    except if_ as e:
>        return except_
>
> def try_2(func, *args):
>    'The last argument is a dictionary {exception type: return value}.'
>    dic = args[-1]
>    try:
>        return func(*args[:-1])
>    except Exception as e:
>        for k,v in dic.items():
>            if isinstance(e, k):
>                return v
>        raise
>
> for string in ['5', 'five']:
>    #   x = float(string) except float('nan') if ValueError
>    x = try_1(float, string, except_ = float('nan'), if_ = ValueError)
>    #   y = float(string) except ValueError: float('nan')
>    y = try_2(float, string, { ValueError: float('nan') })
>    print(x, y)
>
> - example ends -
>
> As a side note, if I just subscribed to python-dev, is it possible to
> quote an old email? Below is my manual cut-and-paste quote:
>
> -- my quote --
>
> Nick Coghlan wrote:
>> P.J. Eby wrote:
>>> At 05:59 PM 8/5/2009 -0700, Raymond Hettinger wrote:
 [Jeffrey E. McAninch, PhD]
> I very often want something like a try-except conditional expression
> similar
> to the if-else conditional.
>
> An example of the proposed syntax might be:
>    x = float(string) except float('nan')
> or possibly
>    x = float(string) except ValueError float('nan')
 +1 I've long wanted something like this.
 One possible spelling is:

   x = float(string) except ValueError else float('nan')
>>> I think 'as' would be better than 'else', since 'else' has a different
>>> meaning in try/except statements, e.g.:
>>>
>>>    x = float(string) except ValueError, TypeError as float('nan')
>>>
>>> Of course, this is a different meaning of 'as', too, but it's not "as"
>>> contradictory, IMO...  ;-)
>>
>> (We're probably well into python-ideas territory at this point, but I'll
>> keep things where the thread started for now)
>>
>> The basic idea appears sound to me as well. I suspect finding an
>> acceptable syntax is going to be the sticking point.
>>
>> Breaking the problem down, we have three things we want to separate:
>>
>> 1. The expression that may raise the exception
>> 2. The expression defining the exceptions to be caught
>> 3. The expression to be used if the exception actually is caught
>>
>>>From there it is possible to come up with all sorts of variants.
>>
>> Option 1:
>>
>> Change the relative order of the clauses by putting the exception
>> definition last:
>>
>>   x = float(string) except float('nan') if ValueError
>>   op(float(string) except float('nan') if ValueError)
>>
>> I actually like this one (that's why I listed it first). It gets the
>> clauses out of order relative to the statement, but the meaning still
>> seems pretty obvious to me.
>>
> A further extension (if we need it):
>
>     result = foo(arg) except float('inf') if ZeroDivisionError else
> float('nan')
>
> The 'else' part handles any other exceptions (not necessarily a good idea!).
>
> or:
>
>     result = foo(arg) except float('inf') if ZeroDivisionError else
> float('nan') if ValueError
>
> Handles a number of different exceptions.
>
>> Option 2:
>>
>> Follow the lamba model and allow a colon inside this form of expression:
>>
>>   x = float(string) except ValueError: float('nan')
>>   op(float(string) except ValueError: float('nan'))
>>
>> This has the virtue of closely matching the statement syntax, but
>> embedding colons inside expressions is somewhat ugly. Yes, lambda
>> already does it, but lambda can hardly be put forward as a paragon of
>> beauty.
>>
> A colon is also used in a dict literal.
>
>> Option 3a/3b:
>>
>> Raymond's except-else suggestion:
>>
>>   x = float(string) except ValueError else float('nan')
>>   op(float(string) except ValueError else float('nan'))
>>
> [snip]
> -1
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/eric.pruitt%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
h

[Python-Dev] PEP Submission

[Eric Pruitt]

Several days ago, around the time the python.org servers went down, I
submitted a PEP to edi...@python.org. When things to have been worked,
I submitted the PEP again. I have not seen any activity on the PEP in
Python-Dev or any reply acknowledging that it was received. Did I
misunderstand the process of submitting a PEP?
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 3145

[Eric Pruitt]

Hello all,

I have been working on adding asynchronous I/O to the Python
subprocess module as part of my Google Summer of Code project. Now
that I have finished documenting and pruning the code, I present PEP
3145 for its inclusion into the Python core code. Any and all feedback
on the PEP (http://www.python.org/dev/peps/pep-3145/) is appreciated.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

Alright, I will re-submit with the contents pasted. I never use double
backquotes as I think them rather ugly; that is the work of an editor
or some automated program in the chain. Plus, it also messed up my
line formatting and now I have lines with one word on them... Anyway,
the contents of PEP 3145:

PEP: 3145
Title: Asynchronous I/O For subprocess.Popen
Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
Type: Standards Track
Content-Type: text/plain
Created: 04-Aug-2009
Python-Version: 3.2

Abstract:

In its present form, the subprocess.Popen implementation is prone to
dead-locking and blocking of the parent Python script while waiting on data
from the child process.

Motivation:

A search for "python asynchronous subprocess" will turn up numerous
accounts of people wanting to execute a child process and communicate with
it from time to time reading only the data that is available instead of
blocking to wait for the program to produce data [1] [2] [3].  The current
behavior of the subprocess module is that when a user sends or receives
data via the stdin, stderr and stdout file objects, dead locks are common
and documented [4] [5].  While communicate can be used to alleviate some of
the buffering issues, it will still cause the parent process to block while
attempting to read data when none is available to be read from the child
process.

Rationale:

There is a documented need for asynchronous, non-blocking functionality in
subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would improve the
utility of the Python standard library that can be used on Unix based and
Windows builds of Python.  Practically every I/O object in Python has a
file-like wrapper of some sort.  Sockets already act as such and for
strings there is StringIO.  Popen can be made to act like a file by simply
using the methods attached the the subprocess.Popen.stderr, stdout and
stdin file-like objects.  But when using the read and write methods of
those options, you do not have the benefit of asynchronous I/O.  In the
proposed solution the wrapper wraps the asynchronous methods to mimic a
file object.

Reference Implementation:

I have been maintaining a Google Code repository that contains all of my
changes including tests and documentation [9] as well as blog detailing
the problems I have come across in the development process [10].

I have been working on implementing non-blocking asynchronous I/O in the
subprocess.Popen module as well as a wrapper class for subprocess.Popen
that makes it so that an executed process can take the place of a file by
duplicating all of the methods and attributes that file objects have.

There are two base functions that have been added to the subprocess.Popen
class: Popen.send and Popen._recv, each with two separate implementations,
one for Windows and one for Unix based systems.  The Windows
implementation uses ctypes to access the functions needed to control pipes
in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,
the Python interface for file control serves the same purpose.  The
different implementations of Popen.send and Popen._recv have identical
arguments to make code that uses these functions work across multiple
platforms.

When calling the Popen._recv function, it requires the pipe name be
passed as an argument so there exists the Popen.recv function that passes
selects stdout as the pipe for Popen._recv by default.  Popen.recv_err
selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err"
are much easier to read and understand than "Popen._recv('stdout' ..." and
"Popen._recv('stderr' ..." respectively.

Since the Popen._recv function does not wait on data to be produced
before returning a value, it may return empty bytes.  Popen.asyncread
handles this issue by returning all data read over a given time
interval.

The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
allow a process to act like a file so that there are no blocking issues
that can arise from using the stdout and stdin file objects produced from
a subprocess.Popen call.


References:

[1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
http://mail.python.org/pipermail/python-bugs-list/2006-December/
  036524.html

[2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
http://ivory.idyll.org/blog/feb-07/problems-with-subprocess

[3] How can I run an external command asynchronously from Python? - Stack
Overflow
http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
  command-asynchronously-from-python

[4] 18.1. subprocess - Subprocess management - Python v2.6.

Re: [Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

I'm bumping this PEP again in hopes of getting some feedback.

Thanks,
Eric

On Tue, Sep 8, 2009 at 23:52, Eric Pruitt  wrote:
> PEP: 3145
> Title: Asynchronous I/O For subprocess.Popen
> Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
> Type: Standards Track
> Content-Type: text/plain
> Created: 04-Aug-2009
> Python-Version: 3.2
>
> Abstract:
>
>    In its present form, the subprocess.Popen implementation is prone to
>    dead-locking and blocking of the parent Python script while waiting on data
>    from the child process.
>
> Motivation:
>
>    A search for "python asynchronous subprocess" will turn up numerous
>    accounts of people wanting to execute a child process and communicate with
>    it from time to time reading only the data that is available instead of
>    blocking to wait for the program to produce data [1] [2] [3].  The current
>    behavior of the subprocess module is that when a user sends or receives
>    data via the stdin, stderr and stdout file objects, dead locks are common
>    and documented [4] [5].  While communicate can be used to alleviate some of
>    the buffering issues, it will still cause the parent process to block while
>    attempting to read data when none is available to be read from the child
>    process.
>
> Rationale:
>
>    There is a documented need for asynchronous, non-blocking functionality in
>    subprocess.Popen [6] [7] [2] [3].  Inclusion of the code would improve the
>    utility of the Python standard library that can be used on Unix based and
>    Windows builds of Python.  Practically every I/O object in Python has a
>    file-like wrapper of some sort.  Sockets already act as such and for
>    strings there is StringIO.  Popen can be made to act like a file by simply
>    using the methods attached the the subprocess.Popen.stderr, stdout and
>    stdin file-like objects.  But when using the read and write methods of
>    those options, you do not have the benefit of asynchronous I/O.  In the
>    proposed solution the wrapper wraps the asynchronous methods to mimic a
>    file object.
>
> Reference Implementation:
>
>    I have been maintaining a Google Code repository that contains all of my
>    changes including tests and documentation [9] as well as blog detailing
>    the problems I have come across in the development process [10].
>
>    I have been working on implementing non-blocking asynchronous I/O in the
>    subprocess.Popen module as well as a wrapper class for subprocess.Popen
>    that makes it so that an executed process can take the place of a file by
>    duplicating all of the methods and attributes that file objects have.
>
>    There are two base functions that have been added to the subprocess.Popen
>    class: Popen.send and Popen._recv, each with two separate implementations,
>    one for Windows and one for Unix based systems.  The Windows
>    implementation uses ctypes to access the functions needed to control pipes
>    in the kernel 32 DLL in an asynchronous manner.  On Unix based systems,
>    the Python interface for file control serves the same purpose.  The
>    different implementations of Popen.send and Popen._recv have identical
>    arguments to make code that uses these functions work across multiple
>    platforms.
>
>    When calling the Popen._recv function, it requires the pipe name be
>    passed as an argument so there exists the Popen.recv function that passes
>    selects stdout as the pipe for Popen._recv by default.  Popen.recv_err
>    selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err"
>    are much easier to read and understand than "Popen._recv('stdout' ..." and
>    "Popen._recv('stderr' ..." respectively.
>
>    Since the Popen._recv function does not wait on data to be produced
>    before returning a value, it may return empty bytes.  Popen.asyncread
>    handles this issue by returning all data read over a given time
>    interval.
>
>    The ProcessIOWrapper class uses the asyncread and asyncwrite functions to
>    allow a process to act like a file so that there are no blocking issues
>    that can arise from using the stdout and stdin file objects produced from
>    a subprocess.Popen call.
>
>
> References:
>
>    [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
>        http://mail.python.org/pipermail/python-bugs-list/2006-December/
>          036524.html
>
>    [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
>        http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
>
>    [3] How can I run an external command asynchronously from Python? - Stack
>    

Re: [Python-Dev] PEP 3145 (With Contents)

[Eric Pruitt]

Hey,

Anatoly, you are free to modify the PEP and code. I do not have any
plans to work on this right now.

Eric

On Mon, Dec 24, 2012 at 04:42:20PM +0300, anatoly techtonik wrote:
> What should I do in case Eric lost interest after his GSoC project for PSF
> appeared as useless for python-dev community? Should I rewrite the proposal
> from scratch?
> 
> On Thu, Dec 20, 2012 at 11:18 PM, Brett Cannon  wrote:
> 
> > You cannot rewrite an existing PEP if you are not one of the original
> > owners, nor can you add yourself as an author to a PEP without permission
> > from the original authors.
> >
> > And please do not CC the peps mailing list on discussions. It should only
> > be used to mail in new PEPs or acceptable patches to PEPs.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com