[issue44709] [3.7] Popen Control Characters in stdout affect shell session
New submission from San :
I was trying to wrap an old game server executable with a built-in console
using Popen.
class ServerWrapper(Thread):
def __init__(self, pathToExecutable: str, statusMonitor: Popen = None,
active: bool = False):
super().__init__()
self.pathToExecutable = pathToExecutable
self.statusMonitor = statusMonitor
self.active = active
def run(self):
self.statusMonitor = Popen(['./bf1942_lnxded', "+statusMonitor", '1'],
encoding='latin_1', stdout=PIPE,
stdin=PIPE, cwd=self.pathToExecutable)
while self.active:
currentLine = self.statusMonitor.stdout.readline()
if currentLine:
print(currentLine)
input('')
def start(self):
self.active = True
super().start()
def stop(self):
self.active = False
I expected to be able to read the output line-by-line using enter, in a 'normal
fashion'.
After importing this from a terminal and setting it up somewhat like so:
wrapper = ServerWrapper('bf1942')
wrapper.start()
However, once the server process started and thereby started writing to stdout,
weird stuff started to happen to my shell with the python interpreter.
Apparently, there are control characters and ansi-escape codes sent from the
process, that somehow manage to manipulate my shell if I specify an encoding.
Consequentially the lines output with 'print(currentLine)' are reduced by these
chars.
Is this the desired behaviour of the decoder? If so then I think this should
potentially be further clarified in the documentation of Popen. I would have
expected the chars to be escaped because the worst thing is, this happens even
if you don't actually read from stdout at all. Seemingly the decoder executes
incoming control sequences immediately (probably for the buffer?), regardless
of if you actually bother to read the stdout or not.
The paragraph in
https://docs.python.org/3.7/library/subprocess.html#security-considerations
sounds like this shouldn't be happening if 'shell' is not set to 'True' at
least.
--
files: shell.png
messages: 397986
nosy: San
priority: normal
severity: normal
status: open
title: [3.7] Popen Control Characters in stdout affect shell session
versions: Python 3.7
Added file: https://bugs.python.org/file50170/shell.png
___
Python tracker
<https://bugs.python.org/issue44709>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44709] [3.7] Popen Control Characters in stdout affect shell session
San added the comment:
I get your point. But I thought setting 'stdout=PIPE' should not send the
stdout directly to the terminal but rather through the pipe first?!
Also what you're saying seems to contradict that this doesn't happen if I don't
set the encoding and let it be a byte-stream. In that case you get the full
byte-stream without any control-characters being executed.
In my opinion it is unintuitive that this is happening.
For instance if I open a file using io, that contains the same control
characters like so:
f = open('cc.txt', 'r', encoding='latin_1')
Then gladly this doesn't happen and the control chars get escaped using \x1b
which is what I would also expect to happen when using Popen.
--
___
Python tracker
<https://bugs.python.org/issue44709>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44709] [3.7] Popen Control Characters in stdout affect shell session
San added the comment: I just realised that this is not a problem of python but rather a problem of the shell that I was using. I was testing the io in a different environment. Now I retried in the same environment that the wrapper was operating and the behaviour is the same using io. Sorry for the confusion and thanks for nudging me the right direction. -- ___ Python tracker <https://bugs.python.org/issue44709> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40034] cgi.parse() does not work with multipart POST requests.
New submission from San :
The cgi.parse stdlib function works in most cases but never works when given a
multipart/form-data POST request because it does not set up pdict in a way
cgi.parse_multipart() likes (boundary as bytes (not str) and including content
length).
$ pwd
/tmp
$
$ /tmp/cpython/python --version
Python 3.9.0a4+
$
$ cat cgi-bin/example.cgi
#!/tmp/cpython/python
import sys, cgi
query_dict = cgi.parse()
write = sys.stdout.buffer.write
write("Content-Type: text/plain; charset=utf-8\r\n\r\n".encode("ascii"))
write(f"Worked, query dict is {query_dict}.\n".encode())
$
$ /tmp/cpython/python -m http.server --cgi & sleep 1
[1] 30201
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...
$
$ # GET (url-encoded) requests work:
$ curl localhost:8000/cgi-bin/example.cgi?example_key=example_value
127.0.0.1 - - [20/Mar/2020 23:33:48] "GET
/cgi-bin/example.cgi?example_key=example_value HTTP/1.1" 200 -
Worked, query dict is {'example_key': ['example_value']}.
$
$ # POST (multipart) requests do not:
$ curl localhost:8000/cgi-bin/example.cgi -F example_key=example_value
127.0.0.1 - - [20/Mar/2020 23:34:15] "POST /cgi-bin/example.cgi HTTP/1.1" 200 -
Traceback (most recent call last):
File "/tmp/cgi-bin/example.cgi", line 3, in
query_dict = cgi.parse()
File "/tmp/cpython/Lib/cgi.py", line 159, in parse
return parse_multipart(fp, pdict)
File "/tmp/cpython/Lib/cgi.py", line 201, in parse_multipart
boundary = pdict['boundary'].decode('ascii')
AttributeError: 'str' object has no attribute 'decode'
127.0.0.1 - - [20/Mar/2020 23:34:16] CGI script exit status 0x100
$
$ $EDITOR /tmp/cpython/Lib/cgi.py
$
$ # After changing cgi.parse POST (multipart) requests work:
$ curl localhost:8000/cgi-bin/example.cgi -F example_key=example_value
127.0.0.1 - - [20/Mar/2020 23:35:10] "POST /cgi-bin/example.cgi HTTP/1.1" 200 -
Worked, query dict is {'example_key': ['example_value']}.
$
--
components: Library (Lib)
messages: 364762
nosy: sangh
priority: normal
severity: normal
status: open
title: cgi.parse() does not work with multipart POST requests.
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue40034>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40034] cgi.parse() does not work with multipart POST requests.
Change by San : -- keywords: +patch pull_requests: +18492 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19130 ___ Python tracker <https://bugs.python.org/issue40034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14204] Support for the NPN extension to TLS/SSL
Changes by Sidney San Martín : -- nosy: +ssm ___ Python tracker <http://bugs.python.org/issue14204> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14652] Better error messages for wsgiref validator failures
New submission from Sidney San Martín : wsgiref’s validation middleware is missing messages for many of its assertions, and its docstring doesn’t reflect read() now requiring an argument (said that it took an optional argument). Here’s a patch to add some and update the comment. -- components: Library (Lib) files: ssm_validate.patch keywords: patch messages: 159053 nosy: ssm priority: normal severity: normal status: open title: Better error messages for wsgiref validator failures type: enhancement versions: Python 3.3 Added file: http://bugs.python.org/file25320/ssm_validate.patch ___ Python tracker <http://bugs.python.org/issue14652> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14835] plistlib: output empty elements correctly
New submission from Sidney San Martín : plistlib’s output is currently byte-for-byte identical to Apple’s, except arrays and dictionaries are stored with self-closing tags (e.g. '') and plistlib outputs empty tags with a newline (e.g. '\n'). This tiny patch makes its output for empty arrays and dictionaries consistent with the OS’s. -- assignee: ronaldoussoren components: Library (Lib), Macintosh files: plistlib_empty_objects.patch keywords: patch messages: 160934 nosy: ronaldoussoren, ssm priority: normal severity: normal status: open title: plistlib: output empty elements correctly type: behavior versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4 Added file: http://bugs.python.org/file25620/plistlib_empty_objects.patch ___ Python tracker <http://bugs.python.org/issue14835> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14652] Better error messages for wsgiref validator failures
Sidney San Martín added the comment: Thanks Jeff, I’m actually a relatively new Python developer and got the impression that it was best practice to always use a tuple for string formatting, for consistency. Here’s an updated patch which drops the tuples for those cases. -- Added file: http://bugs.python.org/file25624/ssm_validate.patch ___ Python tracker <http://bugs.python.org/issue14652> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14835] plistlib: output empty elements correctly
Sidney San Martín added the comment: Hynek: Here you go! Ronald: Emailed it in on Friday. -- Added file: http://bugs.python.org/file25734/plistlib_empty_element_test.patch ___ Python tracker <http://bugs.python.org/issue14835> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14835] plistlib: output empty elements correctly
Sidney San Martín added the comment: Thanks Hynek, awesome! It looks like it’s in now. -- ___ Python tracker <http://bugs.python.org/issue14835> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
