[issue33780] [subprocess] Better Unicode support for shell=True on Windows
New submission from Yoni Rozenshein :
In subprocess, the implementation of shell=True on Windows is to launch a
subprocess with using {comspec} /c "{args}" (normally comspec=cmd.exe).
By default, the output of cmd is encoded with the "active" codepage. In Python
3.6, you can decode this using encoding='oem'.
However, this actually loses information. For example, try creating a file with
a filename in a language that is not your active codepage, and then doing
subprocess.check_output('dir', shell=True). In the output, the filename is
replaced with question marks (not by Python, by cmd!).
To get the correct output, cmd has a "/u" switch (this switch has probably
existed forever - at least since Windows NT 4.0, by my internet search). The
output can then be decoded using encoding='utf-16-le', like any native Windows
string.
Currently, Popen constructs the command line in this hardcoded format:
{comspec} /c "{args}", so you can't get the /u in there with the shell=True
shortcut, and have to write your own wrapping code.
I suggest adding an feature to Popen where /u may be inserted before the /c
within the shell=True shortcut. I've thought of several ways to implement this:
1. A new argument to Popen, which indicates that we want Unicode shell output;
if True, add the /u. Note that we already have a couple of Windows-only
arguments to Popen, so this would not be a precedent.
2. If the encoding argument is 'utf-16-le' or one of its aliases, then add the
/u.
3. If the encoding argument is not None, then add the /u.
--
components: Library (Lib)
messages: 318807
nosy: Yoni Rozenshein
priority: normal
severity: normal
status: open
title: [subprocess] Better Unicode support for shell=True on Windows
type: enhancement
versions: Python 3.6, Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue33780>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33780] [subprocess] Better Unicode support for shell=True on Windows
Yoni Rozenshein added the comment: After reading your messages and especially after reading https://bugs.python.org/issue27179#msg267091 I admit I have been convinced this is much more complicated than I thought, and maybe more of a Windows bug than a Python bug :) -- ___ Python tracker <https://bugs.python.org/issue33780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33884] [multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest
New submission from Yoni Rozenshein : multiprocessing will attempt to pickle things using ForkingPickler when starting a new process in spawn mode (in Windows this is the only mode, in Linux this is a non-default but settable mode). When run within the context of a unit test, if it has to pickle a TestCase subclass, it encounters objects that can't be pickled. The attached file shows a minimum working example (uncomment the two commented lines under __main__ to run with pytest). When run with unittest.main(), it raises: TypeError: cannot serialize '_io.TextIOWrapper' object When run with pytest.main(), it raises: AttributeError: Can't pickle local object 'ArgumentParser.__init__..identity' Note that changing the _child_process in my example to a classmethod/staticmethod or moving it to a top-level function outside the class works around this issue (both with unittest and with pytest). -- components: Library (Lib) files: mp_pickle_issues.py messages: 319811 nosy: Yoni Rozenshein priority: normal severity: normal status: open title: [multiprocessing] Multiprocessing in spawn mode doesn't work when the target is a method in a unittest.TestCase subclass, when run either with unittest or with pytest type: behavior versions: Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file47645/mp_pickle_issues.py ___ Python tracker <https://bugs.python.org/issue33884> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
