[issue39694] Incorrect dictionary unpacking when calling str.format
New submission from Akos Kiss :
My understanding was that in function calls, the keys in an **expression had to
be strings. However, str.format seems to deviate from that and allows
non-string keys in the mapping (and silently ignores them).
Please, see the transcript below:
>>> def f(): pass
...
>>> def g(): pass
...
>>> x = {None: ''}
>>> y = {1: ''}
>>> f(**x)
Traceback (most recent call last):
File "", line 1, in
TypeError: f() keywords must be strings
>>> f(**y)
Traceback (most recent call last):
File "", line 1, in
TypeError: f() keywords must be strings
>>> g(**x)
Traceback (most recent call last):
File "", line 1, in
TypeError: g() keywords must be strings
>>> g(**y)
Traceback (most recent call last):
File "", line 1, in
TypeError: g() keywords must be strings
>>> ''.format(**x)
''
>>> ''.format(**y)
''
I could reproduce this (incorrect?) behavior on macOS with python 3.4-3.7 and
on Ubuntu 18.04 with python 3.6.
--
messages: 362304
nosy: Akos Kiss
priority: normal
severity: normal
status: open
title: Incorrect dictionary unpacking when calling str.format
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7
___
Python tracker
<https://bugs.python.org/issue39694>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39694] Incorrect dictionary unpacking when calling str.format
Akos Kiss added the comment: Re: Eric I had a code where `'sometemplate'.format()` got a dictionary from outside (as a parameter, not from `locals()`), and that dictionary had `None` as a key. Actually, I wasn't even aware of that `None` key until I tried to execute the same code in PyPy where it failed with a `TypeError`. That's when I started to dig down to the root of the incompatibility. Now, my code has a workaround to have an `if key is not None` filter in the comprehension that constructs the dict. (I'm not sure whether it can be called a workaround, since this is how it should have been written in the first place. It turns out I was just (ab)using an implementation detail / deviation.) So much about real-world use cases. There is one more use case that comes to my mind, but it is admittedly theoretical (for now, at least). If I ever wanted to patch/wrap/mock `str.format` then the wrapped version would behave differently from the built-in version (throw an error when the original does not). But this is hypothetical at the moment because I "can't set attributes of built-in/extension type 'str'" and don't have the time to experiment with complex mocking/patching/wrapping approaches. -- ___ Python tracker <https://bugs.python.org/issue39694> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39694] Incorrect dictionary unpacking when calling str.format
Akos Kiss added the comment: I couldn't find any discussion in the language reference about fundamental differences between calls to built-in functions and user-defined functions. (I tried to read Sections 6.3.4. "Calls" and 8.6. "Function definitions" carefully.) Until now I had the impression that built-in and user-defined functions should be as similar as possible. BTW, a user-defined function can also implement support for unused/excess keyword arguments (or arguments used on-demand): it only has to add **kwargs at the end of its signature. And that's exactly how the documentation specifies the signature of format: str.format(*args, **kwargs). That API documentation signals (to me at least) that this function should be called just like any other function that has a (*args, **kwargs) signature, be it built-in of user-defined. -- ___ Python tracker <https://bugs.python.org/issue39694> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39694] Incorrect dictionary unpacking when calling str.format
Akos Kiss added the comment:
I've come up with some extra examples (use cases?):
```py
import collections
class str2(str):
def format(self, *args, **kwargs):
return super().format(*args, **kwargs)
def unpacknonekey(s):
print('input:', type(s), s)
try:
print('str key:', s.format(**{'bar': 'qux'}))
print('none key:', s.format(**{'bar': 'qux', None: ''}))
except TypeError as e:
print('error:', e)
template = 'foo {bar} baz'
unpacknonekey(template)
unpacknonekey(str2(template))
unpacknonekey(collections.UserString(template))
```
The above script gives the following output:
```
input: foo {bar} baz
str key: foo qux baz
none key: foo qux baz
input: foo {bar} baz
str key: foo qux baz
error: format() keywords must be strings
input: foo {bar} baz
str key: foo qux baz
error: format() keywords must be strings
```
This shows inconsistency between `format` of `str` and subclasses of `str` or
the standard library's `UserString`.
--
___
Python tracker
<https://bugs.python.org/issue39694>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31863] Inconsistent returncode/exitcode for terminated child processes on Windows
New submission from Akos Kiss :
I've been working with various approaches for running and terminating
subprocesses on Windows and I've obtained surprisingly different results if I
used different modules and ways of termination. Here is the script I wrote, it
uses the `subprocess` and the `multiprocessing` modules for starting new
subprocesses, and process termination is performed either by the modules' own
`terminate` functions or by `os.kill`.
```py
import multiprocessing
import os
import signal
import subprocess
import sys
import time
def kill_with_os_kill(proc):
print('kill with os.kill(pid,SIGTERM)')
os.kill(proc.pid, signal.SIGTERM)
def kill_with_terminate(proc):
print('kill child with proc.terminate()')
proc.terminate()
def run_and_kill_subprocess(killfn, procarg):
print('run subprocess child with %s' % procarg)
with subprocess.Popen([sys.executable, __file__, procarg]) as proc:
time.sleep(1)
killfn(proc)
proc.wait()
print('child terminated with %s' % proc.returncode)
def run_and_kill_multiprocessing(killfn, procarg):
print('run multiprocessing child with %s' % procarg)
proc = multiprocessing.Process(target=childmain, args=(procarg,))
proc.start()
time.sleep(1)
killfn(proc)
proc.join()
print('child terminated with %s' % proc.exitcode)
def childmain(arg):
print('child process started with %s' % arg)
while True:
pass
if __name__ == '__main__':
if len(sys.argv) < 2:
print('parent process started')
run_and_kill_subprocess(kill_with_os_kill, 'subprocess-oskill')
run_and_kill_subprocess(kill_with_terminate, 'subprocess-terminate')
run_and_kill_multiprocessing(kill_with_os_kill,
'multiprocessing-oskill')
run_and_kill_multiprocessing(kill_with_terminate,
'multiprocessing-terminate')
else:
childmain(sys.argv[1])
```
On macOS, everything works as expected (and I think that Linux will behave
alike):
```
$ python3 killtest.py
parent process started
run subprocess child with subprocess-oskill
child process started with subprocess-oskill
kill with os.kill(pid,SIGTERM)
child terminated with -15
run subprocess child with subprocess-terminate
child process started with subprocess-terminate
kill child with proc.terminate()
child terminated with -15
run multiprocessing child with multiprocessing-oskill
child process started with multiprocessing-oskill
kill with os.kill(pid,SIGTERM)
child terminated with -15
run multiprocessing child with multiprocessing-terminate
child process started with multiprocessing-terminate
kill child with proc.terminate()
child terminated with -15
```
But on Windows, I got:
```
>py -3 killtest.py
parent process started
run subprocess child with subprocess-oskill
child process started with subprocess-oskill
kill with os.kill(pid,SIGTERM)
child terminated with 15
run subprocess child with subprocess-terminate
child process started with subprocess-terminate
kill child with proc.terminate()
child terminated with 1
run multiprocessing child with multiprocessing-oskill
child process started with multiprocessing-oskill
kill with os.kill(pid,SIGTERM)
child terminated with 15
run multiprocessing child with multiprocessing-terminate
child process started with multiprocessing-terminate
kill child with proc.terminate()
child terminated with -15
```
Notes:
- On Windows with `os.kill(pid, sig)`, "sig will cause the process to be
unconditionally killed by the TerminateProcess API, and the exit code will be
set to sig." I.e., it is not possible to detect on Windows whether a process
was terminated by a signal or it exited properly, because `kill` does not
actually raise a signal and no Windows API allows to differentiate between
proper or forced termination.
- The `multiprocessing` module has a workaround for this by terminating the
process with a designated exit code (`TERMINATE = 0x1`) and checking for
that value afterwards, rewriting it to `-SIGTERM` if found. The related
documentation is a bit misleading, as `exitcode` is meant to have "negative
value -N [which] indicates that the child was terminated by signal N" --
however, if the process was indeed killed with `SIGTERM` (and not via
`terminate`), then `exitcode` will be `SIGTERM` and not `-SIGTERM` (see above).
(The documentation of `terminate` does not clarify the situation much by
stating that "on Windows TerminateProcess() is used", since it does not mention
the special exit code -- and well, it's not even a signal after all, so it's
not obvious whether negative or positive exit code is to be expected.)
- The `subprocess` module choses the quite arbitrary exit code of 1 and
documents that "negative value -N indicates that the child was terminated by
signal N&
[issue31863] Inconsistent returncode/exitcode for terminated child processes on Windows
Akos Kiss added the comment: `taskkill /F` sets exit code to 1, indeed. (Confirmed by experiment. Cannot find this behaviour documented, though.) On the other hand, MS Docs state (https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/signal#remarks) that termination by a signal "terminates the calling program with exit code 3". (So, there may be other "valid" exit codes, too.) -- ___ Python tracker <https://bugs.python.org/issue31863> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31863] Inconsistent returncode/exitcode for terminated child processes on Windows
Akos Kiss added the comment:
A follow-up: in addition to `taskkill`, I've taken a look at another "official"
way for killing processes, the `Stop-Process` PowerShell cmdlet
(https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.management/stop-process?view=powershell-5.1).
Yet again, documentation is scarce on what the exit code of the terminated
process will be. But PowerShell and .NET code base is open sourced, so I've dug
a bit deeper and found that `Stop-Process` is based on
`System.Diagnostics.Process.Kill()`
(https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs#L1240),
while `Process.Kill()` uses the `TerminateProcess` Win32 API
(https://github.com/dotnet/corefx/blob/master/src/System.Diagnostics.Process/src/System/Diagnostics/Process.Windows.cs#L93).
Interestingly, `TerminateProcess` is called with -1 (this was surprising, to
me at least, as exit code is unsigned on Windows AFAIK).
Therefore, I've added two new "kill" implementations to my original code
experiment (wont repeat the whole code here, just the additions):
```py
def kill_with_taskkill(proc):
print('kill child with taskkill /F')
subprocess.run(['taskkill', '/F', '/pid', '%s' % proc.pid], check=True)
def kill_with_stopprocess(proc):
print('kill child with powershell stop-process')
subprocess.run(['powershell', 'stop-process', '%s' % proc.pid], check=True)
```
And I got:
```
run subprocess child with subprocess-taskkill
child process started with subprocess-taskkill
kill child with taskkill /F
SUCCESS: The process with PID 4024 has been terminated.
child terminated with 1
run subprocess child with subprocess-stopprocess
child process started with subprocess-stopprocess
kill child with powershell stop-process
child terminated with 4294967295
run multiprocessing child with multiprocessing-taskkill
child process started with multiprocessing-taskkill
kill child with taskkill /F
SUCCESS: The process with PID 5988 has been terminated.
child terminated with 1
run multiprocessing child with multiprocessing-stopprocess
child process started with multiprocessing-stopprocess
kill child with powershell stop-process
child terminated with 4294967295
```
My takeaways from the above are that
1) Windows is not consistent across itself,
2) 1 is not the only "valid" "terminated forcibly" exit code, and
3) negative exit code does not work, even if MS itself tries to use it.
BTW, I really think that killing a process with a code of 1 is questionable, as
quite some apps return 1 themselves just to signal error (but proper
termination). This makes it hard to tell applications' own error signaling and
forced kills apart. But that's a personal opinion.
--
___
Python tracker
<https://bugs.python.org/issue31863>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31863] Inconsistent returncode/exitcode for terminated child processes on Windows
Akos Kiss added the comment: And I thought that my analysis was thorough... Exit code 1 is the way to go, I agree now. -- ___ Python tracker <https://bugs.python.org/issue31863> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
