[issue13648] xml.sax.saxutils.escape does not escapes \x00
New submission from Alexey :
function xml.sax.saxutils.escape('\x00qweqwe<') returns '\x00qweqwe<'
\x00 did not escaped to �
is this is a correct behavior?
this is influences tools like xmpppy, which sends \x00 not encoded and leads to
xmpp error.
--
messages: 150096
nosy: Animus
priority: normal
severity: normal
status: open
title: xml.sax.saxutils.escape does not escapes \x00
type: behavior
versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2
___
Python tracker
<http://bugs.python.org/issue13648>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13648] xml.sax.saxutils.escape does not escapes \x00
Changes by Alexey : -- components: +XML ___ Python tracker <http://bugs.python.org/issue13648> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13648] xml.sax.saxutils.escape does not escapes \x00
Alexey added the comment: sorry, xmpppy uses it's own escape method, but anyway... :) -- ___ Python tracker <http://bugs.python.org/issue13648> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45398] Bugs in set operations in 3.8.10 (detected in Win7)
Alexey added the comment: Is interesting, that in win10 the bug is not reproduced too ! (only in win7!!!) -- ___ Python tracker <https://bugs.python.org/issue45398> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45398] Bugs in set operations in 3.8.10 (detected in Win7)
Change by Alexey : -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue45398> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41360] method _tunnel does not allow for a standard proxy authentication solution
New submission from Alexey : The _tunnel method of the HTTPConnection class in the http.client module, if the CONNECT request is unsuccessful, closes the connection and raises an OSError without returning the response data. This behavior does not allow to implement the authentication procedure on the proxy server using methods similar to those used for authentication on servers (using hooks). And at the moment proxy authentication (Kerberos, Digest, NTLM - all other than Basic) is not supported by the urllib3 and accordingly requests, pip and many others. As a result, a large number of people cannot use Python to create scripts that involve working on the Internet (if they don't know workarounds). This problem has been mentioned many times here (Issue 7291, 24333, 24964). There are many Issues related to this task in requests, urllib3, pip and other. This problem is many years old (at least 5), but there is still no complete solution (as far as I can tell). There are several workarounds, but there is still no so lution that could be used in urllib3, requests, pip and other (in many discussions of Issues _tunnel method is indicated as a reason preventing a proxy authentication solution from being implemented). Hopefully someone can finally solve this problem or explain why it can't be solved -- components: Library (Lib) messages: 374067 nosy: Namyotkin priority: normal severity: normal status: open title: method _tunnel does not allow for a standard proxy authentication solution type: enhancement versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue41360> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24964] Add tunnel CONNECT response headers to httplib / http.client
Change by Alexey : -- nosy: +Namyotkin ___ Python tracker <https://bugs.python.org/issue24964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13648] xml.sax.saxutils.escape does not escapes \x00
Alexey added the comment:
>This is correct behavior. \x00 is not supported in XML:
> not in raw form, and not in escaped form
last sentence in forth paragraph of section 1.3 in XML 1.1 specification says
following:
==
Due to potential problems with APIs,
#x0 is still forbidden both directly and as a character reference.
==
And, second sentence in paragraph 2 in subsection 'Validity constraint:
Notation Declared' of section 4.2.2 says following:
==
The characters to be escaped are the control characters #x0 to #x1F and #x7F
(most of which cannot appear in XML), space #x20, the delimiters '<' #x3C, '>'
#x3E and '"' #x22, the unwise characters '{' #x7B, '}' #x7D, '|' #x7C, '\'
#x5C, '^' #x5E and '`' #x60, as well as all characters above #x7F.
==
(xml 1.1) http://www.w3.org/TR/2006/REC-xml11-20060816/
(xml 1.0) http://www.w3.org/TR/2008/REC-xml-20081126/
--
status: closed -> open
___
Python tracker
<http://bugs.python.org/issue13648>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13648] xml.sax.saxutils.escape does not escapes \x00
Alexey added the comment: What am I trying to say is: if those characters are forbidden, then maybe they need to be escaped rather than ignored? -- ___ Python tracker <http://bugs.python.org/issue13648> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21430] Document ssl.pending()
Alexey added the comment: Please document this method. I'm developing xmpp client using python 3.4 and it's ssl module. Stuck with this same issue for a few days just because of a lack of documentation. I'm asking you: why should I know about this method not from python documentation, but from search engine and perl's forum (http://www.perlmonks.org/?node_id=845342) ? -- nosy: +Animus ___ Python tracker <http://bugs.python.org/issue21430> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23520] test_os failed (python-3.4.3)
New submission from Alexey: Just one test failed for python-3.4.3 ... test_invalid_offset (test.test_os.TestSendfile) ... FAIL ... -- files: test_os_log messages: 236574 nosy: avoevodkin priority: normal severity: normal status: open title: test_os failed (python-3.4.3) versions: Python 3.4 Added file: http://bugs.python.org/file38232/test_os_log ___ Python tracker <http://bugs.python.org/issue23520> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46114] OpenSSL deprecated OpenSSL_version_num() since version 3.0.0
New submission from Alexey Stepanov : Documentation: https://www.openssl.org/docs/manmaster/man3/OpenSSL_version_num.html Python build fail with OpenSSL 3.0.1 (looks like OpenSSL broken deprecated API) -- assignee: christian.heimes components: SSL messages: 408770 nosy: christian.heimes, penguinolog priority: normal severity: normal status: open title: OpenSSL deprecated OpenSSL_version_num() since version 3.0.0 versions: Python 3.10, Python 3.11, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue46114> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46114] OpenSSL deprecated OpenSSL_version_num() since version 3.0.0
Alexey Stepanov added the comment:
Error example:
test_ssl: testing with 'OpenSSL 3.0.1 14 Dec 2021' (3, 0, 0, 1, 0)
...
AssertionError: False is not true : ('OpenSSL 3.0.1 14 Dec 2021', (3, 0, 0, 1,
0), '0x3010')
--
___
Python tracker
<https://bugs.python.org/issue46114>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12970] os.walk() consider some symlinks as dirs instead of non-dirs
Changes by Alexey Smirnov : -- nosy: +alexey-smirnov ___ Python tracker <http://bugs.python.org/issue12970> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12971] os.isdir() should contain skiplinks=False in arguments
Changes by Alexey Smirnov : -- nosy: +alexey-smirnov ___ Python tracker <http://bugs.python.org/issue12971> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13275] Recommend xml.etree for XML processing
New submission from Alexey Shamrin : There are many libraries in Python stdlib [1] xml.dom xml.dom.minidom xml.dom.pulldom xml.sax xml.etree Tutorial mentions xml.dom and xml.sax. [2] In my experience xml.etree is the best library to quickly load some xml and mess with it. It would be nice to recommend people xml.etree in the tutorial [2] and in the beginning of section 19, "Structured Markup Processing Tools". [1] [1]: http://docs.python.org/library/markup.html [2]: http://docs.python.org/tutorial/stdlib.html#batteries-included -- assignee: docs@python components: Documentation messages: 146484 nosy: ash, docs@python priority: normal severity: normal status: open title: Recommend xml.etree for XML processing ___ Python tracker <http://bugs.python.org/issue13275> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12251] subprocess(..., stdout=sys.stderr) closes stderr for child
Changes by Alexey Smirnov : -- nosy: +alexey-smirnov ___ Python tracker <http://bugs.python.org/issue12251> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12105] open() does not able to set flags, such as O_CLOEXEC
Alexey Smirnov added the comment: FreeBSD 8+ also supports O_CLOEXEC in open(). -- ___ Python tracker <http://bugs.python.org/issue12105> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9626] OderedDict.viewitems() does not preserve item order
New submission from Alexey Luchko :
OrderedDict.viewitems() is expected to preserve item order like items() do
>>> from collections import OrderedDict
>>> d = OrderedDict([(1, 2), ("a", "b")])
>>> d.items()
[(1, 2), ('a', 'b')]
But it does not:
>>> list(d.viewitems())
[('a', 'b'), (1, 2)]
--
components: Library (Lib)
messages: 114117
nosy: luch
priority: normal
severity: normal
status: open
title: OderedDict.viewitems() does not preserve item order
type: behavior
versions: Python 2.7
___
Python tracker
<http://bugs.python.org/issue9626>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10346] strange arithmetic behaviour
New submission from Alexey Radkov : The following excerpt will show the issue: $ python Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> 8 * 4 / ( 2 - 7 ) * 6 / 3 -14 >>> Why it is evaluated to -14 ?? In floating point arithmetic it should be -12.8, in integer arithmetic i believe it should be -12 (at least bc and a small dedicated C program evaluate it to -12). Perhaps i do not understand some specific python arithmetic priority or associativity rules, anyway i cannot find a specific combinations of them to yield -14 in this expression. -- components: None messages: 120665 nosy: alexey.radkov priority: normal severity: normal status: open title: strange arithmetic behaviour type: behavior versions: Python 2.7 ___ Python tracker <http://bugs.python.org/issue10346> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12105] open() does not able to set flags, such as O_CLOEXEC
Changes by Alexey Smirnov : -- nosy: +alexey-smirnov ___ Python tracker <http://bugs.python.org/issue12105> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11098] syntax error at end of line in interactive python -u
New submission from Alexey Luchko : Interactive python -u produces syntax error in win2k: > python -u Python 2.7.1 (r271:86832, Nov 27 2010, 18:30:46) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> 1+1 File "", line 1 1+1 ^ SyntaxError: invalid syntax >>> import sys File "", line 1 import sys ^ SyntaxError: invalid syntax >>> Tried Python 2.5.4, 2.7.1 and 3.1.2 on the same win2k machine. The problem does not exist under mac os x 10.5 and freebsd 8.1. -- components: Windows messages: 127732 nosy: luch priority: normal severity: normal status: open title: syntax error at end of line in interactive python -u versions: Python 2.5, Python 2.7, Python 3.1 ___ Python tracker <http://bugs.python.org/issue11098> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11098] syntax error at end of line in interactive python -u
Alexey Luchko added the comment: I reported the issue just because I didn't find it is already known. I don't think it is worth backporting. -- ___ Python tracker <http://bugs.python.org/issue11098> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2931] optparse: various problems with unicode and gettext
New submission from Alexey Shamrin <[EMAIL PROTECTED]>: In the process of trying to use optparse with russian messages, I found several problems with gettext and unicode handling: 1. optparse.OptionParser.error function doesn't work with unicode argument 2. optparse doesn't work when its error messages are gettext-translated 3. optparse fails running 'prog.py --help > out.txt' with unicode help (at least on my system: Windows XP, Russian) I have attached a file demonstrating these problems: test_optparse.py. You can run it either using nose[1] or directly, manually uncommenting test_* functions one-by-one. [1]: http://www.somethingaboutorange.com/mrl/projects/nose/ Here's the result of running `nosetests test_optparse.py`: EEF == ERROR: OptionParser.error function doesn't work with unicode argument -- Traceback (most recent call last): File "c:\python25\lib\site-packages\nose-0.10.2-py2.5.egg\nose\case.py", line 182, in runTest self.test(*self.arg) File "C:\work\test_optparse.py", line 10, in test_unicode_error optparse.OptionParser().error(russian_text) File "C:\Python25\lib\optparse.py", line 1562, in error self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg)) File "C:\Python25\lib\optparse.py", line 1551, in exit sys.stderr.write(msg) UnicodeEncodeError: 'ascii' codec can't encode characters in position 28-34: ordinal not in range(128) == ERROR: optparse doesn't work when its error messages are gettext-translated -- Traceback (most recent call last): File "c:\python25\lib\site-packages\nose-0.10.2-py2.5.egg\nose\case.py", line 182, in runTest self.test(*self.arg) File "C:\work\test_optparse.py", line 25, in test_translated_unicode_error_message optparse.OptionParser().parse_args(["--unknown"]) File "C:\Python25\lib\optparse.py", line 1380, in parse_args self.error(str(err)) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-6: ordinal not in range(128) == FAIL: optparse fails running 'prog.py --help > out.txt' with unicode help -- Traceback (most recent call last): File "c:\python25\lib\site-packages\nose-0.10.2-py2.5.egg\nose\case.py", line 182, in runTest self.test(*self.arg) File "C:\work\test_optparse.py", line 42, in test_redirected_unicode_help assert '?' not in dummy_stdout.getvalue() AssertionError -- Ran 3 tests in 0.000s FAILED (errors=2, failures=1) -- components: Library (Lib) files: test_optparse.py messages: 67129 nosy: ash, gward severity: normal status: open title: optparse: various problems with unicode and gettext Added file: http://bugs.python.org/file10386/test_optparse.py __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2931> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2931] optparse: various problems with unicode and gettext
Alexey Shamrin <[EMAIL PROTECTED]> added the comment: I've also attached a patch that fixes all these issues and also allows the word "error" to be translated with gettext. Regarding the use of `locale.getpreferredencoding` instead of `sys.getdefaultencoding`. On my system (Windows XP, Russian) I get: Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import sys, locale >>> sys.getdefaultencoding() 'ascii' >>> locale.getpreferredencoding() 'cp1251' Using cp1251 on my system makes much more sense. It's used as a default encoding everywhere in the system. For example, in Notepad. -- keywords: +patch Added file: http://bugs.python.org/file10387/optparse.py.patch __ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2931> __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2931] optparse: various problems with unicode and gettext
Alexey Shamrin <[EMAIL PROTECTED]> added the comment: sampablokuper, I don't think your problem is relevant to this issue. In addition to encoding declaration you should use unicode strings: u"your non-ASCII text". Or wait for Python 3.0, where strings will be unicode by default. ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue2931> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue1734346] patch for bug 1170311 "zipfile UnicodeDecodeError"
Alexey Borzenkov <[EMAIL PROTECTED]> added the comment: Martin, I cleared the flag bit because filename was changed in-place, to mark that filename does not need further processing. This was primarily compatibility concern, to accommodate for situations where users try to do such decoding in their own code (this way flag won't be there, so their code won't trigger). Without clearing the flag bit, calling _decodeFilenameFlags second time will fail, as well as any similar user code. I suggest that if users want to know if filename is unicode, they should check that filename is of class unicode. _ Tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue1734346> _ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4494] Python 2.6 fails to build with Py_NO_ENABLE_SHARED
New submission from Alexey Borzenkov <[EMAIL PROTECTED]>: When building python 2.6 with Py_NO_ENABLE_SHARED compilation fails on PC/getpathp.c, because it uses PyWin_DLLVersionString and PyWin_DLLhModule unconditionally, which are implemented in PC/dl_nt.c only when Py_ENABLE_SHARED is defined. The attached patch fixes the problem by wrapping dependent parts in if #ifdef Py_ENABLE_SHARED/#endif. -- components: Windows files: python-2.6-static.patch keywords: patch messages: 76778 nosy: snaury severity: normal status: open title: Python 2.6 fails to build with Py_NO_ENABLE_SHARED type: compile error versions: Python 2.6 Added file: http://bugs.python.org/file12203/python-2.6-static.patch ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue4494> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4494] Python 2.6 fails to build with Py_NO_ENABLE_SHARED
Changes by Alexey Borzenkov <[EMAIL PROTECTED]>: -- nosy: +loewis ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue4494> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47245] potential undefined behavior with subprocess using vfork() on Linux
Alexey Izbyshev added the comment: In short: both this bug report and [1] are invalid. The reason why doing syscall(SYS_vfork) is illegal is explained by Florian Weimer in [2]: >The syscall function in glibc does not protect the on-stack return address >against overwriting, so it can't be used to call SYS_vfork on x86. This is off-topic here because CPython calls vfork() via libc, but I'll still expand the Florian's comment. Suppose one wants to write my_vfork() wrapper over vfork syscall. When vfork syscall returns the first time, my_vfork() has to return to its caller. This necessarily involves knowing the return address. On some architectures this return address is stored on the stack by the caller (e.g. x86). The problem is that any data in my_vfork() stack frame can be overwritten by its caller once it returns the first time. Then, when vfork syscall returns the second time, my_vfork() could be unable to return to its caller because the data it fetches from its (now invalid) stack frame is garbage. This is precisely what happens when one implements my_vfork() as syscall(SYS_vfork). To avoid this, the most common strategy is to store the return address into a register that's guaranteed to be preserved around syscall by the OS ABI. For example, the x86-64 musl implementation [3] stores the retur n address in rdx (which is preserved around syscall) and then restores it after syscall (both on the first and the second return of the syscall). Now back to CPython. The real problem with stack sharing between the child and the parent could be due to compiler bugs, e.g. if a variable stored on the stack is modified in the child branch of "if (vfork())", but the compiler assumes that some other variable sharing the stack slot with the first one is *not* modified in the parent branch (i.e. after vfork() returns the second time). But all production compilers handle vfork() (and setjmp(), which has a similar issue) in a special way to avoid this, and GCC has __attribute__((returns_twice)) that a programmer could use for custom functions behaving in this way (my_vfork() would have to use this attribute). Regarding a separate stack for the child and clone(CLONE_VM|CLONE_VFORK), I considered this in #35823, but this has its own problems. The most important one is that CPython would be in business of choosing the correct stack size for the child's stack, but CPython is *not* in control of all the code executing in the child because it calls into libc. In practice, people use various LD_PRELOAD-based software that wraps various libc functions (e.g. Scratchbox 2 build environment for Sailfish OS is an LD_PRELOAD-based sandbox), so even common-sense assumptions like "execve() in libc can't use a lot of stack!" might turn out to be wrong. CPython *could* work around that by using direct syscalls for everything in the child, or by using some "large" size that "should be enough for everybody", but I wouldn't want to see that unless we have a real problem with vfork(). Note that vfork()-based posix_spawn() implementations in C libraries do *not* have this problem because they fully control all code in the child (e.g. they would use a non-interposable execve() symbol or a direct syscall). > Immediate action item: Add a way for people to disable vfork at runtime by > setting a flag in the subprocess module, just in case. I don't think any action is needed at all, and I think this issue should be closed. > Our current assumptions around the use of vfork() are very much glibc > specific. Could you clarify what glibc-specific assumptions you mean? In #35823 I tried to use as little assumptions as possible. [1] https://bugzilla.kernel.org/show_bug.cgi?id=215813 [2] https://bugzilla.kernel.org/show_bug.cgi?id=215813#c2 [3] https://git.musl-libc.org/cgit/musl/tree/src/process/x86_64/vfork.s?id=ced75472d7e3d73d5b057e36ccbc7b7fcba95104 -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue47245> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47245] potential undefined behavior with subprocess using vfork() on Linux?
Alexey Izbyshev added the comment: > As for glibc specifics, I'm mostly thinking of the calls we do in the child. > According to the "Standard Description (POSIX.1)" calls to anything other > than `_exit()` or `exec*()` are not allowed. But the longer "Linux > Description" in that vfork(2) man page does not say that. Which implies > merely by omission that calls to other things are okay so long as you > understand everything they do to the process heap/stack/state. (I wish it > were *explicit* about that) If we're talking about the kernel side of things, sure, we rely on Linux being "sane" here, though I suppose on *BSDs the situation is similar. > Some of the calls we do from our child_exec() code... many are likely "just" > syscall shims and thus fine - but that is technically up to libc. Yes, but I wouldn't say that "being just syscall shims" is specific for glibc. It's just a "natural" property that just about any libc is likely to possess. (Yeah, I know, those are vague words, but in my experience "glibc-specific" is usually applied to some functionality/bug present in glibc and absent in other libcs, and I don't think we rely on something like that). Of course, there are also LD_PRELOAD things that could be called instead of libc, but good news here is that we don't create new constrains for them (CPython is not the only software that uses vfork()), and they're on their own otherwise. > A few others are Py functions that go elsewhere in CPython and while they may > be fine for practical reasons today with dangerous bits on conditional > branches that technically should not be possible to hit given the state by > the time we're at this point in _posixsubprocess, pose a future risk - anyone > touching implementations of those is likely unaware of vfork'ed child > limitations that must be met. We already have async-signal-safety requirement for all such code because of fork(). Requirements of vfork() are a bit more strict, but at least the set of functions we have to watch for dangerous changes is the same. And I suspect that most practical violations of vfork()-safety also violate async-signal-safety. > For example if one of the potential code paths that trigger an indirect > Py_FatalError() is hit... that fatal exit code is definitely not > post-vfork-child safe. The pre-exec child dying via that could screw up the > vfork parent process's state. Yeah, and it can break the fork parent too, at least because it uses exit() (not _exit()), so stdio buffers will be flushed twice, in the child and in the parent. -- ___ Python tracker <https://bugs.python.org/issue47245> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: The preceding comment is wrong, see discussion in #47245 and https://bugzilla.kernel.org/show_bug.cgi?id=215813#c14 for explanation of why that bug report is irrelevant for CPython. -- ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47245] potential undefined behavior with subprocess using vfork() on Linux?
Alexey Izbyshev added the comment: > 3. We have to fix error-path in order not to change heap state (contents and > allocations), possibly do not touch locks. During vfork() child execution - > the only parent THREAD (not the process) is blocked. For example, it's not > allowed to touch GIL. Child process may die unexpectedly and leave GIL > locked. Is it possible to rewrite children path for vfork() case without any > Py* calls ? As an idea, we can prepare all low-level things (all the pointers > to strings and plain values) before vfork(), so child code will use only that > data. What specifically do you propose to fix? There is no problem with GIL if the child dies because the GIL is locked and unlocked only by the parent and the child never touches it. Similarly, only Py_* calls known to be safe are used. As for "pointers to strings", it's not clear to me what you mean, but if you mean allocations, they are already done before (v)fork(), since the child code is required to be async-signal-safe even if plain fork() is used. -- ___ Python tracker <https://bugs.python.org/issue47245> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47260] os.closerange() can be no-op in a seccomp sandbox
New submission from Alexey Izbyshev :
After #40422 _Py_closerange() assumes that close_range() closes all file
descriptors even if it returns an error (other than ENOSYS):
if (close_range(first, last, 0) == 0 || errno != ENOSYS) {
/* Any errors encountered while closing file descriptors are ignored;
* ENOSYS means no kernel support, though,
* so we'll fallback to the other methods. */
}
else
/* fallbacks */
This assumption can be wrong on Linux if a seccomp sandbox denies the
underlying syscall, pretending that it returns EPERM or EACCES. In this case
_Py_closerange() won't close any descriptors at all, which in the worst case
can be a security issue.
I propose to fix this by falling back to other methods in case of *any*
close_range() error. Note that fallbacks will not be triggered on any problems
with closing individual file descriptors because close_range() is documented to
ignore such errors on both Linux[1] and FreeBSD[2].
[1] https://man7.org/linux/man-pages/man2/close_range.2.html
[2] https://www.freebsd.org/cgi/man.cgi?query=close_range&sektion=2
--
assignee: izbyshev
components: Library (Lib)
keywords: 3.10regression
messages: 416986
nosy: gregory.p.smith, izbyshev, kevans, kevans91
priority: normal
severity: normal
status: open
title: os.closerange() can be no-op in a seccomp sandbox
type: behavior
versions: Python 3.10, Python 3.11
___
Python tracker
<https://bugs.python.org/issue47260>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47260] os.closerange() can be no-op in a seccomp sandbox
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +30443 stage: -> patch review pull_request: https://github.com/python/cpython/pull/32418 ___ Python tracker <https://bugs.python.org/issue47260> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47260] os.closerange() can be no-op in a seccomp sandbox
Alexey Izbyshev added the comment: > It's been years now and that hasn't happened, even with more recent flag > additions. I think it's safe to say it won't, and such a fallback upon error > won't put us back into a bogus pre-close_range situation where we're > needlessly close()ing a bunch of closed fds. Yes, I also find it unlikely that close_range() will start reporting an error if it fails to close some fd, especially if not given some new flag. Such error reporting doesn't seem very useful to userspace because there is no way to associate the error with the fd. But even if such change happens, it will be just a performance regression, not a potential correctness/security issue. -- keywords: -patch stage: patch review -> ___ Python tracker <https://bugs.python.org/issue47260> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47260] os.closerange() can be no-op in a seccomp sandbox
Change by Alexey Izbyshev : -- keywords: +patch stage: -> patch review ___ Python tracker <https://bugs.python.org/issue47260> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34977] Release Windows Store app containing Python
Change by Alexey Izbyshev : -- pull_requests: +16379 pull_request: https://github.com/python/cpython/pull/5812 ___ Python tracker <https://bugs.python.org/issue34977> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35118] Add peek() or first() method in queue
Alexey Volkov added the comment: >Why not just dismiss older queue entries during a normal get() operation? Or >just use a plain deque with access guarded by a lock. You do not know whether the item needs to be removed until you see it. And to see it you need to get it. And if you get it, you cannot push it back to the start of the queue. >FWIW, the standard library queue module doesn't have a straight-forward way to >implement a peek() method. I currently use `q.deque[0]` >Or just use a plain deque with access guarded by a lock. I can. But technically the same applies to almost any queue usage. -- ___ Python tracker <https://bugs.python.org/issue35118> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32592] Drop support of Windows Vista and Windows 7
Alexey Izbyshev added the comment: As far as I understand, commit [1] explicitly prevents CPython from running on Windows 7, and it's included into 3.9. So it seems to be too late to complain, despite that, according to Wikipedia, more than 15% of all Windows PCs are still running Windows 7 [2]. [1] https://github.com/izbyshev/cpython/commit/6a65eba44bfd82ccc8bed4b5c6dd6637549955d5 [2] https://en.wikipedia.org/wiki/Windows_7 -- ___ Python tracker <https://bugs.python.org/issue32592> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32592] Drop support of Windows Vista and Windows 7
Alexey Izbyshev added the comment: > If Win8-only calls are not used, then presumably it should still build and > run on Windows 7, presumably with the flag flipped back to Win7. And if there > are Win8-only calls used and the flag is set to Win7+, I assume that the MSVC > compiler will catch any instances of Win8-only calls at build time? Is this > understanding correct? It's the latter. In the patch, calls to functions like PathCchCanonicalizeEx(), which had previously been conditional on runtime availability of those functions, were made direct. Those functions are not from UCRT (so you can't just distribute it to regain Windows 7 support) and are not available in Windows 7. There was also another patch that adds a check to Python 3.9+ installer that prevents installation on Windows 7 (but again, even if you take the Windows "embedded" distro instead of the installer, it won't run due to Win8-only functions). > If the latter is true, its very likely a lost cause. However, if the former > is the case, then at least devs using Python 3.9+ on Windows 7 will be able > to easily build their own versions with support, though that could change at > any time. For now, it's not entirely lost cause in the sense that the number of Win8-specific patches and their size is very small (I'm not aware of anything not already mentioned, but I may be not up-to-date). So an interested party can probably revert them locally without much hassle. > For those in the know, have there been a lot of reports/discontent > surrounding the dropping of support in Py3.9, by anyone aware of that? I semi-casually follow the bug tracker and remember only bpo-41412 and bpo-42027. Personally, I do consider tying of Windows support in CPython to arbitrary support periods set by Microsoft unfortunate, and it's also true that Win8-specific changes I'm aware of are not critical and don't remove any significant maintenance burden. For now, both relative and absolute usage of Windows 7 is huge and hard to ignore, so developers who use CPython in their own software are faced with an unpleasant choice of sticking to 3.8 or dropping Win 7. (I'm one of them since software I work on has to run in build environments setup for other projects, and such environments routinely stick to old OS versions due to breakage on updates). But this is what we have according to PEP 11, and a little grumbling here certainly won't change that :) -- ___ Python tracker <https://bugs.python.org/issue32592> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32592] Drop support of Windows Vista and Windows 7
Alexey Izbyshev added the comment: > If we had a dedicated maintainer who was supporting Win7 and making releases > for it, then we (i.e. they) could support it. But then, there's nothing to > stop someone doing that already, and even to stop them charging money for it > if they want (which they wouldn't be able to do under the auspices of > python-dev). So I suspect nobody is really that motivated ;) That's totally fair. > (Also, a "little bit of complaining" here is totally worthwhile, as it helps > us gauge community sentiment. Without it, we're limited to trawling forums > and Twitter - especially without conferences to actually meet and hear > directly from people - and so our inputs are biased. Having polite, informed, > discussion on the tracker is a great way for us to see and capture these > positions.) I agree. In my comment, I only intended to contrast "complaining" with stepping up to do the work. I didn't mean to imply that it's inappropriate per-se. -- ___ Python tracker <https://bugs.python.org/issue32592> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24964] Add tunnel CONNECT response headers to httplib / http.client
Change by Alexey Namyotkin : -- pull_requests: +24786 stage: needs patch -> patch review pull_request: https://github.com/python/cpython/pull/26152 ___ Python tracker <https://bugs.python.org/issue24964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24964] Add tunnel CONNECT response headers to httplib / http.client
Alexey Namyotkin added the comment: Thanks, Terry. I signed it. -- ___ Python tracker <https://bugs.python.org/issue24964> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44656] Dangerous mismatch between MAXPATHLEN and MAX_PATH on Windows
New submission from Alexey Izbyshev :
In PC/getpathp.c CPython uses buffers with length MAXPATHLEN+1, which is 257 on
Windows[1]. On Windows 7, where PathCch* functions are not available, CPython
<= 3.8 fallbacks to PathCombineW()/PathCanonicalizeW()[2]. Those functions
assume that the destination buffer has room for MAX_PATH (260) characters. This
creates a dangerous setup: for example, gotlandmark()[3] can overflow the
destination if `filename` is long enough, and `filename` can be user-controlled.
I couldn't devise a simple way to trigger a buffer overflow in a default Python
installation, though it is possible if one, for example, makes sure that the
landmark file ("lib\os.py") can't be found in the default locations and then
supplies their own, long enough paths via e.g. PYTHONPATH environment variable
which eventually end up in gotlandmark(). Even when such buffer overflow is
triggered on my machine, I couldn't notice any change in behavior, probably
because 3 bytes is small enough to not overwrite anything important.
However, I'm not comfortable with this. Could we just raise MAXPATHLEN from 256
to 260 on Windows to avoid such kind of issues for sure?
Please also note that while the issue described above affects only Python <=
3.8 on Windows 7, I think it would make sense to increase MAXPATHLEN in newer
versions too to avoid any similar situations in the future (i.e. if two pieces
of code interact and one of them uses MAX_PATH while another uses MAXPATHLEN).
[1]
https://github.com/python/cpython/blob/0389426fa4af4dfc8b1d7f3f291932d928392d8b/Include/osdefs.h#L13
[2]
https://github.com/python/cpython/blob/0389426fa4af4dfc8b1d7f3f291932d928392d8b/PC/getpathp.c#L278
[3]
https://github.com/python/cpython/blob/0389426fa4af4dfc8b1d7f3f291932d928392d8b/PC/getpathp.c#L333
--
components: Windows
messages: 397655
nosy: izbyshev, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Dangerous mismatch between MAXPATHLEN and MAX_PATH on Windows
type: security
versions: Python 3.6, Python 3.7, Python 3.8
___
Python tracker
<https://bugs.python.org/issue44656>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42738] subprocess: don't close all file descriptors by default (close_fds=False)
Alexey Izbyshev added the comment: > As a concrete example, we have a (non-Python) build system and task runner > that orchestrates many tasks to run in parallel. Some of those tasks end up > invoking Python scripts that use subprocess.run() to run other programs. Our > task runner intentionally passes an inheritable file descriptor that is > unique to each task as a form of a keep-alive token; if the child processes > continue to pass inheritable file descriptors to their children, then we can > determine whether all of the processes spawned from a task have terminated by > checking whither the last open handle to that file descriptor has been > closed. This is particularly important when a processes exits before its > children, sometimes uncleanly due to being force killed by the system or by a > user. I don't see how such scheme could be usable in a general-purpose build system. Python is just one example of a program that closes file descriptors for child processes, but there might be arbitrary more. In general, there is no guarantee that a descriptor would be inherited in a process tree if it contains at least one process running code that you don't fully control. To properly control process trees, I'd suggest to consider prctl(PR_SET_CHILD_SUBREAPER) or PID namespaces on Linux and job objects on Windows (don't know about other OSes). > Regarding security, PEP 446 already makes it so that any files opened from > within a Python program are non-inheritable by default, which I agree is a > good default. One can make the argument that it's not Python's job to enforce > a security policy on file descriptors that a Python process has inherited > from a parent process, since Python cannot distinguish from descriptors that > were accidentally or intentionally inherited. While I agree that such argument could be made, closing FDs for children also affects FDs opened by C extensions, which are not subject to PEP 446. And this is important not only for security: for example, it's easy to create a deadlock if one process is blocked on a read() from a pipe, which it expects to be closed at a proper moment, but the pipe write end is leaked to some unrelated long-running process which keeps it alive indefinitely without being aware of it. And again, even if subprocess didn't close FDs by default (or somehow closed only non-inherited FDs), how could this be used in a wider picture? Any third-party library (even written in Python) could still decide to close e.g. some range of descriptors, so one wouldn't be able to rely on FD inheritance in a general setting, just as one can't rely on inheritance of environment variables (works most of the time, but no promises). Traditional Unix things like inheritance across fork()/exec() and no O_CLOEXEC by default are convenient for quick hacks, but my impression is that such defaults are considered to be design bugs in the modern setting by many. -- ___ Python tracker <https://bugs.python.org/issue42738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38933] unusual behaviour on list of dependable lambdas
New submission from Alexey Burdin : >>> x=[2,3] >>> [f(x) for f in [(lambda a:a[i]) for i in [0,1]]] [3, 3] >>> [f(x) for f in [lambda a:a[0],lambda a:a[1]]] [2, 3] -- components: Interpreter Core messages: 357586 nosy: Alexey Burdin priority: normal severity: normal status: open title: unusual behaviour on list of dependable lambdas type: behavior versions: Python 2.7, Python 3.6, Python 3.7, Python 3.8 ___ Python tracker <https://bugs.python.org/issue38933> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38933] unusual behaviour on list of dependable lambdas
Alexey Burdin added the comment: x=[2,3] [f(x) for f in [(lambda a:a[i]) for i in [0,1]]] #the expected output is [2,3] but actual is [3,3] [f(x) for f in [lambda a:a[0],lambda a:a[1]]] #results [2,3] normally -- ___ Python tracker <https://bugs.python.org/issue38933> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12920] inspect.getsource only works for objects loaded from files, not interactive session
Alexey Volkov added the comment: This is also an issue even for non-interactive scenarios: When doing `python -c ''` inspect.getsource does not work and there are no stack traces. Perhaps this case will be easier to fix? -- nosy: +Ark-kun ___ Python tracker <https://bugs.python.org/issue12920> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42568] Python can't run .pyc files with non-ASCII path on Windows
New submission from Alexey Izbyshev : > python тест.pyc python: Can't reopen .pyc file The issue is caused by _Py_fopen() being used as though it can deal with paths encoded in FS-default encoding (UTF-8 by default on Windows), but in fact it's just a simple wrapper around fopen() from the C runtime, so it uses the current ANSI code page, breaking if PYTHONLEGACYWINDOWSFSENCODING is not enabled. I could find only two callers if _Py_fopen() on Windows: * https://github.com/python/cpython/blob/db68544122f5/Python/pythonrun.c#L380 (which caused this issue) * https://github.com/python/cpython/blob/db68544122f5/Python/errors.c#L1708 PyErr_ProgramText() doesn't seem to be called in CPython, but https://github.com/python/cpython/blob/db68544122f5/Include/pyerrors.h#L243 claims that filename is "decoded from the filesystem encoding", which doesn't match the code. -- components: Interpreter Core, Windows messages: 382490 nosy: izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware priority: normal severity: normal status: open title: Python can't run .pyc files with non-ASCII path on Windows type: behavior versions: Python 3.10, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42568> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42569] Callers of _Py_fopen/_Py_wfopen may be broken after addition of audit hooks
New submission from Alexey Izbyshev :
Before addition of audit hooks in 3.8, _Py_fopen() and _Py_wfopen() were simple
wrappers around corresponding C runtime functions. They didn't require GIL,
reported errors via errno and could be safely called during early interpreter
initialization.
With the addition of PySys_Audit() calls, they can also raise an exception,
which makes it unclear how they should be used. At least one caller[1] is
confused, so an early-added hook (e.g. from sitecustomize.py) that raises a
RuntimeError on at attempt to open the main file causes the following:
$ ./python /home/test/test.py
./python: can't open file '/home/test/test.py': [Errno 22] Invalid argument
Traceback (most recent call last):
File "/home/test/.local/lib/python3.10/site-packages/sitecustomize.py", line
10, in hook
raise RuntimeError("XXX")
RuntimeError: XXX
"Invalid argument" is reported by pymain_run_file() due to a bogus errno, and
the real problem (exception from the hook) is noticed only later.
Could somebody share the current intended status/role of these helpers?
Understanding that seems to be required to deal with issue 32381.
[1] https://github.com/python/cpython/blob/066394018a84/Modules/main.c#L314
--
components: Interpreter Core
messages: 382499
nosy: izbyshev, steve.dower, vstinner
priority: normal
severity: normal
status: open
title: Callers of _Py_fopen/_Py_wfopen may be broken after addition of audit
hooks
type: behavior
versions: Python 3.10, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue42569>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path
Alexey Izbyshev added the comment: Thanks, Eryk, for catching the dup, I missed it somehow. @ZackerySpytz: do you plan to proceed with your PR? If not, I can pick it up -- this issue broke the software I develop after upgrade to 3.8. I filed issue 42569 to hopefully clarify the status of _Py_fopen() which became murky to me. -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue32381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path
Alexey Izbyshev added the comment: Thanks for the patch, Victor, it looks good. Just so it doesn't get lost: the problem with the contract of PyErr_ProgramText() which I mentioned in my dup 42568 is still there. -- ___ Python tracker <https://bugs.python.org/issue32381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path
Alexey Izbyshev added the comment: > It seems like PyErr_ProgramText() is no longer used in Python. Isn't it a part of the public API? I can't find it in the docs, but it seems to be declared in the public header. -- ___ Python tracker <https://bugs.python.org/issue32381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42569] Callers of _Py_fopen/_Py_wfopen may be broken after addition of audit hooks
Alexey Izbyshev added the comment: > To implement PEP 446: create non-inheritable file descriptors. Yes, I understand that was the original role. But currently there is no easy way to deal with errors from the helpers because of exception vs. errno conundrum. Maybe they should be split to two functions each (one that always reports errors via an exception, and the other is a low-level one)? Or, alternatively, keep only exception-reporting variants but check all callers so that they don't use errno and call them from the right context (with GIL, etc.)? -- ___ Python tracker <https://bugs.python.org/issue42569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42569] Callers of _Py_fopen/_Py_wfopen may be broken after addition of audit hooks
Alexey Izbyshev added the comment: > So it should be, "if they fail and you're in a context where exceptions are > allowed, raise an exception" (which will chain back to the one raised from an > audit hook". What exception should be raised if _Py_fopen() fails (returns NULL)? We can't just check errno anymore because _Py_fopen() might have failed because of the audit hook and thus didn't set errno. So it seems to me that addition of audit hooks broke the implicit contract of _Py_fopen()/_Py_wfopen(), and callers were not fixed. -- ___ Python tracker <https://bugs.python.org/issue42569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42585] Segmentation fault on Linux with multiprocess queue
Change by Alexey Izbyshev : -- components: +Library (Lib) nosy: +davin, pitrou versions: -Python 3.6 ___ Python tracker <https://bugs.python.org/issue42585> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path
Alexey Izbyshev added the comment: Thanks for the fix and backports! -- resolution: fixed -> stage: resolved -> patch review status: closed -> open versions: +Python 3.7 ___ Python tracker <https://bugs.python.org/issue32381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32381] Python 3.6 cannot reopen .pyc file with non-ASCII path
Change by Alexey Izbyshev : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed versions: -Python 3.7 ___ Python tracker <https://bugs.python.org/issue32381> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42602] seekable() returns True on pipe objects in Windows
Alexey Izbyshev added the comment: Yes, despite that MSVCRT knows the type of the file descriptor because it calls GetFileType() on its creation, it doesn't check it in lseek() implementation and simply calls SetFilePointer(), which spuriously succeeds for pipes. MSDN says the following[1]: Calling the SetFilePointer function with a handle to a non-seeking device such as a pipe or a communications device is not supported, even though the SetFilePointer function may not return an error. The behavior of the SetFilePointer function in this case is undefined. [1] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-setfilepointer -- components: +Windows nosy: +eryksun, izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware versions: +Python 3.10, Python 3.8 ___ Python tracker <https://bugs.python.org/issue42602> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42569] Callers of _Py_fopen/_Py_wfopen may be broken after addition of audit hooks
Alexey Izbyshev added the comment: Great approach :) -- ___ Python tracker <https://bugs.python.org/issue42569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
New submission from Alexey Izbyshev : On POSIX-conforming systems, O_APPEND flag for open() must ensure that no intervening file modification occurs between changing the file offset and the write operation[1]. In effect, two processes that independently opened the same file with O_APPEND can't overwrite each other's data. On Windows, however, the Microsoft C runtime implements O_APPEND as an lseek(fd, 0, SEEK_END) followed by write(), which obviously doesn't provide the above guarantee. This affects both os.open() and the builtin open() Python functions, which rely on _wopen() from MSVCRT. A demo is attached. While POSIX O_APPEND doesn't guarantee the absence of partial writes, the guarantee of non-overlapping writes alone is still useful in cases such as debug logging from multiple processes without file locking or other synchronization. Moreover, for local filesystems, partial writes don't really occur in practice (barring conditions such as ENOSPC or EIO). Windows offers two ways to achieve non-overlapping appends: 1. WriteFile()[2] with OVERLAPPED structure with Offset and OffsetHigh set to -1. This is essentially per-write O_APPEND. 2. Using a file handle with FILE_APPEND_DATA access right but without FILE_WRITE_DATA access right. While (1) seems easy to add to FileIO, there are some problems: * os.write(fd) can't use it without caller's help because it has no way to know that the fd was opened with O_APPEND (there is no fcntl() in MSVCRT) * write() from MSVCRT (currently used by FileIO) performs some additional remapping of error codes and checks after it calls WriteFile(), so we'd have to emulate that behavior or risk breaking compatibility. I considered to go for (2) by reimplementing _wopen() via CreateFile(), which would also allow us to solve a long-standing issue of missing FILE_SHARE_DELETE on file handles, but hit several problems: * the most serious one is rather silly: we need to honor the current umask to possibly create a read-only file, but there is no way to query it without changing it, which is not thread-safe. Well, actually, I did discover a way: _umask_s(), when called with an invalid mask, returns both EINVAL error and the current umask. But this behavior directly contradicts MSDN, which claims that _umask_s() doesn't modify its second argument on failure[3]. So I'm not willing to rely on this until Microsoft fixes their docs. * os module exposes some MSVCRT-specific flags for use with os.open() (like O_TEMPORARY), which a reimplementation would have to support. It seems easy in most cases, but there is O_TEXT, which enables some obscure legacy behavior in MSVCRT such as removal of a trailing byte 26 (Ctrl-Z) when a file is opened with O_RDWR. More generally, it's unclear to me whether os.open() is explicitly intended to be a gateway to MSVCRT and thus support all current and future flags or is just expected to work similarly to MSVCRT in common cases. So in the end I decided to let _wopen() create the initial fd as usual, but then fix it up via DuplicateHandle() -- see the PR. [1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html [2] https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-writefile [3] https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/umask-s?view=msvc-160 -- components: IO, Windows files: test.py messages: 382784 nosy: eryksun, izbyshev, paul.moore, steve.dower, tim.golden, vstinner, zach.ware priority: normal severity: normal status: open title: Support POSIX atomicity guarantee of O_APPEND on Windows type: enhancement versions: Python 3.10 Added file: https://bugs.python.org/file49661/test.py ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +22575 stage: -> patch review pull_request: https://github.com/python/cpython/pull/23712 ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42655] Fix subprocess extra_groups gid conversion
Alexey Izbyshev added the comment: This bug would have been caught at compile time if `_Py_Gid_Converter()` used `gid_t *` instead of `void *`. I couldn't find any call sites where `void *` would be needed, so probably `_Py_Gid_Converter()` should be fixed too (in a separate PR/issue?). The same applies to `_Py_Uid_Converter()`. -- nosy: +gregory.p.smith, izbyshev ___ Python tracker <https://bugs.python.org/issue42655> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42707] Python uses ANSI CP for stdio on Windows console instead of using console or OEM CP
Alexey Izbyshev added the comment: > I've been struggling to understand today why a simple file redirection > couldn't work properly today (encoding issues) The core issue is that "working properly" is not defined in general when we're talking about piping/redirection, as opposed to the console. Different programs that consume Python's output (or produce its input) can have different expectations wrt. data encoding, and there is no way for Python to know it in advance. In your examples, you use programs like "more" and "type" to print the Python's output back to the console, so in this case using the OEM code page would produce the result that you expect. But, for example, in case Python's output was to be consumed by a C program that uses simple `fopen()/wscanf()/wprintf()` to work with text files, the ANSI code page would be appropriate because that's what the Microsoft C runtime library defaults to for wide character operations. Python has traditionally used the ANSI code page as the default IO encoding for non-console cases (note that Python makes no distinction between non-console `sys.std*` and the builtin `open()` wrt. encoding), and this behavior can't be changed. You can use `PYTHONIOENCODING` or enable the UTF-8 mode[1] to change the default encoding. Note that in your example you could simply use `PYTHONIOENCODING=cp850`, which would remove the need to use `chcp`. [1] https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUTF8 -- nosy: +izbyshev, vstinner versions: +Python 3.10, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42707> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42738] subprocess: don't close all file descriptors by default (close_fds=False)
Alexey Izbyshev added the comment: > Using close_fds=False, subprocess can use posix_spawn() which is safer and > faster than fork+exec. For example, on Linux, the glibc implements it as a > function using vfork which is faster than fork if the parent allocated a lot > of memory. On macOS, posix_spawn() is even a syscall. On Linux, unless you care specifically about users with Python 3.10+ on older kernels, implementing support for closerange() syscall in subprocess would provide better net benefit. This is because (a) performance advantage of posix_spawn() is no longer relevant on Linux after bpo-35823 and (b) supporting closerange() would benefit even those users who still need close_fds=True. -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue42738> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38435] Start the deprecation cycle for subprocess preexec_fn
Change by Alexey Izbyshev : -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue38435> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42736] Add support for making Linux prctl(...) calls to subprocess
Change by Alexey Izbyshev : -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue42736> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42780] os.set_inheritable() fails for O_PATH file descriptors on Linux
Change by Alexey Izbyshev : -- components: +Library (Lib) nosy: +vstinner versions: -Python 3.6, Python 3.7 ___ Python tracker <https://bugs.python.org/issue42780> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42888] Not installed “libgcc_s.so.1” causes parser crash.
Alexey Izbyshev added the comment: I've encountered this issue too. My use case was a 32-bit Python on a 64-bit CentOS system, and my understanding of the issue was that 64-bit libgcc_s is somehow counted as a "provider" of libgcc_s for 32-bit libc by the package manager, so 32-bit libgcc_s is never installed even when "yum install glibc.i686" is used because everything seems "already OK": $ yum deplist glibc ... dependency: libgcc provider: libgcc.x86_64 4.1.2-55.el5 provider: libgcc.i386 4.1.2-55.el5 ... (The above is for CentOS 5, but at the time I tested 6 and 7 as well, with the same result). But I suggest not to dismiss this issue as a packaging bug. Glibc needs libgcc_s for pthread_exit() because it's implemented in terms of pthread_cancel(), and the latter wants to do stack unwinding (probably to cleanup resources for each stack frame). But the code for unwinding lives in libgcc_s. The non-intuitive thing here is that glibc tried to optimize this dependency by dlopen'ing libgcc_s when pthread_cancel() is called instead of making it a startup dependency. Many people consider this a terrible design choice since your program can now fail at an arbitrary moment due to dlopen() failure (and missing libgcc_s is not the only possible reason[1]). Since CPython doesn't use pthread_cancel() directly, I propose a simple solution -- just `return NULL` from thread functions instead of calling pthread_exit(). The last time I looked, pthread_exit() in CPython is only called from top frames of the threads, so a direct `return` should suffice. If the top-level thread function simply returns, no stack unwinding is needed, so glibc never uses its cancellation machinery. I've tested that this solution worked at the time (for 3.8 I think), and the test suite passed. If there is interest in going this way, I can test again. [1] https://www.sourceware.org/bugzilla/show_bug.cgi?id=13119 -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue42888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42888] Not installed “libgcc_s.so.1” causes exit crash.
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +23063 stage: -> patch review pull_request: https://github.com/python/cpython/pull/24241 ___ Python tracker <https://bugs.python.org/issue42888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42888] Not installed “libgcc_s.so.1” causes exit crash.
Alexey Izbyshev added the comment: I've made a PR to remove most calls to pthread_exit(). @xxm: could you test it in your environment? -- ___ Python tracker <https://bugs.python.org/issue42888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
Alexey Izbyshev added the comment: Could anybody provide their thoughts on this RFE? Thanks. -- ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
Alexey Izbyshev added the comment: > It's possible to query the granted access of a kernel handle via > NtQueryObject: ObjectBasicInformation Ah, thanks for the info. But it wouldn't help for option (1) that I had in mind because open() and os.open() currently set only msvcrt-level O_APPEND. It probably could be used in my current PR instead of just assuming the default access rights for file handles, but I'm not sure whether it makes sense: can a new handle have non-default access rights? Or can the default change at this point of Windows history? > Python could implement its own umask value in Windows. os.umask() would set > the C umask value as well, but only for the sake of consistency with C > extensions and embedding. Would it be a long shot to ask MS to add the needed functionality to MSVCRT instead? Or at least to declare the bug with _umask_s() that I described a feature. Maybe Steve Dower could help. > Open attribute flags could also be supported, such as O_ATTR_HIDDEN and > O_ATTR_SYSTEM. These are needed because a hidden or system file is required > to remain as such when it's overwritten, else CreateFileW fails. I didn't know that, thanks. Indeed, a simple open(path, 'w') fails on a hidden file with "Permission denied". > If it's important enough, msvcrt.open() and msvcrt.O_TEXT could be provided. Yes, I'd be glad to move support for more obscure MSVCRT flags to msvcrt.open() -- the less MSVCRT details we leak in os.open(), the more freedom we have to implement it via proper Win32 APIs. === Anyway, even if the blockers for implementing open()/os.open() via CreateFile() are solved, my current PR doesn't seem to conflict with such an implementation (it could just be replaced in the future). Currently, the only way to achieve atomic append in Python on Windows that I know is to use a custom opener that would call CreateFile() with the right arguments via ctypes/pywin32/etc. -- ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42888] Not installed “libgcc_s.so.1” causes exit crash.
Alexey Izbyshev added the comment: Thank you for testing. I've added a NEWS entry to the PR, so it's ready for review by the core devs. Note that PyThread_exit_thread() can still be called by daemon threads if they try to take the GIL after Py_Finalize(), and also via C APIs like PyEval_RestoreThreads() (see bpo-42969), so, in general, libgcc_s is still required for CPython. -- versions: +Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42888> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42969] pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful
Change by Alexey Izbyshev : -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue42969> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
Alexey Izbyshev added the comment: > I don't know what you mean by default access rights. I meant the access rights of the handle created by _wopen(). In my PR I basically assume that _wopen() uses GENERIC_READ/GENERIC_WRITE access rights, but _wopen() doesn't have a contractual obligation to do exactly that AFAIU. For example, if it got some extra access rights, then my code would "drop" them while switching FILE_WRITE_DATA off. > For example, if FILE_WRITE_DATA isn't granted, then open() can't open for > appending. A direct CreateFileW() call can remove FILE_WRITE_DATA from the > desired access. Indeed, I haven't thought about it. Are you aware of a common scenario when a regular file allows appending but not writing? But, at least, this is not a regression: currently open()/os.open() can't open such files for appending too. > An opener could also work like your PR via os.open(), msvcrt.get_osfhandle(), > _winapi.GetFileType(), _winapi.DuplicateHandle(), os.close(), and > msvcrt.open_osfhandle(). True, but it still falls into "etc" category of "ctypes/pywin32/etc" :) Certainly doable, but it seems better to have consistent O_APPEND behavior across platforms out-of-the-box. -- ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
Alexey Izbyshev added the comment: > FYI, here are the access rights applicable to files Thanks, I checked that mapping in headers when I was writing _Py_wopen_noraise() as well. But I've found a catch via ProcessHacker: CreateFile() with GENERIC_WRITE (or FILE_GENERIC_WRITE) additionally grants FILE_READ_ATTRIBUTES for some reason. This is why I add FILE_READ_ATTRIBUTES to FILE_GENERIC_WRITE for DuplicateHandle() -- otherwise, the duplicated handle didn't have the same rights in my testing. So basically I have to deal with (a) _wopen() not guaranteeing contractually any specific access rights and (b) CreateFile() not respecting the specified access rights exactly. This is where my distrust and my original question about "default" access rights come from. > I overlooked a case that's a complication. For O_TEMPORARY, the open uses > FILE_FLAG_DELETE_ON_CLOSE; adds DELETE to the requested access; and adds > FILE_SHARE_DELETE to the share mode. > _Py_wopen_noraise() can easily keep the required DELETE access. The > complication is that you have to be careful not to close the original file > descriptor until after you've successfully created the duplicate file > descriptor. If it fails, you have to return the original file descriptor from > _wopen(). If you close all handles for the kernel File and fail the call, the > side effect of deleting the file is unacceptable. Good catch! But now I realize that the problem with undoing the side effect applies to O_CREAT and O_TRUNC too: we can create and/or truncate the file, but then fail. Even if we could assume that DuplicateHandle() can't fail, _open_osfhandle() can still fail with EMFILE. And since there is no public function in MSVCRT to replace an underlying handle of an existing fd, we can't preallocate an fd to avoid this. There would be no problem if we could just reduce access rights of an existing handle, but it seems there is no such API. I don't like the idea of silently dropping the atomic append guarantee in case of a failure, so I'm not sure how to proceed with the current approach. Moreover, the same issue would apply even in case of direct implementation of os.open()/open() via CreateFile() because we still need to wrap the handle with an fd, and this may fail with EMFILE. For O_CREAT/O_TRUNC, it seems like it could be reasonably avoided: * Truncation can simply be deferred until we have the fd and then performed manually. * To undo the file creation, we could use GetLastError() to learn whether CreateFile() with OPEN_ALWAYS actually created the file, and then delete it on failure (still non-atomic, and deletion can fail, but probably better than nothing). But I still don't know how to deal with O_TEMPORARY, unless there is a way to unset FILE_DELETE_ON_CLOSE on a handle. As an aside, it's also very surprising to me that O_TEMPORARY is allowed for existing files at all. If not for that, there would be no issue on failure apart from non-atomicity. Maybe I should forgo the idea of supporting O_APPEND for os.open(), and instead just support it in FileIO via WriteFile()-with-OVERLAPPED approach... -- ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42606] Support POSIX atomicity guarantee of O_APPEND on Windows
Alexey Izbyshev added the comment: > I think truncation via TRUNCATE_EXISTING (O_TRUNC, with O_WRONLY or O_RDWR) > or overwriting with CREATE_ALWAYS (O_CREAT | O_TRUNC) is at least tolerable > because the caller doesn't care about the existing data. Yes, I had a thought that creating or truncating a file when asked is in some sense "less wrong" than deleting an existing file on open() failure, but I'm still not comfortable with it. It would be nice, for example, if an open() with O_CREAT | O_EXCL failed, then the file would indeed not be created in all cases. >> Truncation can simply be deferred until we have the fd and then performed >> manually. > If the file itself has to be overwritten (i.e. the default, anonymous data > stream), as opposed to a named data stream, it would have to delete all named > data streams and extended attributes in the file. Normally that's all > implemented atomically in the filesystem. > In contrast, TRUNCATE_EXISTING (O_TRUNC) is simple to emulate, since > CreateFileW implents it non-atomically with a subsequent > NtSetInformationFile: FileAllocationInformation system call. Oh. So CREATE_ALWAYS for an existing file has a very different semantics than TRUNCATE_EXISTING, which means we can't easily use OPEN_ALWAYS with a deferred manual truncation, I see. >> But I still don't know how to deal with O_TEMPORARY, unless there is a >> way to unset FILE_DELETE_ON_CLOSE on a handle. > For now, that's possible with NTFS and the Windows API in all supported > versions of Windows by using a second kernel File with DELETE access, which > is opened before the last handle to the first kernel File is closed. After > you close the first open, use the second one to call SetFileInformation: > FileDispositionInfo to undelete the file. So the idea is to delete the file for a brief period, but then undelete it. As I understand it, any attempt to open the file while it's in the deleted state (but still has a directory entry) will fail. This is probably not critical since it could happen only on an unlikely failure of _open_oshandle(), but is still less than perfect. > Windows 10 supports additional flags with FileDispositionInfoEx (21), or > NTAPI FileDispositionInformationEx [1]. This provides a better way to disable > or modify the delete-on-close state per kernel File object, if the filesystem > supports it. This is nice and would reduce our non-atomicity to just the following: if _wopen() would have failed even before CreateFile() (e.g. due to EMFILE), our reimplementation could still create a handle with FILE_DELETE_ON_CLOSE, so if our process is terminated before we unset it, we'll still lose the file. But it seems like the best we can get in any situation when we need to wrap a handle with an fd. === Regarding using WriteFile()-with-OVERLAPPED approach in FileIO, I've looked at edge cases of error remapping in MSVCRT write() again. If the underlying WriteFile() fails, it only remaps ERROR_ACCESS_DENIED to EBADF, presumably to deal with write() on a O_RDONLY fd. But if WriteFile() writes zero bytes and does not fail, it gets more interesting: * If the file type is FILE_TYPE_CHAR and the first byte to write was 26 (Ctrl-Z), it's treated as success. I don't think I understand: it *seems* like it's handling something like writing to the *input* of a console, but I'm not sure it's even possible in this manner. * Anything else is a failure with ENOSPC. This is mysterious too. I've checked how java.io.FileOutputStream deals with WriteFile() succeeding with zero size (just to compare with another language runtime) and haven't found any special handling[1]. Moreover, it seems like FileOutputStream.write(bytes[]) will enter an infinite loop if WriteFile() always returns 0 for a non-empty byte array[2]. So it's unclear to me what MSVCRT is up to here and whether FileIO should do the same. [1] https://github.com/openjdk/jdk/blob/b4ace3e9799dab5970d84094a0fd1b2d64c7f923/src/java.base/windows/native/libjava/io_util_md.c#L522 [2] https://github.com/openjdk/jdk/blob/b4ace3e9799dab5970d84094a0fd1b2d64c7f923/src/java.base/share/native/libjava/io_util.c#L195 -- ___ Python tracker <https://bugs.python.org/issue42606> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43069] Python fails to read a script whose path is `/dev/fd/X`
Alexey Izbyshev added the comment:
I would suggest to start digging from the following piece of code in
`maybe_pyc_file()` (Python/pythonrun.c):
int ispyc = 0;
if (ftell(fp) == 0) {
if (fread(buf, 1, 2, fp) == 2 &&
((unsigned int)buf[1]<<8 | buf[0]) == halfmagic)
ispyc = 1;
rewind(fp);
}
--
nosy: +izbyshev
___
Python tracker
<https://bugs.python.org/issue43069>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43113] os.posix_spawn errors with wrong information when shebang points to not-existing file
Alexey Izbyshev added the comment: > Ideally, the error would say: > FileNotFoundError: ./demo: /usr/bin/hugo: bad interpreter: No such file or > directory The kernel simply returns ENOENT on an attempt to execve() a file with non-existing hash-bang interpreter. The same occurs on an attempt to run a dynamically linked ELF executable with INTERP program header containing a non-existing path. And, of course, the same error is returned if the executable file itself doesn't exist, so there is no simple way to distinguish such cases. Bash simply assumes[1] that if a file contains a hash-bang and the error from execve() is not recognized otherwise, it's a "bad interpreter". Note that all of the above is completely unrelated to os.posix_spawn(): subprocess or os.execve() would produce the same message. [1] https://git.savannah.gnu.org/cgit/bash.git/tree/execute_cmd.c?h=bash-5.1#n5854 -- nosy: +izbyshev ___ Python tracker <https://bugs.python.org/issue43113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43113] os.posix_spawn errors with wrong information when shebang points to not-existing file
Alexey Izbyshev added the comment: > FileNotFoundError: [Errno 2] No such file or directory: Either './demo' or > the interpreter of './demo' not found. This doesn't sound good to me because a very probable and a very improbable reasons are combined together without any distinction. Another possible issue is that usage of the word "interpreter" in this narrow sense might be non-obvious for users. ISTM that the most minimal way to check for the possibility of interpreter issue would be do something like `access(exe_path, X_OK)` in case of ENOENT: if it's successful, then a "bad interpreter" condition is likely. But in case of os.posix_spawnp(), the search in PATH is performed by libc, so CPython doesn't know exe_path. OTOH, subprocess and os.exec*p do perform their own search in PATH, but in case of subprocess it happens in the context of the child process, so we'll probably need to devise a separate error code to report to the parent via the error pipe to distinguish this condition. So far, I'm not convinced that the result is worth it, but I do agree that such mysterious "No such file" errors are not user-friendly. -- ___ Python tracker <https://bugs.python.org/issue43113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43113] os.posix_spawn errors with wrong information when shebang points to not-existing file
Alexey Izbyshev added the comment: > IMO the fix is simple: only create OSError from the errno, never pass a > filename. This will remove a normally helpful piece of the error message in exchange to being marginally less confusing in a rare case of non-existing interpreter (the user will still be left wondering why the file they tried to run exists, but "No such file or directory" is reported). So the only "win" here would be for CPython developers because users will be less likely to report a bug like this one. > posix_spawn() is really complex function which can fail in many different > ways. This issue is not specific to posix_spawn(): subprocess and os.execve() report the filename too. Any fix would need to change those too for consistency. -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue43113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43113] os.posix_spawn errors with wrong information when shebang points to not-existing file
Alexey Izbyshev added the comment: How do you propose to approach documentation of such behavior? The underlying cause is the ambiguity of ENOENT error code from execve() returned by the kernel, so it applies to all places where Python can call execve(), including os.posixspawn(), os.execve() and subprocess, so it's not clear to me where such documentation should be placed. And, of course, this behavior is not specific to CPython. The Linux man pages mention various causes of this error[1], though POSIX doesn't[2]. While ENOENT ambiguity is indeed confusing, one of the top results of my DDG search on "linux no such file or directory but script exists" is this link[3]. [1] https://man7.org/linux/man-pages/man2/execve.2.html [2] https://pubs.opengroup.org/onlinepubs/9699919799/functions/execve.html [3] https://stackoverflow.com/questions/3949161/no-such-file-or-directory-but-it-exists -- ___ Python tracker <https://bugs.python.org/issue43113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43113] os.posix_spawn errors with wrong information when shebang points to not-existing file
Alexey Izbyshev added the comment: I generally agree, but getting a good, short error message seems to be the hard part here. I previously complained[1] about the following proposal by @hroncok: FileNotFoundError: [Errno 2] No such file or directory: Either './demo' or the interpreter of './demo' not found. But may be it's just me. Does anybody else feel that mentioning "the interpreter" is this way could be confusing in prevalent cases when the actual problem is missing './demo' itself? If we can come up with a good message, I can look into turning it into a PR. The error message above also reads to me like there are no other possible reasons of ENOENT. On Linux, binfmt_misc[2] provides a way to run arbitrary code on execve(). This is used, for example, to transparently run binaries for foreign arches via qemu-user, so probably ENOENT would be returned if QEMU itself it missing. QEMU *may* be thought as a kind of interpreter here, though it's completely unrelated to a hash-bang or an ELF interpreter. But I don't know about other POSIX platforms. As a theoretical example, if the dynamic library loader is implemented in the kernel, the system call could return ENOENT in case of a missing library. Do we need to worry about it? Does anybody know about the situation on macOS, where posix_spawn() is a system call too? [1] https://bugs.python.org/issue43113#msg386210 [2] https://en.wikipedia.org/wiki/Binfmt_misc -- ___ Python tracker <https://bugs.python.org/issue43113> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43308] subprocess.Popen leaks file descriptors opened for DEVNULL or PIPE stdin/stdout/stderr arguments
Change by Alexey Izbyshev : -- nosy: +gregory.p.smith ___ Python tracker <https://bugs.python.org/issue43308> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35118] Add peek() or first() method in queue
Alexey Volkov added the comment: >For Queue, I'm not sure I've ever seen any use case for peek. What do you >have in mind? I want to examine the first (oldest) element in queue and remove it if it's too old. The queue should not be modified unless the oldest element is too old. -- nosy: +Ark-kun ___ Python tracker <https://bugs.python.org/issue35118> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: I'd really like to get this merged eventually because vfork()-based solution is fundamentally more generic than posix_spawn(). Apart from having no issue with close_fds=True, it will also continue to allow subprocess to add any process context tweaks without waiting for availability of corresponding features in posix_spawn(). If there is no objection from Gregory or other devs, I can pick up from where I left in two weeks. -- ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41388] IDLE fails to detect corresponding opening parenthesis
New submission from Alexey Burdin :
```
answers_field_order=sorted(
set(j for i in data['items'] for j in i),
key=cmp_to_key(lambda x,y:(
-1 if (x,y) in answer_order
else (0 if x==y else 1)))
)
```
when the cursor is placed in line 5 col 31 (between `)` and `))` ) hitting
Ctrl-0 (Show surrounding parens) produces an error sound, though there's no
error, the script works fine.
```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 18.04.4 LTS
Release:18.04
Codename: bionic
$ uname -a
Linux odd-one 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020
x86_64 x86_64 x86_64 GNU/Linux
$ python3.8
Python 3.8.0 (default, Oct 28 2019, 16:14:01)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib
>>> importlib.util.find_spec('idlelib.pyshell')
ModuleSpec(name='idlelib.pyshell',
loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f1ea8ee14c0>,
origin='/usr/lib/python3.8/idlelib/pyshell.py')
>>> exit()
$ dpkg-query -S /usr/lib/python3.8/idlelib/pyshell.py
idle-python3.8: /usr/lib/python3.8/idlelib/pyshell.py
$ dpkg -l | grep idle-python3\.8
ii idle-python3.8 3.8.0-3~18.04
all IDE for Python (v3.8) using Tkinter
```
--
assignee: terry.reedy
components: IDLE
messages: 374205
nosy: Alexey Burdin, terry.reedy
priority: normal
severity: normal
status: open
title: IDLE fails to detect corresponding opening parenthesis
type: behavior
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue41388>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7291] urllib2 cannot handle https with proxy requiring auth
Change by Alexey Namyotkin : -- nosy: +alexey.namyotkin ___ Python tracker <https://bugs.python.org/issue7291> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7291] urllib2 cannot handle https with proxy requiring auth
Alexey Namyotkin added the comment: It has been 5 years, now the urllib3 is actively used, but it also inherited this problem: if no authentication data has been received, then the method _tunnel raises an exception OSError that does not contain response headers. Accordingly, this exception cannot be handled. And perhaps this is an obstacle to building a convenient system of authentication on a proxy server in a widely used library requests (it would be nice to be able to just provide an argument proxy_auth, similar to how it is done for server authorization). Now, if a user wants to send a https request through a proxy that requires complex authentication (Kerberos, NTLM, Digest, other) using the urllib3, he must first send a separate request to the proxy, receive a response, extract the necessary data to form the header Proxy-Authorization, then generate this header and pass it to the ProxyManager. And if we are talking about Requests, then the situation there is worse, because you cannot pass proxy headers directly (https://github.com/psf/requests/issues/2708). If we were to aim to simplify the authentication procedure on the proxy server for the user, then where would we start, do we need to change the http.client so that the error returned by the method _tunnel contains headers? Or maybe it's not worth changing anything at all and the path with preliminary preparation by user of the header Proxy-Authorization is the only correct one? Martin Panter, could you also give your opinion? Thank you in advance. -- ___ Python tracker <https://bugs.python.org/issue7291> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: Well, much later than promised, but I'm picking it up. Since in the meantime support for setting uid/gid/groups was merged, and I'm aware about potential issues with calling corresponding C library functions in a vfork()-child, I asked a question on musl mailing list: https://www.openwall.com/lists/musl/2020/10/12/1 So, it seems we'll need to fallback to fork() if set*id() is needed, which is in line with our previous discussion about avoidance of vfork() in privileged processes anyway. I'm also discussing -Wclobbered warnings with a GCC developer. I wouldn't like to restructure code just to avoid GCC false positives, so currently I'm leaning towards disabling this warning entirely for subprocess_fork_exec() and documenting that arbitrary stores to local variables between vfork() and child_exec() are not allowed due to stack sharing, but we'll see if a better solution emerges. -- assignee: -> izbyshev ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: I've updated my PR. * After a discussion with Alexander Monakov (a GCC developer), moved vfork() into a small function to isolate it from both subprocess_fork_exec() and child_exec(). This appears to be the best strategy to avoid -Wclobbered false positives as well as potential compiler bugs. * Got rid of VFORK_USABLE checks in function parameter lists. Now `child_sigmask` parameter is always present, but is NULL if vfork() is not available or usable. The type is `void *` to avoid hard dependence on sigset_t, which other CPython source files appear to avoid. * Disabled vfork() in case setuid()/setgid()/setgroups() needed. * Added more comments explaining vfork()-related stuff. * Added commit message and NEWS entry. Potential improvements: * To avoid repeating long parameter lists in several functions, we can move them to a struct. The downside is that we'd need to convert child_exec() to use that struct all over the place. I don't have a strong preference here. * Are any documentation changes needed? We don't change any interfaces, so I'm not sure. Potential future directions: * If desired, a vfork() opt-out may be implemented. But it'd need to disable posix_spawn() on Linux as well, since it might be implemented via vfork(), and we have no good way to check that. -- ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36034] Suprise halt caused by -Werror=implicit-function-declaration in ./Modules/posixmodule.c
Change by Alexey Izbyshev : -- resolution: -> not a bug ___ Python tracker <https://bugs.python.org/issue36034> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Change by Alexey Izbyshev : -- pull_requests: +21862 pull_request: https://github.com/python/cpython/pull/22944 ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: > Thank you for taking this on! I'm calling it fixed for now as the buildbots > are looking happy with it. If issues with it arise we can address them. Thank you for reviewing and merging! Using POSIX_CALL for pthread_sigmask() is incorrect, however, since it *returns* the error instead of setting errno. I've opened a PR with a fix and additional handling of the first pthread_sigmask() call in the parent (which can be done easily). I've also noticed a memory leak why doing the above: cwd_obj2 is not released on the error path. It probably slipped with patches adding user/groups support. I'll file a separate issue. Would you also clarify why you disallowed vfork() in case setsid() is needed? Have you discovered any potential issues with setsid()? Note that setsid() doesn't suffer from thread sync issues like setuid()/setgid()/etc. -- ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: > regarding excluding the setsid() case: I was being conservative as I couldn't > find a reference of what was and wasn't allowed after vfork. Yes, there is no list of functions allowed after vfork(), except for the conservative POSIX.1 list consisting only of _exit() and execve(), so we can only take async-signal-safe functions as a first approximation and work from there. Thankfully, on Linux, C libraries don't do anything fancy in most cases. But, for example, it appears that on FreeBSD we wouldn't be able to use sigprocmask()/sigaction()[1]. BTW, commit[2] and the linked review are an interesting reading for anybody who would like to support posix_spawn() and/or vfork() in subprocess on FreeBSD. > Confirming, in glibc is appears to be a shim for the setsid syscall (based on > not finding any code implementing anything special for it) and in uclibc > (*much* easier to read) it is clearly just a setsid syscall shim. I also recommend musl[3] when simplicity (and correctness) is required :) [1] https://svnweb.freebsd.org/base/head/lib/libc/gen/posix_spawn.c?view=markup&pathrev=362111#l126 [2] https://svnweb.freebsd.org/base?view=revision&revision=352712 [3] https://git.musl-libc.org/cgit/musl/tree/src/unistd/setsid.c?id=a5aff1972c9e3981566414b09a28e331ccd2be5d -- ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35823] Use vfork() in subprocess on Linux
Alexey Izbyshev added the comment: @ronaldoussoren > I'd prefer to not use vfork on macOS. For one I don't particularly trust that > vfork would work reliably when using higher level APIs, but more importantly > posix_spawn on macOS has some options that are hard to achieve otherwise and > might be useful to expose in subprocess. I can't comment on vfork() usability on macOS myself, but given the number of issues/considerations described here, I expect that significant research would be needed to check that. Regarding your second point about extra posix_spawn() options, I actually don't see why it would be an argument against vfork(). Even on Linux, subprocess prefers posix_spawn() to vfork()/fork() when possible, so vfork() support doesn't hinder posix_spawn(). -- ___ Python tracker <https://bugs.python.org/issue35823> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42146] subprocess.Popen() leaks cwd in case of uid/gid overflow
New submission from Alexey Izbyshev : The following test demonstrates the leak: ``` import subprocess cwd = 'x' * 10**6 for __ in range(100): try: subprocess.call(['/xxx'], cwd=cwd, user=2**64) except OverflowError: pass from resource import * print(getrusage(RUSAGE_SELF).ru_maxrss) ``` The leak was introduced by bpo-36046. Previously, `cleanup:` label was not reachable after `cwd_obj2` was initialized at https://github.com/python/cpython/blob/492d513ccbebeec40a8ba85cbd894a027ca5b2b3/Modules/_posixsubprocess.c#L892 I'll submit a PR with a simple fix suitable for backporting to 3.9. Also, I think it might make sense to unify the two almost-identical cleanup paths we have now. I'll follow up with another PR. -- assignee: izbyshev components: Extension Modules keywords: 3.9regression messages: 379575 nosy: gregory.p.smith, izbyshev, patrick.mclean priority: normal severity: normal status: open title: subprocess.Popen() leaks cwd in case of uid/gid overflow type: behavior versions: Python 3.10, Python 3.9 ___ Python tracker <https://bugs.python.org/issue42146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42146] subprocess.Popen() leaks cwd in case of uid/gid overflow
Change by Alexey Izbyshev : -- keywords: +patch pull_requests: +21882 stage: -> patch review pull_request: https://github.com/python/cpython/pull/22966 ___ Python tracker <https://bugs.python.org/issue42146> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
