[issue32421] Keeping an exception in cache can segfault the interpreter
Yonatan Zunger added the comment: Apologies -- I've been completely swamped and haven't had time to build a repro case. I don't think I have enough state on the way I found this to reconstruct it easily right now, so I'll close it and re-open if I find it again. -- resolution: -> not a bug stage: test needed -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue32421> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32421] Keeping an exception in cache can segfault the interpreter
New submission from Yonatan Zunger :
Using the following decorator to memoize a function causes the interpreter to
segfault on exit if any exceptions were stored in the cache. Changing the bad
line to instead store copy.copy(e) makes the problem go away.
Almost certainly some kind of problem with the GC of the traceback not
happening at the moment the interpreter expects it, but I haven't had time to
investigate any more deeply yet.
def memoize(keyFunc=lambda x: x, cacheExceptions: bool=True):
def wrapper(func):
return _Memo(func, keyFunc, cacheExceptions)
return wrapper
class _Memo(object):
def __init__(self, func, keyFunc, cacheExceptions: bool) -> None:
self.func = func
self.keyFunc = keyFunc
self.cacheExceptions = cacheExceptions
self.lock = threading.Lock()
self.cache = {}
def __call__(self, *args, **kwargs) -> Any:
key = self.keyFunc(*args, **kwargs)
assert isinstance(key, collections.Hashable)
with self.lock:
if key in self.cache:
value = self.cache[key]
if isinstance(value, BaseException):
raise value
return value
try:
result = self.func(*args, **kwargs)
except BaseException as e:
if self.cacheExceptions:
with self.lock:
self.cache[key] = e # BAD LINE
six.reraise(*sys.exc_info())
else:
with self.lock:
self.cache[key] = result
return result
--
components: Interpreter Core
messages: 308978
nosy: zunger
priority: normal
severity: normal
status: open
title: Keeping an exception in cache can segfault the interpreter
type: crash
versions: Python 3.6
___
Python tracker
<https://bugs.python.org/issue32421>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32421] Keeping an exception in cache can segfault the interpreter
Yonatan Zunger added the comment: It's just a clean version of reraise. Part of the six (Py2/Py3 compatibility) library, but pretty useful on its own anyway. On Sun, Dec 24, 2017 at 8:37 PM, Nitish wrote: > > Nitish added the comment: > > What is six.reraise in except block? > > -- > nosy: +nitishch > > ___ > Python tracker > <https://bugs.python.org/issue32421> > ___ > -- ___ Python tracker <https://bugs.python.org/issue32421> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue32421] Keeping an exception in cache can segfault the interpreter
Yonatan Zunger added the comment:
3.6.2. But Nick Coghlan pointed out elsewhere that the underlying issue is
more likely to be one of the other objects being pinned in-memory because
of the traceback, so the exception may actually be just an incidental
thing. I'm going to have to work on a better minimal repro case -- this
particular example had both a subprocess.Popen open at the time and a
message-channel object that holds some os.pipes, so there are all sorts of
fun things that may be causing the trouble.
I'll update the bug when I have a chance to do that and narrow it down a
bit more.
On Tue, Dec 26, 2017 at 3:26 AM, Julien Palard
wrote:
>
> Julien Palard added the comment:
>
> Tested with:
>
> @memoize()
> def foo(x):
> raise Exception("From foo")
>
>
> for _ ∈ range(5):
> try:
> foo(42)
> except Exception as err:
> print(err)
>
> using a python 3.5.4, 3.6.4, and master(13a6c09), and was unable to
> reproduce it, which version of Python are you using? How are you
> reproducing it?
>
> --
> nosy: +mdk
>
> ___
> Python tracker
> <https://bugs.python.org/issue32421>
> ___
>
--
___
Python tracker
<https://bugs.python.org/issue32421>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34115] code.InteractiveConsole.interact() closes stdin
New submission from Yonatan Zunger : code.InteractiveConsole.interact() closes stdin on exit, which can be very surprising to successive code, not least future calls to interact(). A simple repro with a workaround is: import code import io import os import sys def run(): print(sys.stdin.buffer.raw) dupstdin = os.dup(0) try: code.InteractiveConsole().interact() except SystemExit: pass finally: # Workaround: Without this line, the second call to run() will fail with a ValueError when # it tries to call input(). sys.stdin = io.TextIOWrapper( io.BufferedReader(io.FileIO(dupstdin, mode='rb', closefd=False)), encoding='utf8') run() run() - The exciting behavior appears to happen inside the exec() of a 'quit()' command, and I haven't searched it out further. - That behavior inside exec() is likely there for a good reason, in which case the best fix is probably to just save and restore stdin in the code library. -- components: Library (Lib) messages: 321666 nosy: Yonatan Zunger priority: normal severity: normal status: open title: code.InteractiveConsole.interact() closes stdin type: behavior versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue34115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34115] code.InteractiveConsole.interact() closes stdin
Yonatan Zunger added the comment: Testing your code sample on OS X (10.13.6) with Python 3.6.2: - quit() in the console yields Exc 0: closed is True Python 3.6.2 (default, Apr 17 2018, 12:29:33) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> Traceback (most recent call last): File "./test.py", line 6, in code.InteractiveConsole().interact() File "/Users/zunger/.pyenv/versions/3.6.2/lib/python3.6/code.py", line 228, in interact line = self.raw_input(prompt) File "/Users/zunger/.pyenv/versions/3.6.2/lib/python3.6/code.py", line 275, in raw_input return input(prompt) ValueError: I/O operation on closed file. - ^D and raise SystemExit in console both yield: Python 3.6.2 (default, Apr 17 2018, 12:29:33) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> ^D now exiting InteractiveConsole... Try 0: closed is False Python 3.6.2 (default, Apr 17 2018, 12:29:33) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> ^D now exiting InteractiveConsole... Try 1: closed is False So it looks like the issue is specific to quit(), as you say. I'm not sure I understand what the purpose of the sys.stdin.close() call is, though: why would we *want* to alert the parent shell in some way beyond a SystemExit as the default (non-overrideable) behavior? This may be a documentation issue, but it seems extremely surprising to me; no other function does this additional thing, even when raising a SystemExit. I would think that any caller which is running InteractiveConsole.interact() and also wants to do something when it finishes would want to handle that additional logic itself. -- ___ Python tracker <https://bugs.python.org/issue34115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34115] code.InteractiveConsole.interact() closes stdin
Yonatan Zunger added the comment: Or perhaps in an alternate phrasing: The sys.stdin.close behavior makes sense if quit is being used inside IDLE, but is very surprising from the perspective of the `code` module. (Really, even the SystemExit is surprising there, and should be documented!) What if we added a stdin-preserving workaround to code.InteractiveConsole.interact, and added documentation of the SystemExit behavior to that module as well, but left _Quitter itself as-is? -- ___ Python tracker <https://bugs.python.org/issue34115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34115] code.InteractiveConsole.interact() closes stdin
Yonatan Zunger added the comment: Definitely agree about the difference. I'd say that either SystemExit or EOFError would be a reasonable thing for the interactive session to do, but the combination of closing stdin and SystemExit is really weird. Honestly, I would have just expected interact() to return like an ordinary function when it was done; the logic of "it's time to close the terminal window" feels like it belongs at a *much* higher level of the stack. On Fri, Jul 20, 2018 at 10:16 PM Terry J. Reedy wrote: > > Terry J. Reedy added the comment: > > There is an important difference between a program saying 'I am done > executing' and a user saying 'I am done with the interactive session'. > This is especially true in an IDE where 'session' can include many editing > and shell sessions. > > 'Stop executing' happens when execution reaches the end of the file, > which causes EOFError upon a read attempt. It can also be done gracefully > before the end of input with sys.exit(), which raises SystemExit. > > In Interactive Python, 'leave session' can be done with SystemExit or the > EOF control signal, which appears to raise EOFError. This suggests that > quit() and exit(), which were added because newbies did not know the proper > way to exit, should raise EOFError rather than SystemExit. The fact that > 'quit' displays 'Use quit() or Ctrl-Z plus Return to exit' (EOF on Windows) > suggests the same. But I need to experiment (another day). > > -- > > ___ > Python tracker > <https://bugs.python.org/issue34115> > ___ > -- ___ Python tracker <https://bugs.python.org/issue34115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34115] code.InteractiveConsole.interact() closes stdin
Yonatan Zunger added the comment: Eryk: Thanks for finding that! So that I'm sure I understand, if 34187 is resolved, does that mean the stdin.close() is no longer required at all in _Quitter? On Sun, Jul 22, 2018 at 2:12 AM Eryk Sun wrote: > > Eryk Sun added the comment: > > > On Windows Console, sys.stdin.close() does not prevent a second > > interact call. This might be considered a bug. > > This is a bug in io._WindowsConsoleIO. > > In Python 3, the sys.std* file objects that get created at startup use > closefd=False: > > >>> sys.stdin.buffer.raw.closefd > False > >>> sys.stdout.buffer.raw.closefd > False > >>> sys.stderr.buffer.raw.closefd > False > > Since the REPL uses C FILE streams (or in 3.6+ the underlying console file > handle in Windows), closing sys.stdin does not cause the REPL to exit, and > the PyOS_ReadLine call in the interactive loop continues to work in both > POSIX and Windows. > > That said, closing sys.stdin should cause input() to raise ValueError due > to sys.stdin.fileno() failing (i.e. take the non-tty path) and subsequently > sys.stdin.readline() failing. A second call to > code.InteractiveConsole.interact() should thus fail. The issue is that the > fileno() method of _WindowsConsoleIO isn't raising ValueError like it > should when the file is closed and closefd is false. I've created issue > 34187 with a suggested fix. > > -- > nosy: +eryksun > > ___ > Python tracker > <https://bugs.python.org/issue34115> > ___ > -- ___ Python tracker <https://bugs.python.org/issue34115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
