[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8
I'm in favor of adding a public C API to get and set the frame evaluation function. API excluded from the limited API. I wrote https://github.com/python/cpython/pull/17340 to add such API to Python 3.8.1. For me, it's a Python 3.8.0 regression. Victor ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MCHIISI7A4THO23SSXYRJOFNZQCHTBR4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8
Hi, On 21/11/2019 8:04 pm, Brett Cannon wrote: An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but there seems to be a stalemate in the issue. A key question is at what API level should setting the frame evaluation function live at? No one is suggesting the stable ABI, but there isn't agreement between CPython or the internal API (there's also seems to be a suggestion to potentially remove PEP 523 support entirely). And regardless of either, there also seems to be a disagreement about providing getters and setters to continue to try and hide PyInterpreterState regardless of which API level support is provided at (if any). If you have an opinion please weight in on the issue. I have opinions :) I am opposed to adding any new API to modify the frame evaluation function (the interpreter). Although I would like to remove PEP 523, for now I propose that we do absolutely nothing. The pydev/pycharm debugger can be easily modified to use the internal data structure, so there is no urgency to do anything. Longer term I would like to remove PEP 523. I will write a PEP for this, but in the meantime let me explain why. First of all, PEP 523 doesn't explain who gets priority if two modules want to change the interpreter. Who wins? Or does it just crash? In case that seems unlikely, consider that PyCharm and PyDev already (mis)use PEP 523 for debugging and that cProfile could be sped up by using PEP 523 (please don't). What happens if you want to debug the profiled code? PEP 523 is not a good fit for debugging. --- Because using sys.settrace is so slow and there is no API specifically for debugging, the PyDev debugger has little choice but to use PEP 523. Debuggers using this approach need to directly modify bytecode, which is fragile. We need a good API for debuggers and other tools. PEP 523 isn't great for "JIT"s either. - (I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer" would be a better term, IMO) A frame is a poor region for optimization, any effective optimizer would only use PEP 523 as a starting hook to allow it to monitor execution and select better regions for optimization. It is however, fine for experimental purposes. I have no problem keeping until we have evidence of working "JIT". As mentioned above, any such optimizer would also interact very poorly with a PEP 523 based debugger. PEP 523 prevents some optimizations --- Making Python-to-Python calls without a C-level call is prohibited by PEP 523, but would allow faster to Python-to-Python calls and faster switching between stacks of coroutines. Alternative API for tools based loosely on the JVM -- The JVM does not offer any means to modify the evaluation engine, but it does provide a way for tools to interact with the JVM. JVM debuggers support watchpoints, but Python debuggers do not. JVM debuggers can interact with the JVM while the JVM runs at (near) full speed, whereas Pdb causes the CPython interpreter to slow down a lot. We can gain these advantages by adding a tooling API to (C)Python. A tooling API would support both debuggers and profilers by providing the following features: * Breakpoints * Watchpoints (both local and global variables) * Callbacks for events (call/return/exception/entry/exit/line). Cheers, Mark. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4CMAHUCPF2J3F7H75KFISZENASSBM6RE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2019-11-15 - 2019-11-22) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7171 (+23) closed 43475 (+57) total 50646 (+80) Open issues with patches: 2826 Issues opened (61) == #25172: Unix-only crypt should not be present on Windows. https://bugs.python.org/issue25172 reopened by xdegaye #32371: Delay-loading of python dll is impossible when using some C ma https://bugs.python.org/issue32371 reopened by Pierre Chatelier #37957: Allow regrtest to receive a file with test (and subtests) to i https://bugs.python.org/issue37957 reopened by pablogsal #38817: Immutable types inplace operations work incorrect in async https://bugs.python.org/issue38817 opened by bryzgaloff #38818: Modify PyInterpreterState.eval_frame to pass tstate (PyThreadS https://bugs.python.org/issue38818 opened by vstinner #38820: Make Python compatible with OpenSSL 3.0.0 https://bugs.python.org/issue38820 opened by christian.heimes #38822: Inconsistent os.stat behavior for directory with Access Denied https://bugs.python.org/issue38822 opened by CrouZ #38823: Improve stdlib module initialization error handling. https://bugs.python.org/issue38823 opened by brandtbucher #38825: shutil.disk_usage - Lacking documentation https://bugs.python.org/issue38825 opened by Lord Anton Hvornum #38826: Regular Expression Denial of Service in urllib.request.Abstrac https://bugs.python.org/issue38826 opened by bc #38828: http.cookiejar handle cookie.version to be None https://bugs.python.org/issue38828 opened by kovid #38829: Make the function flush_io accessible in the C-API https://bugs.python.org/issue38829 opened by Jean-Didier #38832: setup.py can report builtin modules as missing modules https://bugs.python.org/issue38832 opened by christian.heimes #38833: Issue with multiprocessing.Pool & multiprocessing.Queue https://bugs.python.org/issue38833 opened by bigbizze #38834: TypedDict: no way to tell which (if any) keys are optional at https://bugs.python.org/issue38834 opened by Zac Hatfield-Dodds #38836: Links are duplicated in documentation search result https://bugs.python.org/issue38836 opened by MaT1g3R #38840: incorrect __all__ list in multiprocessing.managers module https://bugs.python.org/issue38840 opened by xdegaye #38842: test_multiprocessing_spawn altered the execution environment i https://bugs.python.org/issue38842 opened by pablogsal #38843: Document argparse behaviour when custom namespace object alrea https://bugs.python.org/issue38843 opened by zerkms #38845: test_shared_memory_SharedMemoryServer_ignores_sigint and other https://bugs.python.org/issue38845 opened by tlecarrour #38846: async: Return context manager from open(_unix)_connection https://bugs.python.org/issue38846 opened by srittau #38848: compileall fails when the platform lacks a functional sem_ope https://bugs.python.org/issue38848 opened by xdegaye #38849: test_timestamp_naive fails on android https://bugs.python.org/issue38849 opened by xdegaye #38850: test_largefile fails on android https://bugs.python.org/issue38850 opened by xdegaye #38851: UDPLITE tests fail on android https://bugs.python.org/issue38851 opened by xdegaye #38852: test_recursion_limit in test_threading crashes with SIGSEGV on https://bugs.python.org/issue38852 opened by xdegaye #38854: Decorator breaks inspect.getsource https://bugs.python.org/issue38854 opened by crusaderky #38855: test_unpack.py does not catch the unpacking of a set https://bugs.python.org/issue38855 opened by Ylem #38856: wait_closed() can raise ConnectionResetError https://bugs.python.org/issue38856 opened by yselivanov #38858: new_interpreter() should reuse more Py_InitializeFromConfig() https://bugs.python.org/issue38858 opened by vstinner #38860: GenericPyCData_new does not invoke new or init https://bugs.python.org/issue38860 opened by b1tninja #38861: zipfile: Corrupts filenames containing non-UTF8 characters https://bugs.python.org/issue38861 opened by jgoerzen #38862: IDLE: Include end newlines in whitespace fix. https://bugs.python.org/issue38862 opened by terry.reedy #38864: dbm: Can't open database with bytes-encoded filename https://bugs.python.org/issue38864 opened by jgoerzen #38865: Can Py_Finalize() be called if the current interpreter is not https://bugs.python.org/issue38865 opened by vstinner #38866: test_pyclbr replace asyncore https://bugs.python.org/issue38866 opened by jacksonriley #38867: Enable Tkinter on Windows ARM https://bugs.python.org/issue38867 opened by steve.dower #38868: Shutil cannot delete a folder that contains an .ini file https://bugs.python.org/issue38868 opened by QueenSvetlana #38870: Expose ast.unparse in the ast module https://bugs.python.org/issue38870 opened by pablogsal #38871: lib2to3 generates invalid code with filter and
[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8
I agree with Mark on "for now I propose that we do absolutely nothing". (I'll wait on a PEP for the rest of his points.) The capability of PEP 523 (swapping in a different PyEval_EvalFrame() impl.) is deep in the CPython runtime functionality. It is low-level, highly impactful, and there-be-dragons. So in my mind it makes perfect sense to keep it "internal", which was an unintended (but not necessarily incorrect) side effect of making PyInterpreterState opaque. Nothing says "don't use this unless you know what you are doing" better than requiring that extensions define Py_BUILD_CORE_MODULE (or Py_BUILD_CORE or whatever) if they want to use it. Accessor functions seem like overkill in that case since they would require the same Py_BUILD_CORE_MODULE that the PyInterpreterState field now requires. -eric On Thu, Nov 21, 2019 at 1:06 PM Brett Cannon wrote: > > An unfortunate side-effect of making PyInterpreterState in Python 3.8 opaque > is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) support. > https://www.python.org/dev/peps/pep-0523/ was opened to try and fix this, but > there seems to be a stalemate in the issue. > > A key question is at what API level should setting the frame evaluation > function live at? No one is suggesting the stable ABI, but there isn't > agreement between CPython or the internal API (there's also seems to be a > suggestion to potentially remove PEP 523 support entirely). > > And regardless of either, there also seems to be a disagreement about > providing getters and setters to continue to try and hide PyInterpreterState > regardless of which API level support is provided at (if any). > > If you have an opinion please weight in on the issue. > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL3NHRAGN5WAMJC4IHAHEXF3QF/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QCHNEWGBXHA2IQX32F2NAQEWKAXWZQNY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Restricted Entry Point from PEP-551/578
I did a quick hack up of letting configs control what command line options were available. I'm not sure y'all wanted it but here it is. I'm happy to take suggestions including tossing the whole thing and chalking it up to experience. It's lightly tested with the syslog implementation from the spython repo. https://github.com/python/cpython/compare/master...jsnklln:cmdline_options_controled_by_config On Thu, Nov 21, 2019 at 9:36 PM Terry Reedy wrote: > On 11/21/2019 4:46 PM, Steve Dower wrote: > > (though some won't be raised until 3.8.1... we should probably mark > > those, or at least update that page to warn that events may have been > > added over time). > > I included this in a new audit doc issue. > https://bugs.python.org/issue38892 > > -- > Terry Jan Reedy > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/OBHSBXUMM6EML35MY76QPAP7PQVNEHBY/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Jason Killen jsnk...@gmail.com Pain is inevitable, misery is optional. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/XMJWSYDFOLGCIMIODRXJQ47G42B55TN5/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8
I modified "make install" to install internal header files, so it's possible to use the internal C API in debuggers and profilers. For example, to be able to inspect Python internals without having to call functions (which might modify the Python internal state). I made this change to be able to move more APIs to the internal API, to reduce the size of the public C API. If it wouldn't be possible to use the internal C API, it would be more risky to move things into the internal API. We might break legit use cases with no solution for users. Here PyCharm (for example) could be modified to use the internal C API, no? Define Py_BUILD_CORE_MODULE macro and include "internal/pycore_pystate.h". Victor Le ven. 22 nov. 2019 à 20:36, Eric Snow a écrit : > > I agree with Mark on "for now I propose that we do absolutely > nothing". (I'll wait on a PEP for the rest of his points.) The > capability of PEP 523 (swapping in a different PyEval_EvalFrame() > impl.) is deep in the CPython runtime functionality. It is low-level, > highly impactful, and there-be-dragons. > > So in my mind it makes perfect sense to keep it "internal", which was > an unintended (but not necessarily incorrect) side effect of making > PyInterpreterState opaque. Nothing says "don't use this unless you > know what you are doing" better than requiring that extensions define > Py_BUILD_CORE_MODULE (or Py_BUILD_CORE or whatever) if they want to > use it. > > Accessor functions seem like overkill in that case since they would > require the same Py_BUILD_CORE_MODULE that the PyInterpreterState > field now requires. > > -eric > > On Thu, Nov 21, 2019 at 1:06 PM Brett Cannon wrote: > > > > An unfortunate side-effect of making PyInterpreterState in Python 3.8 > > opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) > > support. https://www.python.org/dev/peps/pep-0523/ was opened to try and > > fix this, but there seems to be a stalemate in the issue. > > > > A key question is at what API level should setting the frame evaluation > > function live at? No one is suggesting the stable ABI, but there isn't > > agreement between CPython or the internal API (there's also seems to be a > > suggestion to potentially remove PEP 523 support entirely). > > > > And regardless of either, there also seems to be a disagreement about > > providing getters and setters to continue to try and hide > > PyInterpreterState regardless of which API level support is provided at (if > > any). > > > > If you have an opinion please weight in on the issue. > > ___ > > Python-Dev mailing list -- python-dev@python.org > > To unsubscribe send an email to python-dev-le...@python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > > https://mail.python.org/archives/list/python-dev@python.org/message/4UZJYAZL3NHRAGN5WAMJC4IHAHEXF3QF/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/QCHNEWGBXHA2IQX32F2NAQEWKAXWZQNY/ > Code of Conduct: http://python.org/psf/codeofconduct/ -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/V5HLZZDDDHM3E5YXFBDIFFGN2FO5EAFS/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Stalemate on bringing back PEP 523 support into Python 3.8
On Fri, Nov 22, 2019 at 10:10 AM Mark Shannon wrote: > Hi, > > On 21/11/2019 8:04 pm, Brett Cannon wrote: > > An unfortunate side-effect of making PyInterpreterState in Python 3.8 > opaque is it removed [PEP 523](https://www.python.org/dev/peps/pep-0523/) > support. https://www.python.org/dev/peps/pep-0523/ was opened to try and > fix this, but there seems to be a stalemate in the issue. > > > > A key question is at what API level should setting the frame evaluation > function live at? No one is suggesting the stable ABI, but there isn't > agreement between CPython or the internal API (there's also seems to be a > suggestion to potentially remove PEP 523 support entirely). > > > > And regardless of either, there also seems to be a disagreement about > providing getters and setters to continue to try and hide > PyInterpreterState regardless of which API level support is provided at (if > any). > > > > If you have an opinion please weight in on the issue. > > I have opinions :) > > I am opposed to adding any new API to modify the frame evaluation > function (the interpreter). Although I would like to remove PEP 523, for > now I propose that we do absolutely nothing. > The pydev/pycharm debugger can be easily modified to use the internal > data structure, so there is no urgency to do anything. > > > > Longer term I would like to remove PEP 523. I will write a PEP for this, > but in the meantime let me explain why. > > First of all, PEP 523 doesn't explain who gets priority if two modules > want to change the interpreter. Who wins? Or does it just crash? > In case that seems unlikely, consider that PyCharm and PyDev already > (mis)use PEP 523 for debugging and that cProfile could be sped up by > using PEP 523 (please don't). What happens if you want to debug the > profiled code? > It is a process wide global for use in special circumstances. Those two examples are debuggers. Others are not. But as a global it should be expected that it isn't for general application level use. Only one user per process. Or else the old manual chaining rule applies. If you find it set when you are setting your own, you need to call the previously set one. and when unsetting, if it isn't set to yours, you should leave it set and turn your own into a no-op. Realistically these aren't problems and most people things aren't going to use the API. I don't think anyone has ever even encountered a combination of uses at once situation yet given the complexity and obscurity. At Google we use it to provide C/C++ level sampling profilers access to the Python level program state by plumbing all Python calls through a unique C function per Python callable such that a C level stack trace symbolizes to information directly related to which Python function was called. So that light weight sampling profilers can get an accurate picture at the function level of where time is being spent in a process executing Python code. Without heavy handed live running scary signal based interpreter state internal introspections (ie: vmprof). It's a cool hack to get Python functions to show up on the native C stack by proxy. No, I don't expect others to have implemented that (it requires a build step that creates C stubs for all python callables, links those in and loads it all up - not something most people have a way to do) [yes I call this a hack, so no I don't block anything on it - i'm just pointing out that there are many creative uses of being able to inject oneself as a C function call within a Python call stack whether we ultimately want that to be a thing or not] Another use was made that recorded live types of functions at runtime to be fed into type analyzers and ultimately feed back into adding practical annotations on existing code. your event callback approach would work for that case. > > PEP 523 is not a good fit for debugging. > --- > Because using sys.settrace is so slow and there is no API specifically > for debugging, the PyDev debugger has little choice but to use PEP 523. > Debuggers using this approach need to directly modify bytecode, which is > fragile. > We need a good API for debuggers and other tools. > > > PEP 523 isn't great for "JIT"s either. > - > (I use "JIT" as it is the term used in PEP 523. "Adaptive optimizer" > would be a better term, IMO) > A frame is a poor region for optimization, any effective optimizer would > only use PEP 523 as a starting hook to allow it to monitor execution and > select better regions for optimization. > It is however, fine for experimental purposes. I have no problem keeping > until we have evidence of working "JIT". > Does evidence require an open source release? An existing JIT implementation *was* mentioned in the PEP... As mentioned above, any such optimizer would also interact very poorly > with a PEP 523 based debugger. > > > PEP 523 prevents some optimizations > ---
