Re: [Python-Dev] Is there a reference manual for Python bytecode?
On 12/26/15 10:19 PM, Nick Coghlan wrote: On 27 December 2015 at 12:23, Guido van Rossum wrote: On Sat, Dec 26, 2015 at 6:06 PM, Brett Cannon wrote: Ned also neglected to mention his byterun project which is a pure Python implementation of the CPython eval loop: https://github.com/nedbat/byterun From the commit log it looks like it's a co-production between Ned and Allison Kaptur (who gave the talk I mentioned). Yes, Allison was very helpful in pushing it forward. And I should also mention that I started with a dormant project that Paul Swartz wrote. And: it doesn't work completely. There are things it doesn't handle properly, and I turned to other projects some time ago. If someone wants to pick it up, that would be cool. It occurred to me that "byterun" would make a good see-also link from the dis module docs, and looking into that idea brought me to this article Allison wrote about it for the "500 lines" project: http://aosabook.org/en/500L/a-python-interpreter-written-in-python.html For a detailed semantic reference, byterun's eval loop is likely one of the most readable sources of information: https://github.com/nedbat/byterun/blob/master/byterun/pyvm2.py I started working on byterun because I knew I didn't understand bytecode well enough to implement branch coverage measurement in coverage.py. Its primary goal was teaching (me!) how the bytecode worked. Recently though, I've started a new implementation of branch coverage based on the ast rather than the bytecode. This was prompted by the "async" keywords in 3.5. "async for" and "for" compile very differently to bytecode, which caused headaches for a bytecode-based understanding of flow, so I'm trying out an ast-based understanding. --Ned. In terms of formal documentation, the main problem with providing reference bytecode tables is keeping them up to date as the eval loop changes. However, it would theoretically be possible to create a custom Sphinx directive that uses the dis module to generate the tables automatically during the docs build process, rather than maintaining them by hand - something like that could be experimented with outside CPython, and potentially incorporated into the dis module docs if folks are able to figure out something that works well. Regards, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is there a reference manual for Python bytecode?
Thanks for your help so far (I'm experimenting with the peephole optimizer - hence my question before as I was trying to work out how to know what the small integer hard-coded offsets should be when looking ahead in the bytecode). I've successfully added a new opcode (generated by the optimizer and understood by the interpreter loop) but when adding a second I unexpectedly got the following error. I'm not doing anything different to what I did with the first opcode as far as I can tell (I have a TARGET(FOO) in ceval.c and have obviously defined the new opcode's value in opcode.h). """ ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi XXX lineno: 241, opcode: 1 Fatal Python error: Py_Initialize: can't import _frozen_importlib Traceback (most recent call last): File "", line 698, in File "", line 751, in BuiltinImporter File "", line 241, in _requires_builtin SystemError: unknown opcode Aborted (core dumped) generate-posix-vars failed make: *** [pybuilddir.txt] Error 1 """ If I #ifdef out the code in peephole.c which generates my new (2nd) opcode, then the error does not occur. I tried a "make clean" first, but that didn't help (I realise that does not necessarily rule out a makefile dependency issue). Does anyone know if this is a well-known symptom of forgetting to add something somewhere when adding a new opcode, or do I need to track it down some more myself? I did not have this problem when introducing my first new opcode. Thanks, E. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is there a reference manual for Python bytecode?
Can you show the diffs you have so far? Somebody's got to look at your code. --Guido (mobile) On Dec 27, 2015 16:51, "Erik" wrote: > Thanks for your help so far (I'm experimenting with the peephole optimizer > - hence my question before as I was trying to work out how to know what the > small integer hard-coded offsets should be when looking ahead in the > bytecode). > > > I've successfully added a new opcode (generated by the optimizer and > understood by the interpreter loop) but when adding a second I unexpectedly > got the following error. I'm not doing anything different to what I did > with the first opcode as far as I can tell (I have a TARGET(FOO) in ceval.c > and have obviously defined the new opcode's value in opcode.h). > > > """ > ./python -E -S -m sysconfig --generate-posix-vars ;\ > if test $? -ne 0 ; then \ > echo "generate-posix-vars failed" ; \ > rm -f ./pybuilddir.txt ; \ > exit 1 ; \ > fi > XXX lineno: 241, opcode: 1 > Fatal Python error: Py_Initialize: can't import _frozen_importlib > Traceback (most recent call last): > File "", line 698, in > File "", line 751, in BuiltinImporter > File "", line 241, in _requires_builtin > SystemError: unknown opcode > Aborted (core dumped) > generate-posix-vars failed > make: *** [pybuilddir.txt] Error 1 > """ > > If I #ifdef out the code in peephole.c which generates my new (2nd) > opcode, then the error does not occur. I tried a "make clean" first, but > that didn't help (I realise that does not necessarily rule out a makefile > dependency issue). > > Does anyone know if this is a well-known symptom of forgetting to add > something somewhere when adding a new opcode, or do I need to track it down > some more myself? I did not have this problem when introducing my first new > opcode. > > Thanks, E. > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is there a reference manual for Python bytecode?
You can look at https://docs.python.org/devguide/compiler.html to see if you missed something. As for the _frozen_importlib problem, that typically manifests itself when you have invalid bytecode (that module is frozen bytecode that gets compiled into the interpreter and is the first bit of Python code that gets run). On Sun, 27 Dec 2015, 16:41 Guido van Rossum wrote: > Can you show the diffs you have so far? Somebody's got to look at your > code. > > --Guido (mobile) > On Dec 27, 2015 16:51, "Erik" wrote: > >> Thanks for your help so far (I'm experimenting with the peephole >> optimizer - hence my question before as I was trying to work out how to >> know what the small integer hard-coded offsets should be when looking ahead >> in the bytecode). >> >> >> I've successfully added a new opcode (generated by the optimizer and >> understood by the interpreter loop) but when adding a second I unexpectedly >> got the following error. I'm not doing anything different to what I did >> with the first opcode as far as I can tell (I have a TARGET(FOO) in ceval.c >> and have obviously defined the new opcode's value in opcode.h). >> >> >> """ >> ./python -E -S -m sysconfig --generate-posix-vars ;\ >> if test $? -ne 0 ; then \ >> echo "generate-posix-vars failed" ; \ >> rm -f ./pybuilddir.txt ; \ >> exit 1 ; \ >> fi >> XXX lineno: 241, opcode: 1 >> Fatal Python error: Py_Initialize: can't import _frozen_importlib >> Traceback (most recent call last): >> File "", line 698, in >> File "", line 751, in BuiltinImporter >> File "", line 241, in _requires_builtin >> SystemError: unknown opcode >> Aborted (core dumped) >> generate-posix-vars failed >> make: *** [pybuilddir.txt] Error 1 >> """ >> >> If I #ifdef out the code in peephole.c which generates my new (2nd) >> opcode, then the error does not occur. I tried a "make clean" first, but >> that didn't help (I realise that does not necessarily rule out a makefile >> dependency issue). >> >> Does anyone know if this is a well-known symptom of forgetting to add >> something somewhere when adding a new opcode, or do I need to track it down >> some more myself? I did not have this problem when introducing my first new >> opcode. >> >> Thanks, E. >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is there a reference manual for Python bytecode?
On 28/12/15 00:41, Guido van Rossum wrote: Can you show the diffs you have so far? Somebody's got to look at your code. Sounds like it's not a well-known symptom then. I agree, but that Somebody should be me (initially, at least) - I don't want to waste other people's time if I made a silly mistake. I'm happy to post my diffs once I'm done (if only to document that what I tried is not worth spending time on). E. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
