[Python-Dev] Modifying Grammar/grammar and other foul acts
Python-devs,
I'm writing to you for some help in understanding the Python grammar. As an
excuse to deep dive into Python's tokenizer / grammar, I decided (as a
hideous, hideous joke) to want to allow braces where colons are allowed (as
flow control).
Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer
As a first example:
funcdef: ('def' NAME parameters ['->' test] ':' suite |
'def' NAME parameters ['->' test] '{' suite '}' )
I reran Parser/pgen and the dfa changes, but python (3.1) when recompiled,
throws errors on things like:
def a() { None }
Strangely enough:
lambdef: ( 'lambda' [varargslist] ':' test |
'lambda' [varargslist] '{' test '}' )
works fine! I this simplely some difference between "test" and "suite".
I have tried tackling this with gdb, looking at err_input clearly isn't
enough.
(gdb) break err_input
(gdb) break PyParser_ASTFromString
import sys
b = compile("def a() {pass}","sys.stdout","single")
# yet a simple grammar fix is enough for this!
c = compile("lambda x {None}","sys.stdout","single")
I'm in over my head!
Any insights / help would be appreciated. Full-on flaming is also
appropriate, but would be less appreciated.
Specific questions
1.) I assume the Grammar/grammar is read top to bottom. Confirm?
2.) More help figuring out how to debug what python *thinks* it's seeing
when it see "def a() {pass}". It's not getting to the ast construction
stage, as near as I can tell. What additional breakpoints can I set to see
where it's failing.
Gregg L.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Modifying Grammar/grammar and other foul acts
Sorry, re: question one, forgive the ill-formed question. I meant more, are
the parser rules applied "first matching". Essentially trying to confirm
that the parser is "top down" or "bottom up" or whether or not it even
matters.
Thanks for the tip -- it seems to be exactly what I want. To make it
explicit, this seems to be fuller (unix) recipe for how to make this style
of debugging happen.
$ ./configure --with-pydebug
$ make
$ set PYTHONDEBUG=1
$ ./python -d # then this shows the parsing info
On Sat, Mar 6, 2010 at 10:56 AM, "Martin v. Löwis" wrote:
> > 1.) I assume the Grammar/grammar is read top to bottom. Confirm?
>
> Confirm - but this is not surprising: *any* source file is typically
> read from top to bottom. Randoma access reading is typically done for
> binary files, only.
>
> So you must be asking something else, but I can't guess what that might be.
>
> > 2.) More help figuring out how to debug what python *thinks* it's
> > seeing when it see "def a() {pass}". It's not getting to the ast
> > construction stage, as near as I can tell. What additional breakpoints
> > can I set to see where it's failing.
>
> Enable the D() debugging in parser.c (configure with --with-pydebug,
> and set PYTHONDEBUG).
>
> Regards,
> Martin
>
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Modifying Grammar/grammar and other foul acts
Thank you for the advice everyone. This seed has finally born (rotten)
fruit at:
http://writeonly.wordpress.com/2010/04/01/whython-python-for-people-who-hate-whitespace/
http://bitbucket.org/gregglind/python-whython3k/
On Fri, Mar 12, 2010 at 4:13 AM, Georg Brandl wrote:
> Am 09.03.2010 14:42, schrieb Jeremy Hylton:
> > On Sat, Mar 6, 2010 at 11:27 AM, Gregg Lind
> wrote:
> >> Python-devs,
> >>
> >> I'm writing to you for some help in understanding the Python grammar.
> As an
> >> excuse to deep dive into Python's tokenizer / grammar, I decided (as a
> >> hideous, hideous joke) to want to allow braces where colons are allowed
> (as
> >> flow control).
> >>
> >> Starting from PEP 306 (and branch r311), I hacked on Grammar/Grammer
> >>
> >> As a first example:
> >>
> >> funcdef: ('def' NAME parameters ['->' test] ':' suite |
> >> 'def' NAME parameters ['->' test] '{' suite '}' )
> >>
> >> I reran Parser/pgen and the dfa changes, but python (3.1) when
> recompiled,
> >> throws errors on things like:
> >>
> >> def a() { None }
> >>
> >> Strangely enough:
> >>
> >> lambdef: ( 'lambda' [varargslist] ':' test |
> >>'lambda' [varargslist] '{' test '}' )
> >>
> >> works fine! I this simplely some difference between "test" and "suite".
> >>
> >> I have tried tackling this with gdb, looking at err_input clearly isn't
> >> enough.
> >>
> >> (gdb) break err_input
> >> (gdb) break PyParser_ASTFromString
> >> import sys
> >> b = compile("def a() {pass}","sys.stdout","single")
> >> # yet a simple grammar fix is enough for this!
> >> c = compile("lambda x {None}","sys.stdout","single")
> >>
> >> I'm in over my head!
> >
> > You don't say what errors occur when you try to compile strings in
> > your new language. You may have changed the Grammar, which allows you
> > to tokenize the input. That isn't enough to get the input to compile.
> > You also need to change the compiler to understand the new tokens.
>
> In particular, many AST creation functions check for specific counts of
> children on many nodes. I haven't checked, but in the case of the
> "funcdef" rule, it may check for either 7 or 5 children to determine
> whether the optional return annotation ['->' test] is present.
>
> Georg
>
>
> --
> Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
> Four shall be the number of spaces thou shalt indent, and the number of thy
> indenting shall be four. Eight shalt thou not indent, nor either indent
> thou
> two, excepting that thou then proceed to four. Tabs are right out.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/gregg.lind%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
