[Python-Dev] Re: Regressions caused the recent work on ceval.c and frame objects

[dw-git]

> Will all packages that use Cython have to upgrade Cython to work with 3.10?

For this particular issue you'll only have to upgrade if you use "profiling" (I 
doubt that many packages routinely build with Cython profiling turned on). 
However, it's possible there are other 3.10 bugfixes in Cython - I'm not 
completely sure. I think the easiest way to know is to try to build a package 
for 3.10.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/52VFBHR7AHTXPLC434E4BPXNXVUU3SVF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Regressions caused the recent work on ceval.c and frame objects

[Miro Hrončok]

On 20. 09. 21 7:40, Christopher Barker wrote:

Will all packages that use Cython have to upgrade Cython to work with 3.10?


I am not sure if all, but many will do.

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MSKPT6P3JNVVDXLWXTVSB2B33DLNW6KA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] f-strings in the grammar

[Pablo Galindo Salgado]

Hi,

I have started a project to move the parsing off-strings to the parser and
the grammar. Appart
from some maintenance improvements (we can drop a considerable amount of
hand-written code),
there are some interesting things we **could** (emphasis on could) get out
of this and I wanted
to discuss what people think about them.

* The parser will likely have "\n" characters and backslashes in f-strings
expressions, which currently is impossible:

>>> f"blah blah {'\n'} blah"
  File "", line 1
f"blah blah {'\n'} blah"
^
SyntaxError: f-string expression part cannot include a backslash

* The parser will allow nesting quote characters. This means that we
**could** allow reusing the same quote type in nested expressions
like this:

f"some text { my_dict["string1"] } more text"

* The parser will naturally allow more control over error messages and AST
positions.

* The **grammar** of the f-string will be fully specified without
ambiguity. Currently, the "grammar" that we have in the docs
(
https://docs.python.org/3/reference/lexical_analysis.html#formatted-string-literals)
is not really formal grammar because
not only is mixing lexing details with grammar details (the definition of "
literal_char") but also is not compatible with the current python
lexing schema (for instance, it recognizes "{{" as its own token, which the
language doesn't allow because something like "{{a:b}:c}"
is tokenized as "{", "{", "a" ... not as "{{", "a". Adding a formal grammar
could help syntax highlighters, IDEs, parsers and other tools
to make sure they properly recognize everything that there is.

There may be some other advantages that we have not explored still.

The work is at a point where the main idea works (all the grammar is
already there and working), but we need to make sure that all existing
errors and specifics are properly ported to the new code, which is a
considerable amount of work still so I wanted to make sure we are on the
same page before we decide to invest more time on this (Batuhan is helping
me with this and Lyssandros will likely join us). We are doing
this work in this branch:
https://github.com/we-like-parsers/cpython/blob/fstring-grammar

Tell me what you think.

P.S. If you are interested to help with this project, please reach out to
me. If we decide to go ahead we can use your help! :)

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/54N3MOYVBDSJQZTU6MTCPLUPIFSDN5IS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Regressions caused the recent work on ceval.c and frame objects

[Inada Naoki]

msgpack 1.0.2 sdist includes c++ code generated by Cython 0.29.21
released at 2020-07-09.
Python 3.10 can build it without any source modifications.
So I don't expect Python 3.10 requires re-generation for most Cython users.

Regards,

On Mon, Sep 20, 2021 at 4:11 PM  wrote:
>
> > Will all packages that use Cython have to upgrade Cython to work with 3.10?
>
> For this particular issue you'll only have to upgrade if you use "profiling" 
> (I doubt that many packages routinely build with Cython profiling turned on). 
> However, it's possible there are other 3.10 bugfixes in Cython - I'm not 
> completely sure. I think the easiest way to know is to try to build a package 
> for 3.10.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/52VFBHR7AHTXPLC434E4BPXNXVUU3SVF/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7MLTC3NK535YOQAKD6VQTT4QZIREG57N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Serhiy Storchaka]

20.09.21 14:18, Pablo Galindo Salgado пише:
> * The parser will likely have "\n" characters and backslashes in
> f-strings expressions, which currently is impossible:

What about characters "\x7b", "\x7d", "\x5c", etc?

What about newlines in single quotes? Currently this works:

f'''{1 +
2}'''

But this does not:

f'{1 +
2}'

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IJEJ5UVVKHEH6QGXZ3LONZPAITNMBULL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Pablo Galindo Salgado]

>
> What about characters "\x7b", "\x7d", "\x5c", etc?
> What about newlines in single quotes? Currently this works:


This is from the current branch:

>>> f"ble { '\x7b' }"
'ble {'

>>> f"{1 +
...  2}"
'3'

>>> f'{1 +
...  2}'
'3'

On Mon, 20 Sept 2021 at 13:52, Serhiy Storchaka  wrote:

> 20.09.21 14:18, Pablo Galindo Salgado пише:
> > * The parser will likely have "\n" characters and backslashes in
> > f-strings expressions, which currently is impossible:
>
> What about characters "\x7b", "\x7d", "\x5c", etc?
>
> What about newlines in single quotes? Currently this works:
>
> f'''{1 +
> 2}'''
>
> But this does not:
>
> f'{1 +
> 2}'
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IJEJ5UVVKHEH6QGXZ3LONZPAITNMBULL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2A4PFQKX2URNSKM5S6PRZX4YZTPWS3KT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Terry Reedy]

On 9/20/2021 7:18 AM, Pablo Galindo Salgado wrote:

there are some interesting things we **could** (emphasis on could) get 
out of this and I wanted

to discuss what people think about them.

* The parser will allow nesting quote characters. This means that we 
**could** allow reusing the same quote type in nested expressions

like this:

f"some text { my_dict["string1"] } more text"


I believe that this will disable regex-based processing, such as syntax 
highlighters, as in IDLE.  I also think that it will be sometimes 
confusing to human readers.


--
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TWSJKE4KKSW7YD3OCHKGKJC52VUG6FY5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Terry Reedy]

On 9/20/2021 8:46 AM, Serhiy Storchaka wrote:

20.09.21 14:18, Pablo Galindo Salgado пише:

* The parser will likely have "\n" characters and backslashes in
f-strings expressions, which currently is impossible:


What about characters "\x7b", "\x7d", "\x5c", etc?

What about newlines in single quotes? Currently this works:

f'''{1 +
2}'''

But this does not:

f'{1 +
2}'


The later is an error with or without the 'f' prefix and I think that 
this should continue to be the case.



--
Terry Jan Reedy


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z2IQGYH77V72D7TEDMIFWOHTN4MHKIAB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Eric V. Smith]

On 9/20/2021 11:21 AM, Terry Reedy wrote:

On 9/20/2021 8:46 AM, Serhiy Storchaka wrote:

20.09.21 14:18, Pablo Galindo Salgado пише:

* The parser will likely have "\n" characters and backslashes in
f-strings expressions, which currently is impossible:


What about characters "\x7b", "\x7d", "\x5c", etc?

What about newlines in single quotes? Currently this works:

f'''{1 +
2}'''

But this does not:

f'{1 +
2}'


The later is an error with or without the 'f' prefix and I think that 
this should continue to be the case.


The thought is that anything that's within braces {} and is a valid 
expression should be allowed. Basically, the opening brace puts you in 
"parse expression" mode. Personally, I'd be okay with this particular 
change.


Eric

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QWCISJYDR6LYXOD4DAKUTA3EYV3XQQIM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Eric V. Smith]

On 9/20/2021 11:19 AM, Terry Reedy wrote:

On 9/20/2021 7:18 AM, Pablo Galindo Salgado wrote:

there are some interesting things we **could** (emphasis on could) 
get out of this and I wanted

to discuss what people think about them.

* The parser will allow nesting quote characters. This means that we 
**could** allow reusing the same quote type in nested expressions

like this:

f"some text { my_dict["string1"] } more text"


I believe that this will disable regex-based processing, such as 
syntax highlighters, as in IDLE.  I also think that it will be 
sometimes confusing to human readers.


When I initially wrote f-strings, it was an explicit design goal to be 
just like existing strings, but with a new prefix. That's why there are 
all of the machinations in the parser for scanning within f-strings: the 
parser had already done its duty, so there needed to be a separate stage 
to decode inside the f-strings. Since they look just like regular 
strings, most tools could add the lowest possible level of support just 
by adding 'f' to existing prefixes they support: 'r', 'b', 'u'. The 
upside is that if you don't care about what's inside an f-string, your 
work is done.


I definitely share your concern about making f-strings more complicated 
to parse for tool vendors: basically all editors, alternative 
implementations, etc.: really anyone who parses python source code. But 
maybe we've already crossed this bridge with the PEG parser. Although I 
realize there's a difference between lexing and parsing. While the PEG 
parser just makes parsing more complicated, this change would make what 
was lexing into a more sophisticated parsing problem.


In 2018 or 2019 at PyCon in Cleveland I talked to several tool vendors. 
It's been so long ago that I don't remember who, but I'm pretty sure it 
was PyCharm and 2 or 3 other editors. All of them supported making this 
change, even understanding the complications it would cause them. I 
don't recall if I talked to anyone who maintains an alternative 
implementation, but we should probably discuss it with MicroPython, 
Cython, PyPy, etc., and understand where they stand on it.


In general I'm supportive of this change, because as Pablo points out 
there are definite benefits. But I think if we do accept it we should 
understand what sort of burden we're putting on tool and implementation 
authors. It would probably be a good idea to discuss it at the upcoming 
dev sprints.


Eric


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DN5HB7CBS7I2FXI74UBM4ZZVMSNVDQ57/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Thomas Grainger]

I don't think the python syntax should be beholden to syntax highlighting 
tools, eventually some syntax feature that PEG enables will require every 
parser or highlighter to switch to a similar or more powerful parse tool
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FI6QGN5GJJMVXOEM3VDZ7CAKIEUU2S4R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Guido van Rossum]

The current restrictions will also confuse some users (e.g. those used to
bash, and IIRC JS, where the rules are similar as what Pablo is proposing).

On Mon, Sep 20, 2021 at 8:24 AM Terry Reedy  wrote:

> On 9/20/2021 7:18 AM, Pablo Galindo Salgado wrote:
>
> > there are some interesting things we **could** (emphasis on could) get
> > out of this and I wanted
> > to discuss what people think about them.
> >
> > * The parser will allow nesting quote characters. This means that we
> > **could** allow reusing the same quote type in nested expressions
> > like this:
> >
> > f"some text { my_dict["string1"] } more text"
>
> I believe that this will disable regex-based processing, such as syntax
> highlighters, as in IDLE.  I also think that it will be sometimes
> confusing to human readers.
>
> --
> Terry Jan Reedy
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/TWSJKE4KKSW7YD3OCHKGKJC52VUG6FY5/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FAYGYTKQACEOXVDAT44CBW75PZ4MKLJZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Pablo Galindo Salgado]

Thanks a lot, Eric for your message! I actually share some of these worries
myself
and that's why I wanted to have a bigger conversation.

I wanted to also make clear that the change doesn't force us to do
*everything*. This means
that we can absolutely have some of the improvements but not others (for
example allowing
backslashes but not nesting). So is important to be clear that is not "all
or nothing". We just need to
decide what set of things in the design space we want :)

On Mon, 20 Sept 2021 at 16:52, Eric V. Smith  wrote:

> On 9/20/2021 11:19 AM, Terry Reedy wrote:
> > On 9/20/2021 7:18 AM, Pablo Galindo Salgado wrote:
> >
> >> there are some interesting things we **could** (emphasis on could)
> >> get out of this and I wanted
> >> to discuss what people think about them.
> >>
> >> * The parser will allow nesting quote characters. This means that we
> >> **could** allow reusing the same quote type in nested expressions
> >> like this:
> >>
> >> f"some text { my_dict["string1"] } more text"
> >
> > I believe that this will disable regex-based processing, such as
> > syntax highlighters, as in IDLE.  I also think that it will be
> > sometimes confusing to human readers.
>
> When I initially wrote f-strings, it was an explicit design goal to be
> just like existing strings, but with a new prefix. That's why there are
> all of the machinations in the parser for scanning within f-strings: the
> parser had already done its duty, so there needed to be a separate stage
> to decode inside the f-strings. Since they look just like regular
> strings, most tools could add the lowest possible level of support just
> by adding 'f' to existing prefixes they support: 'r', 'b', 'u'. The
> upside is that if you don't care about what's inside an f-string, your
> work is done.
>
> I definitely share your concern about making f-strings more complicated
> to parse for tool vendors: basically all editors, alternative
> implementations, etc.: really anyone who parses python source code. But
> maybe we've already crossed this bridge with the PEG parser. Although I
> realize there's a difference between lexing and parsing. While the PEG
> parser just makes parsing more complicated, this change would make what
> was lexing into a more sophisticated parsing problem.
>
> In 2018 or 2019 at PyCon in Cleveland I talked to several tool vendors.
> It's been so long ago that I don't remember who, but I'm pretty sure it
> was PyCharm and 2 or 3 other editors. All of them supported making this
> change, even understanding the complications it would cause them. I
> don't recall if I talked to anyone who maintains an alternative
> implementation, but we should probably discuss it with MicroPython,
> Cython, PyPy, etc., and understand where they stand on it.
>
> In general I'm supportive of this change, because as Pablo points out
> there are definite benefits. But I think if we do accept it we should
> understand what sort of burden we're putting on tool and implementation
> authors. It would probably be a good idea to discuss it at the upcoming
> dev sprints.
>
> Eric
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/DN5HB7CBS7I2FXI74UBM4ZZVMSNVDQ57/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YQU26OGHOTMJHCEO47YOUNEFGLWQ5DCG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Jeremiah Paige]

I just want to say that I am very excited to see where this goes. As an
author of a package that tries to recreate compiled f-strings at runtime,
they are a hard thing to generate given the current tools within Python.

On Mon, Sep 20, 2021 at 4:23 AM Pablo Galindo Salgado 
wrote:

>
> Tell me what you think.
>
> P.S. If you are interested to help with this project, please reach out to
> me. If we decide to go ahead we can use your help! :)
>

I don't know the CPython API very well, but if there is anything I can do
to help, I would be happy to assist.

Regards,
Jeremiah
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PXPNJHAOOQDR2Q7BAVGUJFNWOGVXBC5M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Terry Reedy]

On 9/20/2021 11:48 AM, Eric V. Smith wrote:


When I initially wrote f-strings, it was an explicit design goal to be 
just like existing strings, but with a new prefix. That's why there are 
all of the machinations in the parser for scanning within f-strings: the 
parser had already done its duty, so there needed to be a separate stage 
to decode inside the f-strings. Since they look just like regular 
strings, most tools could add the lowest possible level of support just 
by adding 'f' to existing prefixes they support: 'r', 'b', 'u'.


Which is what I did with IDLE.  Of course 'just add' was complicated by 
uppercase being allowed and 'f' being compatible with 'r' but not 'u' or 
'b'.


I definitely share your concern about making f-strings more complicated 
to parse for tool vendors: basically all editors, alternative 
implementations, etc.: really anyone who parses python source code. But 
maybe we've already crossed this bridge with the PEG parser.


I think we are on the far side of the bridge with contextual keywords. 
I don't believe the new code for highlighting the new match statement is 
exactly correct.  As I remember, properly classifying '_' in all the 
examples we created was too difficult, and maybe not possible.


Although I 
realize there's a difference between lexing and parsing. While the PEG 
parser just makes parsing more complicated, this change would make what 
was lexing into a more sophisticated parsing problem.


I have no love for the RE code.  I would try ast.parse if I was not sure 
it would be too slow.  I would be happy if a simplified and fast minimal 
lexer/parser were added for everyone to use.  It would not have to make 
exactly the same distinctions that IDLE currently does.


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SAYU6SMP4KT7G7AQ6WVQYUDOSZPKHJMS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Patrick Reader]

> The current restrictions will also confuse some users (e.g. those used to 
> bash, and IIRC JS, where the rules are similar as what Pablo is proposing).
> -- 
> --Guido van Rossum (python.org/~guido )

WRT the similar syntax in bash (and similar shells), there are two options:

"string `code` string"

"string $(code) string"

The latter, $(), allows fully-featured nesting in the way Pablo is suggesting:

"string $(code "string2 $(code2) string2" code) string"

The former, using backticks, does not allow nesting directly, but it allows 
extra backslashes inside the backticks to escape the nested ones, like this:

"string `code "string2 \`code2\` string2" code` string"

This can be nested infinitely using lots of backslashes. Is this worth 
considering as another option? It doesn't have the disadvantage of complicating 
lexing (as much), although nesting with backslashes is quite ugly. IMO nesting 
things in f-strings would be ugly anyway, so I don't think that would matter 
too much.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/R5NNGXYOU74VEXCBF7API7EFRGLN7MWJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Erlend Aasland]

On 20 Sep 2021, at 13:18, Pablo Galindo Salgado 
mailto:pablog...@gmail.com>> wrote:

We are doing this work in this branch: 
https://github.com/we-like-parsers/cpython/blob/fstring-grammar

That link is broken. Assuming you mean 
https://github.com/we-like-parsers/cpython/tree/fstring-grammar?


E
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W2VMLMV6OFPIG6NEL45NAOUJ3CHBIFK2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should the definition of an "(async) iterator" include __iter__?

[Brett Cannon]

On Sun, Sep 19, 2021 at 8:15 AM Steve Holden  wrote:

> I understood that _iterables_ are required to have an __iter__ method, not
> iterators.
>
> Therefore, are we simply discussing whether all iterators should be
> iterable?
>

At this point it's more about how to document this.


> At the moment the CPython implementation does't require that AFAIK.
>

Correct. I plan to go through the docs and clarify things. I opened
https://bugs.python.org/issue45250 to track this.


>
> regards
>  Steve
>
> On Tue, Sep 14, 2021 at 8:39 PM Guido van Rossum  wrote:
>
>> My view of this is:
>>
>> A. It's not an iterator if it doesn't define `__next__`.
>>
>> B. It is strongly recommended that iterators also define `__iter__`.
>>
>> In "standards" language, I think (A) is MUST and (B) is merely OUGHT or
>> maybe SHOULD.
>>
>> On Tue, Sep 14, 2021 at 12:30 PM Brett Cannon  wrote:
>>
>>> Over in https://github.com/python/typeshed/issues/6030 I have managed
>>> to kick up a discussion over what exactly an "iterator" is. If you look at
>>> https://docs.python.org/3/library/functions.html#iter you will see the
>>> docs say it "Return[s] an iterator
>>>  object." Great,
>>> but you go the glossary definition of "iterator" at
>>> https://docs.python.org/3/glossary.html#term-iterator you will see it
>>> says "[i]terators are required to have an __iter__()
>>> 
>>> method" which neither `for` nor `iter()` actually enforce.
>>>
>>> Is there something to do here? Do we loosen the definition of "iterator"
>>> to say they *should* define __iter__? Leave it as-is with an
>>> understanding that we know that it's technically inaccurate for iter() but
>>> that we want to encourage people to define __iter__? I'm assuming people
>>> don't want to change `for` and `iter()` to start requiring __iter__ be
>>> defined if we decided to go down the "remove the __aiter__ requirement"
>>> from aiter() last week.
>>>
>>> BTW all of this applies to async iterators as well.
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/3W7TDX5KNVQVGT5CUHBK33M7VNTP25DZ/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>> *Pronouns: he/him **(why is my pronoun here?)*
>> 
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/OICGRBPLXO6WXO4CHTGUK46WIHO7PDUU/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KHDMNMW6XEPYOZ5AQ22AN6YTN2POMHQE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WHUPQJQAKMKSOR54PCSWQQYYKLJ5ZKZS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Guido van Rossum]

On Mon, Sep 20, 2021 at 1:07 PM Patrick Reader <_...@pxeger.com> wrote:

> > The current restrictions will also confuse some users (e.g. those used
> to bash, and IIRC JS, where the rules are similar as what Pablo is
> proposing).
> > --
> > --Guido van Rossum (python.org/~guido )
>
> WRT the similar syntax in bash (and similar shells), there are two options:
>
> "string `code` string"
>
> "string $(code) string"
>
> The latter, $(), allows fully-featured nesting in the way Pablo is
> suggesting:
>
> "string $(code "string2 $(code2) string2" code) string"
>
> The former, using backticks, does not allow nesting directly, but it
> allows extra backslashes inside the backticks to escape the nested ones,
> like this:
>
> "string `code "string2 \`code2\` string2" code` string"
>
> This can be nested infinitely using lots of backslashes. Is this worth
> considering as another option? It doesn't have the disadvantage of
> complicating lexing (as much), although nesting with backslashes is quite
> ugly. IMO nesting things in f-strings would be ugly anyway, so I don't
> think that would matter too much.
>

F-strings are more like $(...), since the interpolation syntax uses {...}
delimiters. So it probably should work that way. JS interpolation works
that way too, see
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#nesting_templates
.

I wouldn't want to do anything to bring `backticks` back in the language.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YDGLXUWRG5BCPF5AK3AOC4DIJOMMLTL6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Brett Cannon]

On Mon, Sep 20, 2021 at 8:58 AM Thomas Grainger  wrote:

> I don't think the python syntax should be beholden to syntax highlighting
> tools, eventually some syntax feature that PEG enables will require every
> parser or highlighter to switch to a similar or more powerful parse tool
>

But that's not how syntax highlighting works in editors. You typically
don't get to choose the parsing tool used for syntax highlighting, you just
define the grammar using whatever is provided by the editor (which has
always been regexes based on my experience). So there's no way to "require"
every editor out there to switch to a PEG parser or equivalent to support
Python's grammar because that's asking every editor to change how syntax
highlighting is implemented at a lower level.

Having said all that, I think as long as we understand that this is a
side-effect then it's fine; syntax highlighting is usually not tied to
semantics in an editor so it shouldn't be a blocker on this. If people care
they simply won't use the same type of quotes in their code (which I bet is
what most people will do unless Black says otherwise 😉).

But I also think this means we definitely have to get a parser module for
tools together as this is way more potential breakage than just parentheses
for `with` statements and I don't know if formatting tools can just move to
the AST module at that point. 😅
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I4POAK22LZW4RNFGFFKQ6BILRLCSQO2I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Pablo Galindo Salgado]

>> But I also think this means we definitely have to get a parser module

What is in this context a "parse" module? Because that will massively
change depending who you ask. We already expose APIs that return AST
objects that can be used for all sort of things and a tokenizer module that
exposes some form of lexing that is relatively close to the one that
CPython uses internally. The only missing piece would be something that
returns a CST with enough information to reconstruct the source but at this
point that is absolutely arbitrary because nothing in CPython would use
that tree. Not only that, but the requirements from such CST will change
quite a lot depending on who you ask and that impacts a lot the APIs that
we would need to offer.

Offering a parse module here can involve quite a high maintainance cost
without the certainly that will be useful to all set of users.

That also without considering that many tools parsing Python code are not
written on Python and will not be able to leverage it.

On Mon, 20 Sep 2021, 22:39 Brett Cannon,  wrote:

>
>
> On Mon, Sep 20, 2021 at 8:58 AM Thomas Grainger  wrote:
>
>> I don't think the python syntax should be beholden to syntax highlighting
>> tools, eventually some syntax feature that PEG enables will require every
>> parser or highlighter to switch to a similar or more powerful parse tool
>>
>
> But that's not how syntax highlighting works in editors. You typically
> don't get to choose the parsing tool used for syntax highlighting, you just
> define the grammar using whatever is provided by the editor (which has
> always been regexes based on my experience). So there's no way to "require"
> every editor out there to switch to a PEG parser or equivalent to support
> Python's grammar because that's asking every editor to change how syntax
> highlighting is implemented at a lower level.
>
> Having said all that, I think as long as we understand that this is a
> side-effect then it's fine; syntax highlighting is usually not tied to
> semantics in an editor so it shouldn't be a blocker on this. If people care
> they simply won't use the same type of quotes in their code (which I bet is
> what most people will do unless Black says otherwise 😉).
>
> But I also think this means we definitely have to get a parser module for
> tools together as this is way more potential breakage than just parentheses
> for `with` statements and I don't know if formatting tools can just move to
> the AST module at that point. 😅
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/I4POAK22LZW4RNFGFFKQ6BILRLCSQO2I/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JCGJIU6ZHH4LLSL666EP2GBLMEJDVIOL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Stephen J. Turnbull]

Eric V. Smith writes:

 > >> But this does not:
 > >>
 > >> f'{1 +
 > >> 2}'
 > >
 > > The later is an error with or without the 'f' prefix and I think that 
 > > this should continue to be the case.
 > >
 > The thought is that anything that's within braces {} and is a valid 
 > expression should be allowed.

-0  FWIW, some thoughts specific to me, I don't know how
representative they might be of others.

I guess you could argue that the braces are a kind of expression-level
parenthesis, but I don't "see" them that way.  I see *one* string with
eval'able format expressions embedded in it, so that single-quoted
strings can't have embedded newlines.  I also don't see the braces as
expression-level syntax (after all, they already have two different
meanings at expression level), I see them as part of f-string syntax.
So even with triple-quoted strings, my eyes "want" to see parentheses
or line continuation (which already work).

I'm sure I could get used to the syntax.  But ...

Is this syntax useful?  Or is it just a variant of purity trying to
escape Pandora's virtualbox?  I mean, am I going to see it often
enough to get used to it?  Or am I going to WTF at it for the rest of
my life?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RPFHA55JDGX522UL2KXIRZKDPIOVDP66/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[David Mertz, Ph.D.]

I know I'm strongly -1 on allowing much more than currently exists for
f-strings. For basically the same reason Stephen explains.

Newlines inside braces, for example, go way too far away from readability.
Nested expressions also feel like an attractive nuisance. I use f-strings
all the time, but in much the same way a thousand character regular
expression is an abuse (even if perfectly well defined grammatically),
really complex f-strings worries look and feel much the same.

On Mon, Sep 20, 2021, 9:39 PM Stephen J. Turnbull <
stephenjturnb...@gmail.com> wrote:

> Eric V. Smith writes:
>
>  > >> But this does not:
>  > >>
>  > >> f'{1 +
>  > >> 2}'
>  > >
>  > > The later is an error with or without the 'f' prefix and I think that
>  > > this should continue to be the case.
>  > >
>  > The thought is that anything that's within braces {} and is a valid
>  > expression should be allowed.
>
> -0  FWIW, some thoughts specific to me, I don't know how
> representative they might be of others.
>
> I guess you could argue that the braces are a kind of expression-level
> parenthesis, but I don't "see" them that way.  I see *one* string with
> eval'able format expressions embedded in it, so that single-quoted
> strings can't have embedded newlines.  I also don't see the braces as
> expression-level syntax (after all, they already have two different
> meanings at expression level), I see them as part of f-string syntax.
> So even with triple-quoted strings, my eyes "want" to see parentheses
> or line continuation (which already work).
>
> I'm sure I could get used to the syntax.  But ...
>
> Is this syntax useful?  Or is it just a variant of purity trying to
> escape Pandora's virtualbox?  I mean, am I going to see it often
> enough to get used to it?  Or am I going to WTF at it for the rest of
> my life?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RPFHA55JDGX522UL2KXIRZKDPIOVDP66/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I7ZQTEZJMTUZF5KQSNQVTUCFEXYAVPAO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Guido van Rossum]

[Stephen J. Turnbull]

> Is this syntax useful?  Or is it just a variant of purity trying to
> escape Pandora's virtualbox?  I mean, am I going to see it often
> enough to get used to it?  Or am I going to WTF at it for the rest of
> my life?
>

I don't know about the line breaks, but in recent weeks I've found myself
more than once having to remind myself that inside interpolations, you must
use the other type of quote. Things like

print(f"{source.removesuffix(".py")}.c: $(srcdir)/{source}")

Learning that inside {} you can write any expression is easy (it's a real
Aha! moment -- that's the power of f-strings). Remembering that you have to
switch up the quote characters there is hard -- it doesn't occur very
often, and the reason is obscure. By the time my mental parser has made it
to the argument list of removesuffix() it has already forgotten that it's
inside an f-string and my fingers just reach for my favorite quote
character.

And the error isn't really helping either:
```
>>> print(f"{source.removesuffix(".py")}.c: $(srcdir)/{source}")
  File "", line 1
print(f"{source.removesuffix(".py")}.c: $(srcdir)/{source}")
  ^
SyntaxError: f-string: unmatched '('
```

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IWXRQFTNWMOW6ZLBOQW3U4KE2TDKXNWO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: f-strings in the grammar

[Stephen J. Turnbull]

Guido van Rossum writes:

 > I don't know about the line breaks, but in recent weeks I've found myself
 > more than once having to remind myself that inside interpolations, you must
 > use the other type of quote.

My earlier remarks were specifically directed to line breaks.

I see the point, but I think the question should be readability, as
David points out.  I don't think there's a problem with the opening
quote in your example.  Even in an ordinary string literal it's
obvious to me that the embedded quotation marks are not intended to
terminate the string:

s = "Here is a singleton " and here is an initial for "something."

But how about that last quotation mark?  I tried to construct a
similarly visually ambiguous f-string where braces "hide" the embedded
quotation marks, and couldn't do it without a trailing quote followed
immediately by an embedded literal line break.

So I'm cautiously sympathetic to this extension, as long as embedded
line breaks are not permitted in singly-quoted f-strings.

However, I myself will almost certainly automatically "correct" such
quotation marks if they are allowed.  So this is unlikely to be a plus
or a minus for me.

Steve




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZILJFTV6UXO63F76PSY6VCPNGTLMYIMR/
Code of Conduct: http://python.org/psf/codeofconduct/