[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Fri, 17 Apr 2020 13:59:46 -0600
Eric Snow  wrote:
> > For bytes or buffer objects, I understand that you propose to share
> > the exact same PyObject objects between interpreters, at least in the
> > first implementation.
> >
> > It may be better to have one proxy object in each interpreter which
> > would control which interpreters can read and which interpreters can
> > write into the object. It's a similar but more complex issue than
> > singletons.  
> 
> It isn't the same Python object.  It is the buffer that gets shared
> (for objects that support the buffer protocol).

When such a buffer dies, in which interpreter is the original object's
destructor called?

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NAPY6WAIW2YWZROHX3BNOPXW6GGT42U3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Paul Moore]

On Sat, 18 Apr 2020 at 00:03, Eric Snow  wrote:
>
> On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith  wrote:

> > I know you want folks to consider PEP 554 on its own merits, ignoring
> > the GIL-splitting work, but let's be realistic: purely as a
> > concurrency framework, there's at least a dozen more
> > mature/featureful/compelling options in the stdlib and on PyPI, and as
> > an isolation mechanism, subinterpreters have been around for >20 years
> > and in that time they've found 3 users and no previous champions.
> > Obviously the GIL stuff is the only reason PEP 554 might be worth
> > accepting.
>
> Saying it's "obviously" the "only" reason is a bit much. :)  PEP 554
> exposes existing functionality that hasn't been all that popular
> (until recently for some reason ) mostly because it is old, was
> never publicized (until recently), and involved using the C-API.  As
> soon as folks learn about it they want it, for various reasons
> including (relative) isolation and reduced resource usage in
> large-scale deployment scenarios.  It becomes even more attractive if
> you say subinterpreters allow you to work around the GIL in a single
> process, but that isn't the only reason.

As some context in contrast to Nathaniel's view, I remember back
around 15 years ago, when I first discovered the subinterpreter
support in the C API, I was tremendously interested in it, and very
frustrated that it wasn't exposed in Python. (To be fair, I was still
using Perl at the time, and it reminded me of Perl's implementation of
fork on Windows using subinterpreters, so maybe I was primed to see
use cases for it). I never did enough extension coding to use it in C,
but I remained interested in it for some time.

This PEP is very exciting for me, precisely because it gives me a
chance to explore subinterpreters from Python, which I've been
interested in for all those years. I've not really thought of it in
terms of concurrency, so the GIL isn't key to me, but more as an
alternative code structuring option. I don't know if I'll use it much
in production code, but being able to experiment and assess the
options is something I'm looking forward to. So I for one am grateful
to you for doing this work, and I'm in favour of it being in 3.9, just
so I can get my hands on it at last.

I do think that making it initially provisional may be a sensible
option, as it gives us the option to refine the Python API while
people are still learning how to use the new model.

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LIGC4BUYDUU7JIH7MCJLNAEQYKFW2LHX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Luciano Ramalho]

It is not clear to me (as an end-user of Python) what is the value of
exposing subinterpreters at the Python level except for multi core
concurrency, but that requires one GIL per subinterpreter.

Paul Moore mentions “alternative code structuring” as a benefit, but I
don’t know what that means in concrete terms.

If Paul or anyone else would make a case for the subinterpreters sharing a
single GIL, that would be great.

I understand many PEPs have to do with reducing technical debt in CPython
internals, which is great. But PEP 554 is about exposing a feature to
Python end users, so without use cases for them I believe this PEP should
be put in the back burner until such cases are articulated more clearly for
the wider Python community.

Thanks,

Luciano

On Sat, Apr 18, 2020 at 07:09 Paul Moore  wrote:

> On Sat, 18 Apr 2020 at 00:03, Eric Snow 
> wrote:
> >
> > On Fri, Apr 17, 2020 at 2:59 PM Nathaniel Smith  wrote:
>
> > > I know you want folks to consider PEP 554 on its own merits, ignoring
> > > the GIL-splitting work, but let's be realistic: purely as a
> > > concurrency framework, there's at least a dozen more
> > > mature/featureful/compelling options in the stdlib and on PyPI, and as
> > > an isolation mechanism, subinterpreters have been around for >20 years
> > > and in that time they've found 3 users and no previous champions.
> > > Obviously the GIL stuff is the only reason PEP 554 might be worth
> > > accepting.
> >
> > Saying it's "obviously" the "only" reason is a bit much. :)  PEP 554
> > exposes existing functionality that hasn't been all that popular
> > (until recently for some reason ) mostly because it is old, was
> > never publicized (until recently), and involved using the C-API.  As
> > soon as folks learn about it they want it, for various reasons
> > including (relative) isolation and reduced resource usage in
> > large-scale deployment scenarios.  It becomes even more attractive if
> > you say subinterpreters allow you to work around the GIL in a single
> > process, but that isn't the only reason.
>
> As some context in contrast to Nathaniel's view, I remember back
> around 15 years ago, when I first discovered the subinterpreter
> support in the C API, I was tremendously interested in it, and very
> frustrated that it wasn't exposed in Python. (To be fair, I was still
> using Perl at the time, and it reminded me of Perl's implementation of
> fork on Windows using subinterpreters, so maybe I was primed to see
> use cases for it). I never did enough extension coding to use it in C,
> but I remained interested in it for some time.
>
> This PEP is very exciting for me, precisely because it gives me a
> chance to explore subinterpreters from Python, which I've been
> interested in for all those years. I've not really thought of it in
> terms of concurrency, so the GIL isn't key to me, but more as an
> alternative code structuring option. I don't know if I'll use it much
> in production code, but being able to experiment and assess the
> options is something I'm looking forward to. So I for one am grateful
> to you for doing this work, and I'm in favour of it being in 3.9, just
> so I can get my hands on it at last.
>
> I do think that making it initially provisional may be a sensible
> option, as it gives us the option to refine the Python API while
> people are still learning how to use the new model.
>
> Paul
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LIGC4BUYDUU7JIH7MCJLNAEQYKFW2LHX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
Luciano Ramalho
|  Author of Fluent Python (O'Reilly, 2015)
| http://shop.oreilly.com/product/0636920032519.do
|  Technical Principal at ThoughtWorks
|  Twitter: @ramalhoorg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UNCDNFDQBBPWYFY45T3VQ4VIXVE3SDGO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Eric Snow]

On Sat, Apr 18, 2020, 01:16 Antoine Pitrou  wrote:

> On Fri, 17 Apr 2020 13:59:46 -0600
> Eric Snow  wrote:
> > > For bytes or buffer objects, I understand that you propose to share
> > > the exact same PyObject objects between interpreters, at least in the
> > > first implementation.
> > >
> > > It may be better to have one proxy object in each interpreter which
> > > would control which interpreters can read and which interpreters can
> > > write into the object. It's a similar but more complex issue than
> > > singletons.
> >
> > It isn't the same Python object.  It is the buffer that gets shared
> > (for objects that support the buffer protocol).
>
> When such a buffer dies, in which interpreter is the original object's
> destructor called?
>

In the original interpreter (via a "pending call").

-eric

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MGAMVLARCQXQ2Z2O5MN5W324KF47ATCR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

[Antoine Pitrou]

On Sat, 18 Apr 2020 10:08:27 -0600
Eric Snow  wrote:
> On Sat, Apr 18, 2020, 01:16 Antoine Pitrou  wrote:
> 
> > On Fri, 17 Apr 2020 13:59:46 -0600
> > Eric Snow  wrote:  
> > > > For bytes or buffer objects, I understand that you propose to share
> > > > the exact same PyObject objects between interpreters, at least in the
> > > > first implementation.
> > > >
> > > > It may be better to have one proxy object in each interpreter which
> > > > would control which interpreters can read and which interpreters can
> > > > write into the object. It's a similar but more complex issue than
> > > > singletons.  
> > >
> > > It isn't the same Python object.  It is the buffer that gets shared
> > > (for objects that support the buffer protocol).  
> >
> > When such a buffer dies, in which interpreter is the original object's
> > destructor called?
> >  
> 
> In the original interpreter (via a "pending call").

Great, thank you!

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M2PWQXFCYJOPZTJPL664NJOPHQPEITIX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 554 comments

[Antoine Pitrou]

Hello,

First, I would like to say that I have no fondamental problem with this
PEP. While I agree with Nathaniel that the rationale given about the CSP
concurrency model seems a bit weak, the author is obviously expressing
his opinion there and I won't object to that.  However, I think the PEP
is desirable for other reasons.  Mostly, I hope that by making the
subinterpreters functionality available to pure Python programmers
(while it was formally an advanced and arcane part of the C API), we
will spur of bunch of interesting third-party experimentations,
including possibilities that we on python-dev have not thought about.

The appeal of the PEP for experimentations is multiple:
1) ability to concurrently run independent execution environments
   without spawning child processes (which on some platforms and in some
   situations may not be very desirable: for example on Windows where
   the cost of spawning is rather high; also, child processes may
   crash, and sometimes it is not easy for the parent to recover,
   especially if a synchronization primitive is left in an unexpected
   state)
2) the potential for parallelizing CPU-bound pure Python code
   in a single process, if a per-interpreter GIL is finally implemented
3) easier support for sharing large data between separate execution
   environments, without the hassle of setting up shared memory or the
   fragility of relying on fork() semantics

(and as I said, I hope people find other applications)

As for the argument that we already have asyncio and several other
packages, I actually think that combining these different concurrency
mechanisms would be interesting complex applications (such as
distributed systems).  For that, however, I think the PEP as currently
written is a bit lacking, see below.

Now for the detailed comments.

* I think the module should indeed be provisional.  Experimentation may
  discover warts that call for a change in the API or semantics.  Let's
  not prevent ourselves from fixing those issues.

* The "association" timing seems quirky and potentially annoying: an
  interpreter only becomes associated with a channel the first time it
  calls recv() or send().  How about, instead, associating an
  interpreter with a channel as soon as that channel is given to it
  through `Interpreter.run(..., channels=...)` (or received through
  `recv()`)? 

* How hard would it be, in the current implementation, to add buffering
  to channels?  It doesn't have to be infinite: you can choose a fixed
  buffer size (or make it configurable in the create() function, which
  allows passing 0 for unbuffered).  Like Nathaniel, I think unbuffered
  channels will quickly be annoying to work with (yes, you can create a
  helper thread... now you have one additional thread per channel,
  which isn't pretty -- especially with the GIL).

* In the same vein, I think channels should allow adding readiness
  callbacks (that are called whenever a channel becomes ready for
  sending or receiving, respectively).  This would make it easy to plug
  them into an event loop or other concurrency systems (such as
  Future-based concurrency).  Note that each interpreter "associated"
  with a channel should be able to set its own readiness callback: so
  one callback per Python object representing the channel, but
  potentially multiple callbacks for the underlying channel primitive.

  (how would the callback be scheduled for execution in the right
  interpreter? perhaps using `_PyEval_AddPendingCall()` or a similar
  mechanism?)

* I think either `interpreters.get_main()` or `interpreters.is_main()`
  is desirable.  Inevitable, the slight differences between main and
  non-main interpreters will surface in non-trivial applications
  (finalization issues in distributed systems can really be hairy).  It
  seems this should be mostly costless to provide, so let's do it.

* I do think a minimal synchronization primitive would be nice.
  Either a Lock (in the Python sense) or a Semaphore: both should be
  relatively easy to provide, by wrapping an OS-level synchronization
  primitive.  Then you can recreate all high-level synchronization
  primitives, like the threading and multiprocessing modules do (using
  a Lock or a Semaphore, respectively).

  (note you should be able to emulate a semaphore using blocking send()
  and recv() calls, but that's probably not very efficient, and
  efficiency is important)

Of course, I hope these are all actionable before beta1 :-)  If not,
here is my preferential priority list:

* High priority: fix association timing
* High priority: either buffering /or/ readiness callbacks
* Middle priority: get_main() /or/ is_main()
* Middle / low priority: a simple synchronization primitive

But I would stress the more of these we provide, the more we encourage
people to experiment without pulling too much of their hair.

(also, of course, I hope other people read the PEP and emit feedback)

Best regards

Antoine.

_

[Python-Dev] Re: PEP 554 comments

[Antoine Pitrou]

On Sat, 18 Apr 2020 19:02:47 +0200
Antoine Pitrou  wrote:
> 
> * I do think a minimal synchronization primitive would be nice.
>   Either a Lock (in the Python sense) or a Semaphore: both should be
>   relatively easy to provide, by wrapping an OS-level synchronization
>   primitive.  Then you can recreate all high-level synchronization
>   primitives, like the threading and multiprocessing modules do (using
>   a Lock or a Semaphore, respectively).

By the way, perhaps this could be even be implemented as making
_threading.Lock shareable.  This would probably require some changes in
the underlying C Lock structure (e.g. pointing to an
atomically-refcounted shared control block), but nothing intractable,
and reasonably efficient.

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CWSJBHQAFH72EPQRB5HJRP5YWIYDICOZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Further PEP 615 Discussion: Equality and hash of ZoneInfo

[Paul Ganssle]

A few weeks ago, I submitted PEP 615 to the steering council
 for a decision.
There's been a decent amount of discussion there with some very good
questions. I think they are mostly resolved (though I'm happy to have
other people look over the logic of some of my responses there), but
Victor brought up the question of the equality and hash behavior of
ZoneInfo, and while I'm leaning one way, I could easily be convinced -
particularly if people have /real/ use cases that they are planning to
implement that would be affected by this.

I've summed up the options on the discourse thread
,
and would appreciate if anyone is able to give some feedback.

Thanks!

Paul



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JSA6LNJH6NP7JN32NMPWYNYUDAIAKFFX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Further PEP 615 Discussion: Equality and hash of ZoneInfo

[Chris Angelico]

On Sun, Apr 19, 2020 at 3:39 AM Paul Ganssle  wrote:
>
> A few weeks ago, I submitted PEP 615 to the steering council for a decision. 
> There's been a decent amount of discussion there with some very good 
> questions. I think they are mostly resolved (though I'm happy to have other 
> people look over the logic of some of my responses there), but Victor brought 
> up the question of the equality and hash behavior of ZoneInfo, and while I'm 
> leaning one way, I could easily be convinced - particularly if people have 
> real use cases that they are planning to implement that would be affected by 
> this.
>
> I've summed up the options on the discourse thread, and would appreciate if 
> anyone is able to give some feedback.
>
> Thanks!

In every use-case that I've ever had, and every one that I can
imagine, I've not cared about the difference between "US/Eastern" and
"America/New_York". In fact, if ZoneInfo("US/Eastern") returned
something that had a name of "America/New_York", I would be fine with
that. Similarly, Australia/Melbourne and Australia/Sydney are, to my
knowledge, equivalent. (If I'm wrong on my own country's history of
timezones, then I apologize and withdraw the example, but I'm talking
about cases where you absolutely cannot tell the difference based on
the displayed time.) Having those compare equal would be convenient.

I don't think it's a problem to have equivalent objects unable to
coexist in a set. That's just the way sets work - len({5, 5.0}) is 1,
not 2.

Since options 3 and 4 are the most expensive, I'm fine with the idea
of a future method that would test for equivalence, rather than having
them actually compare equal; but I'd also be fine with having
ZoneInfo("US/Eastern") actually return the same object that
ZoneInfo("America/New_York") returns. For the equality comparison, I
would be happy with proposal 2.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JPUGSSXX2MWF3ABH3QNHXSMNVDWMRVJS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Carl Meyer]

The PEP is exciting and is very clearly presented, thank you all for
the hard work!

Considering the comments in the PEP about the new parser not
preserving a parse tree or CST, I have some questions about the future
options for Python language-services tooling which requires a CST in
order to round-trip and modify Python code. Examples in this space
include auto-formatters, refactoring tools, linters with autofix, etc.
Today many such tools (e.g. Black, 2to3) are based on lib2to3. Other
tools already have their own parser (e.g. LibCST -- which I help
maintain -- and Jedi both use parso, a fork of pgen2).

1) 2to3 and lib2to3 are not mentioned in the PEP, but are a documented
part of the standard library used by some very popular tools, and
currently depend on pgen2. A quick search of the PEP 617 pull request
does not suggest that it modifies lib2to3. Will lib2to3 also be
removed in Python 3.10 along with the old parser? It might be good for
the PEP to address the future of 2to3 and lib2to3 explicitly.

2) As these tools make the necessary adaptations to support Python
3.10, which may no longer be parsable with an LL(1) parser, will we be
able to leverage any part of pegen to construct a lossless Python CST,
or will we likely need to fork pegen outside of CPython or build a
wholly new parser? It would be neat if an alternate grammar could be
written in pegen that has access to all tokens (including NL and
COMMENT) for this purpose; that would save a lot of code duplication
and potential for inconsistency. I haven't had a chance to fully read
through the PEP 617 pull request, but it looks like its tokenizer
wrapper currently discards NL and COMMENT. I understand this is a
distinct use case with distinct needs and I'm not suggesting that we
should make significant sacrifices in the performance or
maintainability of pegen to serve it, but if it's possible to enable
some sharing by making API choices now before it's merged, that seems
worth considering.

Carl
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5MPFFCYOEDKEPKNSNIDZ7H6AYTXUFFAY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 554 comments

[Greg Ewing]

On 19/04/20 5:02 am, Antoine Pitrou wrote:

* How hard would it be, in the current implementation, to add buffering
   to channels?

* In the same vein, I think channels should allow adding readiness
   callbacks


Of these, I think the callbacks are more fundamental. If you
have a non-buffered channel with readiness callbacks, you can
implement a buffered channel on top of it.

--
Greg
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ISWUN7TNPQE6YPDI5HGENYTZ25I2MCBA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Guido van Rossum]

On Sat, Apr 18, 2020 at 4:53 PM Carl Meyer  wrote:

> The PEP is exciting and is very clearly presented, thank you all for
> the hard work!
>
> Considering the comments in the PEP about the new parser not
> preserving a parse tree or CST, I have some questions about the future
> options for Python language-services tooling which requires a CST in
> order to round-trip and modify Python code. Examples in this space
> include auto-formatters, refactoring tools, linters with autofix, etc.
> Today many such tools (e.g. Black, 2to3) are based on lib2to3. Other
> tools already have their own parser (e.g. LibCST -- which I help
> maintain -- and Jedi both use parso, a fork of pgen2).
>

Right, LibCST is very exciting. Note that AFAIK none of the tools you
mention depend on the old parser module. (Though I'm not denying that there
might be tools depending on it -- that's why we're keeping it until 3.10.)


> 1) 2to3 and lib2to3 are not mentioned in the PEP, but are a documented
> part of the standard library used by some very popular tools, and
> currently depend on pgen2. A quick search of the PEP 617 pull request
> does not suggest that it modifies lib2to3. Will lib2to3 also be
> removed in Python 3.10 along with the old parser? It might be good for
> the PEP to address the future of 2to3 and lib2to3 explicitly.
>

Note that, while there is indeed a docs page about 2to3
, the only docs for *lib2to3*
in the standard library reference are a link to the source code and a
single "*Note:* The lib2to3

API should be considered unstable and may change drastically in the future."

Fortunately,  in order to support the 2to3 application, lib2to3 doesn't
need to change, because the syntax of Python 2 is no longer changing. :-)
Choosing to remove 2to3 is an independent decision. And lib2to3 does not
depend in any way on the old parser module. (It doesn't even use the
standard tokenize module, but incorporates its own version that is slightly
tweaked to support Python 2.)


> 2) As these tools make the necessary adaptations to support Python
> 3.10, which may no longer be parsable with an LL(1) parser, will we be
> able to leverage any part of pegen to construct a lossless Python CST,
> or will we likely need to fork pegen outside of CPython or build a
> wholly new parser? It would be neat if an alternate grammar could be
> written in pegen that has access to all tokens (including NL and
> COMMENT) for this purpose; that would save a lot of code duplication
> and potential for inconsistency. I haven't had a chance to fully read
> through the PEP 617 pull request, but it looks like its tokenizer
> wrapper currently discards NL and COMMENT. I understand this is a
> distinct use case with distinct needs and I'm not suggesting that we
> should make significant sacrifices in the performance or
> maintainability of pegen to serve it, but if it's possible to enable
> some sharing by making API choices now before it's merged, that seems
> worth considering.
>

You've mentioned a few different tools that already use different
technologies: LibCST depends on parso which has a fork of pgen2, lib2to3
which has the original pgen2. I wonder if this would be an opportunity to
move such parsing support out of the standard library completely. There are
already two versions of pegen, but neither is in the standard library:
there is the original pegen  repo
which is where things started, and there is a fork of that code in the CPython
Tools

directory (not yet in the upstream repo, but see PR 19503
).

The pegen tool has two generators, one generating C code and one generating
Python code. I think that the C generator is really only relevant for
CPython itself: it relies on the builtin tokenizer (the one written in C,
not the stdlib tokenize.py) and the generated C code depends on many
internal APIs. In fact the C generator in the original pegen repo doesn't
work with Python 3.9 because those internal APIs are no longer exported.
(It also doesn't work with Python 3.7 or older because it makes critical
use of the walrus operator. :-) Also, once we started getting serious about
replacing the old parser, we worked exclusively on the C generator in the
CPython Tools directory, so the version in the original pegen repo is
lagging quite a bit behind (is is the Python grammar in that repo). But as
I said you're not gonna need it.

On the other hand, the Python generator is designed to be flexible, and
while it defaults to using the stdlib tokenize.py tokenizer, you can easily
hook up your own. Putting this version in the stdlib would be a mistake,
because the code is pretty immature; it is really waiting for a good home,
and if parso or LibCST were to decide to incorporate 

[Python-Dev] Re: PEP 617: New PEG parser for CPython

[Nam Nguyen]

On Sat, Apr 18, 2020 at 9:45 PM Guido van Rossum  wrote:


> But I never found the time to make it an appealing parser generator tool
> for other languages, even though that was on my mind as a future
> possibility.
>

I simply want to +1 on this. A general purpose parser library in the stdlib
would be fantastic for its many parsing needs. I'm so looking forward to
that future.
Thanks,
Nam
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HWBHOAZ4HMGQZ7LSLO7WEQMPDZRLGG7A/
Code of Conduct: http://python.org/psf/codeofconduct/