Re: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation

[Steve Holden]

I suspect this discussion would be better moved to python-ideas, since it
seems clear there needs to be some work on determining the exact
requirements


On Sat, May 18, 2019 at 4:03 PM Dan Ryan  wrote:

> It probably makes the most sense to formally outline all of the behavioral
> changes, expectations, and constraints you are asking for as Brett said
> because dependency and package management are tricky and it's not clear (to
> me at least) what the request is, what problem is being encountered, and
> how it is proposed that it be resolved.
>
> First glance makes it sound a bit like the a prerequisite for this request
> would be allowing multiple versions of the same package to be installed in
> a given python environment to begin with, or else some runtime hacking of
> paths would be necessary.  I may be misunderstanding, but that's why a
> formal approach for something like this might make some sense
>
> Dan Ryan
> gh: @techalchemy // e: d...@danryan.co
>
> From: Q [mailto:qiang.f...@zoho.com.cn]
> Sent: Friday, May 17, 2019 10:32 PM
> To: Daniel Holth
> Cc: Brett Cannon; Python-Dev
> Subject: Re: [Python-Dev] Feature request: Change a Dependency Package
> Version During Package Initiation
>
> Sorry, there is an error in my last post. It's corrected as:
> The abaqus python distribution has libA 1.0, the external package to be
> installed is LibB, which requires libA2.0, and we don't want to change
> every import in LibB to be import /somewhere/libA2.0, because the source
> code may have not been provided.
>
>
>  在 星期六, 2019-05-18 10:26:04 Q  撰写 
>
> From the viewpoint of the package user, people don't want to change every
> import to "require a version at runtime". If "set up a python path in a
> wrapper script" is a better strategy, can you please give an example for
> the following use case:
> The abaqus python distribution has libA 1.0, the external package to be
> installed is LibB, which requires libA2.0, and we don't want to change
> every import in LibB to be import libA 1.0, because the source code may
> have not been provided.
> In emacs, you can always advice a command to change its behaviour, which
> makes it very user friendly, and that what all the following suggestion is
> about. I have no idea at all how it could be implemented though.
> https://github.com/mitsuhiko/multiversion/issues/1
> "The work around is to let a version to be set within a package and have
> that propagate to all modules in that package. For example in the root
> init.py if I set tonado version to be 2.2.1 then all modules in that
> package will use tornado 2.2.1 when I import tornado."
>
>  在 星期五, 2019-05-17 23:38:55 Daniel Holth  撰写 
>
>
>
> This sounds exactly like what people used to do with eggs. You could have
> multiple versions of a package on the path as eggs and then require a
> version at runtime. The approach has problems. Ruby also abandoned a
> strategy where random app code depends on package management code at
> runtime.
>
> One better strategy is to set up a python path in a wrapper script.
>
> On Fri, May 17, 2019, 11:27 Brett Cannon  wrote:
>
> Thanks for the idea but there are currently no plans to support such a
> feature. If you would like to see it then you will need to write a PEP with
> a proof-of-concept to demonstrate how you would expect such a feature to
> work.
>
> On Fri., May 17, 2019, 07:55 Q via Python-Dev, 
> wrote:
>
> A lot of the Python code we use in production are used directly as imports
> in other python
> distributions (such as the python comes with the finite element software
> Abaqus and MSC Marc), many
> packages (such as matplotlib, numpy) that may have varying versioned
> dependencies.
>
> I was wondering if this could be expanded to allow a version to be set
> within a package and have
> that propagate to all modules in that package. For example in the root
> init.py if I set
> multiversion(tornado, 2.2.1) then all modules in that package will use
> tornado 2.2.1 when I import
> tornado.
>
> See a relevant issue on github:
> https://github.com/mitsuhiko/multiversion/issues/1
>
> Thank you!
> Qiang
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version

[Victor Stinner]

Hi Gregory,

IMHO your remarks are not directly related to the PEP 587 and can be
addressed in parallel.

> It sounds like PyOxidizer and Hermetic Python are on the same page and
> we're working towards a more official solution. But I want to make sure
> by explicitly stating what PyOxidizer is doing.
>
> Essentially, to facilitate in-memory import, we need to register a
> custom sys.meta_path importer *before* any file-based imports are
> attempted. (...)
> I /think/ providing a 2-phase
> initialization that stops between _Py_InitializeCore() and
> _Py_InitializeMainInterpreter() would get the job done for PyOxidizer
> today. (...)

Extract of PEP 587: "This extracts a subset of the API design from the
PEP 432 development and refactoring work that is now considered
sufficiently stable to make public (allowing 3rd party embedding
applications access to the same configuration APIs that the native
CPython CLI is now using)."

We know that my PEP 587 is incomplete, but the work will continue in
Python 3.9 to support your use case.

The PEP 587 introduces an experimental separation between "core" and
"main" initialization phases. PyConfig._init_main=0 stops at the
"core" phase, then you are free to run C and Python,
_Py_InitializeMain() finishes the Python initialization ("main"
phase).


> > In the "Isolate Python" section, I suggest to set the "isolated"
> > parameter to 1 which imply setting user_site_directory to 0. So
> > sys.path isn't modified afterwards. What you pass to PyConfig is what
> > you get in sys.path in this case.
>
> Regarding site.py, I agree it is problematic for embedding scenarios.
> Some features of site.py can be useful. Others aren't. It would be
> useful to have more granular control over which bits of site.run are
> run. My naive suggestion would be to add individual flags to control
> which functions site.py:main() runs. That way embedders can cherry-pick
> site.py features without having to manually import the module and call
> functions within. That feels much more robust for long-term maintainability.

I agree that more work can be done on the site module. IMHO core
features which are needed by everybody should be done before calling
site. Maybe using a frozen "presite" module or whatever. I would be
interested to make possible to use Python for most cases without the
site module.


> Regarding Python calling exit(), this is problematic for embedding
> scenarios.

I am working on that. I fixed dozens of functions. For example,
Py_RunMain() should not longer exit if there is an uncaught SystemExit
when calling PyErr_Print(). SystemExit is now handled separately
before calling PyErr_Print(). The work is not done, but it should be
way better than Python 3.6 and 3.7 state.


> This thread called attention to exit() during interpreter
> initialization. But it is also a problem elsewhere. For example,
> PyErr_PrintEx() will call Py_Exit() if the exception is a SystemExit.
> There's definitely room to improve the exception handling mechanism to
> give embedders better control when SystemExit is raised. As it stands,
> we need to check for SystemExit manually and reimplement
> _Py_HandleSystemExit() to emulate its behavior for e.g. exception value
> handling (fun fact: you can pass non-None, non-integer values to
> sys.exit/SystemExit).

I don't know well these functions, maybe new functions are needed. It
can be done without/outside the PEP 587.


> Having a more efficient member lookup for BuiltinImporter and
> FrozenImporter might shave off a millisecond or two from startup. This
> would require some kind of derived data structure. (...)

I don't think that the structures/API to define frozen/builtin modules
has to change. We can convert these lists into an hash table during
the initialization of the importlib module.

I'm not saying that the current API is perfect, just that IMHO it can
be solved without the API.


> Unfortunately, as long as there is a global data structure that can be 
> mutated any time
> (the API contract doesn't prohibit modifying these global arrays after
> initialization), you would need to check for "cache invalidation" on
> every lookup, undermining performance benefits.

Do you really expect an application modifying these lists dynamically?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 587 "Python Initialization Configuration" version 4

[Victor Stinner]

Hi,

I expected the version 3 of my PEP to be complete, but Gregory Szorc
and Steve Dower spotted a few more issues ;-)

The main change of the version 4 is the introduction of "Python
Configuration" and "Isolated Configuration" default configuration
which are well better defined.

The new "Isolated Configuration" provides sane default values to
isolate Python from the system. For example, to embed Python into an
application. Using the environment are now opt-in options, rather than
an opt-out options. For example, environment variables, command line
arguments and global configuration variables are ignored by default.

Building a customized Python which behaves as the regular Python
becomes easier using the new Py_RunMain() function. Moreover, using
the "Python Configuration", PyConfig.argv arguments are now parsed the
same way the regular Python parses command line arguments, and
PyConfig.xoptions are handled as -X opt command line options.

I replaced all macros with functions. Macros can cause issues when
used from different programming languages, whereas functions are
always well supported.

PyPreConfig structure doesn't allocate memory anymore (the allocator
field becomes an integer, instead of a string). I removed the
"Constant PyConfig" special case which introduced too many exceptions
for little benefit.

See the "Version History" section for the full changes.

HTML version:
https://www.python.org/dev/peps/pep-0587/

Full text below.

Victor


PEP: 587
Title: Python Initialization Configuration
Author: Victor Stinner , Nick Coghlan 
BDFL-Delegate: Thomas Wouters 
Discussions-To: python-dev@python.org
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Mar-2019
Python-Version: 3.8

Abstract


Add a new C API to configure the Python Initialization providing finer
control on the whole configuration and better error reporting.

It becomes possible to read the configuration and then override some
computed parameters before it is applied. It also becomes possible to
completely override how Python computes the module search paths
(``sys.path``).

The new `Isolated Configuration`_ provides sane default values to
isolate Python from the system. For example, to embed Python into an
application. Using the environment are now opt-in options, rather than
an opt-out options. For example, environment variables, command line
arguments and global configuration variables are ignored by default.

Building a customized Python which behaves as the regular Python becomes
easier using the new ``Py_RunMain()`` function. Moreover, using the
`Python Configuration`_, ``PyConfig.argv`` arguments are now parsed the
same way the regular Python parses command line arguments, and
``PyConfig.xoptions`` are handled as ``-X opt`` command line options.

This extracts a subset of the API design from the PEP 432 development
and refactoring work that is now considered sufficiently stable to make
public (allowing 3rd party embedding applications access to the same
configuration APIs that the native CPython CLI is now using).


Rationale
=

Python is highly configurable but its configuration evolved organically.
The initialization configuration is scattered all around the code using
different ways to set them: global configuration variables (ex:
``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``),
command line arguments (ex: ``-b``), configuration files (ex:
``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A
straightforward and reliable way to configure Python is needed.

Some configuration parameters are not accessible from the C API, or not
easily. For example, there is no API to override the default values of
``sys.executable``.

Some options like ``PYTHONPATH`` can only be set using an environment
variable which has a side effect on Python child processes if not unset
properly.

Some options also depends on other options: see `Priority and Rules`_.
Python 3.7 API does not provide a consistent view of the overall
configuration.

The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as
input whereas the Python filesystem encoding is set during the
initialization which can lead to mojibake.

Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory
allocation failure which is not convenient when Python is embedded.
Moreover, ``Py_Main()`` could exit directly the process rather than
returning an exit code. Proposed new API reports the error or exit code
to the caller which can decide how to handle it.

Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly
was almost impossible in Python 3.6. The code base has been deeply
reworked in Python 3.7 and then in Python 3.8 to read the configuration
into a structure with no side effect. It becomes possible to clear the
configuration (release memory) and read again the configuration if the
encoding changed . It is required to implement properly the UTF-8 which
changes the encoding using ``-X ut

[Python-Dev] ast changes for "debug" f-strings

[Eric V. Smith]

I added an optional "expr_text" field to the FormattedValue node, which 
represents the text of the expression in a "debug" f-string expression. 
So in f'{x=}', expr_text would be "x=".


This strictly speaking isn't necessary. I could have added another 
Constant node for "x=" and left FormattedValue alone. I didn't for three 
reasons: it was expedient; it didn't require a lot of surgery to 
f-string parsing, which the extra Constant node would require; and it 
allowed the Python/ast_unparse.c code to produce a string that was more 
consistent with input string.


Now that I have more time, I'm rethinking this decision. I'd rather not 
have the extra churn that modifying the ast node will cause to 
downstream code. Someone (Serhiy?) already pointed this out somewhere, 
but now I can't find the original message.


But the issue of the fidelity of ast_unparse is still there. I think the 
only place this code is used is "from __future__ import annotations", 
but for all I know there's a plan to use it elsewhere.


Does anyone care that f'{x=}' would become f'x={x!r}' if I removed 
expr_text from the FormattedValue node?


This isn't the only place where ast_unparse isn't exact with what it 
produces. It's not designed to give the exact source text back, just 
something equivalent.


But it's not clear to me how equivalent it needs to be. Even if I did 
add the extra Constant node, then you'd have the following discrepancy:


f'value: {x=}' would produce:
[Expr(value=JoinedStr(values=[Constant(value='value: ', kind=None), 
Constant(value='x=', kind=None), FormattedValue(value=Name(id='x', 
ctx=Load()), conversion=114, format_spec=None, expr_text='x=')]))]


Note that there are two consecutive Constant nodes here. So this:

f'value: x={x!r}' would produce the slightly different:
[Expr(value=JoinedStr(values=[Constant(value='value: x=', kind=None), 
FormattedValue(value=Name(id='x', ctx=Load()), conversion=114, 
format_spec=None)]))]


Note that there is a single Constant node, with the combined string in it.

I'm not sure how much we care about all of this, but let me know if you 
have a strong feeling about it.


My suggestion for the 3.8b1 release is to add the extra Constant node 
and remove expr_text from the FormattedValue node. This would at least 
mean that there's no change to the ast definitions due to this feature.


If we decide that the two consecutive Constant nodes generated by 
f'value: {x=}' need to be combined, then we can do that post 3.8b1. But 
it's a lot more work to combine the nodes while the f-string is being 
compiled.


Let me know your thoughts, since time is of the essence.

Eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Plan to accept PEP 544, PEP 586, PEP 589, PEP 591 -- last call for discussion

[Guido van Rossum]

Please send all discussions about these PEPs to the typing-sig mailing
list, so we have the archives all in one place. Read the full message here:

https://mail.python.org/archives/list/typing-...@python.org/thread/GCI2TF43SESP2XVSETWOW33SRRWQTHM4/

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ast changes for "debug" f-strings

[Batuhan Taskaya]

> This strictly speaking isn't necessary. I could have added another
Constant node for "x=" and left FormattedValue alone. I didn't for three
reasons: it was expedient; it didn't require a lot of surgery to
f-string parsing, which the extra Constant node would require; and it
allowed the Python/ast_unparse.c code to produce a string that was more
consistent with input string.

Agreed.

> Does anyone care that f'{x=}' would become f'x={x!r}' if I removed
expr_text from the FormattedValue node?

Yes, when i was implementing f-string debugging support to Berker's astor
project
the roundtrip tests i wrote is failing because of it adds an extra `!r` to
end. Then
i realized you added a new field (expr_text) for that.

> I'm not sure how much we care about all of this, but let me know if you
have a strong feeling about it.

I don't think we should complicate this. The current version is very simple
and understandable.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Plan to accept PEP 544, PEP 586, PEP 589, PEP 591 -- last call for discussion

[Victor Stinner]

FYI, copy of Guido's message:
---
I'm planning to accept the following PEPs related to the type system
later this week:

PEP 544: Protocols
PEP 586: Literal Types
PEP 591: Final qualifier and @final decorator
PEP 589: TypedDict

All of these have been proposed for and discussed before.

All of these have been implemented in mypy and (AFAIK) in some other
type checkers (with the exception of Protocols for module objects,
which is in the design stage for mypy but not controversial in any
way).

No substantial opposition came up during the discussions so far. Some
changes were made to some of the PEPs and it seems everyone is happy
with the results.

I expect that these PEPs will be accepted in time to get their
implementations into the stdlib typing.py module in 3.8 beta 1, which
is due May 31st (i.e., in less than two weeks!).
---

But please follow Guido's request:

Le lun. 20 mai 2019 à 16:39, Guido van Rossum  a écrit :
> Please send all discussions about these PEPs to the typing-sig mailing list, 
> so we have the archives all in one place. Read the full message here:

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Guido van Rossum]

On Thu, May 16, 2019 at 3:57 PM Steve Dower  wrote:

> [...]
> We still have the policy of not removing modules that exist in the
> Python 2 standard library. But 3.9 won't be covered by that :)
>

I didn't even remember that. Where's that written down? And by the time 3.8
.0(final) comes out, 2.7 has only about two months of life left...

FWIW I am strongly in favor of getting rid of the `parser` module, in 3.8
is we can, otherwise in 3.9 (after strong deprecation in 3.8).

I am interested in switching CPython's parsing strategy to something else
(what exactly remains to be seen) and any new approach is unlikely to reuse
the current CST technology. (OTOH I think it would be wise to keep the
current AST.)


> But I'm in favor of having a proper CST module that matches the version
> of Python it's in. It doesn't help people on earlier versions (yet), but
> given how closely tied it is to the Python version you're on I think it
> makes sense in the stdlib.
>

I presume this is in reference to Łukasz's
https://bugs.python.org/issue7. I think we should act on that issue,
but I don't think there's a good reason to tie deletion (or deprecation) of
the `parser` module to whatever we do there.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Guido van Rossum]

On Thu, May 16, 2019 at 3:51 PM Pablo Galindo Salgado 
wrote:

> [Nathaniel Smith]
>
>Will the folks using forks be happy to switch to the stdlib version?
> >For example I can imagine that if black wants to process 3.7 input
> >code while running on 3.6, it might prefer a parser on PyPI even if
> >he stdlib version were public, since the PyPI version can be updated
> >independently of the host Python.
>
> The tool can parse arbitrary grammars, the one that is packed into is just
> one of them.
>
> I think it would be useful, among other things because the standard library
> lacks currently a proper CST solution. The ast module is heavily leveraged
> for
> things like formatters,
>

Actually, I think the `ast` module doesn't work very well for formatters,
because it loses comments. (Retaining comments and all details of
whitespace is the specific use case for which I created pgen2.)


> static code analyzers...etc but CST can be very useful as
> Łukasz describes here:
>
> https://bugs.python.org/issue7
>
> I think is missing an important gap in the stdlib and the closest thing we
> have
> (the current parser module) is not useful for any of that. Also, the core
> to generating
> the hypothetical new package (with some new API over it may be) is already
> undocumented
> as an implementation detail of lib2to3 (and some people are already using
> it directly).
>

I wonder if lib2to3 is actually something that would benefit from moving
out of the stdlib. (Wasn't it on Amber's list?) As Łukasz points out in
that issue, it is outdated. Maybe if it was out of the stdlib it would
attract more contributors. Then again, I have recently started exploring
the idea of a PEG parser for Python. Maybe a revamped version of the core
of lib2to3 based on PEG instead of pgen would be interesting to some folks.

I do agree that the two versions of tokenize.py should be unified (and the
result kept in the stdlib). However there are some issues here, because
tokenize.py is closely tied to the names and numbers of the tokens, and the
latter information is currently generated based on the contents of the
Grammar file. This may get in the way of using it to tokenize old Python
versions.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ast changes for "debug" f-strings

[Serhiy Storchaka]

20.05.19 16:25, Eric V. Smith пише:
I added an optional "expr_text" field to the FormattedValue node, which 
represents the text of the expression in a "debug" f-string expression. 
So in f'{x=}', expr_text would be "x=".


This strictly speaking isn't necessary. I could have added another 
Constant node for "x=" and left FormattedValue alone. I didn't for three 
reasons: it was expedient; it didn't require a lot of surgery to 
f-string parsing, which the extra Constant node would require; and it 
allowed the Python/ast_unparse.c code to produce a string that was more 
consistent with input string.


Now that I have more time, I'm rethinking this decision. I'd rather not 
have the extra churn that modifying the ast node will cause to 
downstream code. Someone (Serhiy?) already pointed this out somewhere, 
but now I can't find the original message.


Of course I am for removing "expr_text". This makes the AST simpler. I 
do not care about ast_unparse because it does nor roundtrip in general. 
For example it can add and remove parenthesis, commas, backslashes. 
Currently it even has a bug in handling some corner cases in f-strings 
-- nobody complains.


In theory it is possible to detect when !d was used (parse the previous 
Constant value and compare the result with the FormattedValue 
expression). We can implement this if there is a need after beta1.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] ast changes for "debug" f-strings

[Eric V. Smith]

On 5/20/2019 10:32 AM, Batuhan Taskaya wrote:

 > This strictly speaking isn't necessary. I could have added another
Constant node for "x=" and left FormattedValue alone. I didn't for three
reasons: it was expedient; it didn't require a lot of surgery to
f-string parsing, which the extra Constant node would require; and it
allowed the Python/ast_unparse.c code to produce a string that was more
consistent with input string.

Agreed.

 > Does anyone care that f'{x=}' would become f'x={x!r}' if I removed
expr_text from the FormattedValue node?

Yes, when i was implementing f-string debugging support to Berker's 
astor project
the roundtrip tests i wrote is failing because of it adds an extra `!r` 
to end. Then

i realized you added a new field (expr_text) for that.

 > I'm not sure how much we care about all of this, but let me know if you
have a strong feeling about it.

I don't think we should complicate this. The current version is very 
simple and understandable.


I think the salient question is: does the lack of expr_text make 
anything more difficult for anyone? As Serhiy said in the other email, 
lots of things are lost while round-tripping, this would just be another 
one. Anyone really interested in high-fidelity round trips already can't 
use the ast module to unparse from.


Eric

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Pablo Galindo Salgado]

> Actually, I think the `ast` module doesn't work very well for formatters,
because it loses comments. (Retaining comments and all details of
whitespace is the specific use case for which I created pgen2.)

Some uses I have seen include using it to check that the code before and
after the formatting has no functional changes (both have the same ast) or
to augment the information obtained with other sources. But yeah, I agree
that static code analyzers and linters are a much bigger target.

>I wonder if lib2to3 is actually something that would benefit from moving
out of the stdlib. (Wasn't it on Amber's list?) As Łukasz points out in
that issue, it is outdated. Maybe if it was out of the stdlib it would
attract more contributors. Then again, I have recently started exploring
the idea of a PEG parser for Python. Maybe a revamped version of the core
of lib2to3 based on PEG instead of pgen would be interesting to some folks.

I was thinking more on the line of leveraging some parts lib2to3 having
some CST-related solution similar to the ast module, not exposing the whole
functionality of lib2to3. Basically, it would be a more high-level
abstraction to substitute the current parser module. Technically you should
be able to reconstruct some primitives that lib2to3 uses on top of the
output that the parser module generates (modulo some extra information from
the grammar), but the raw output that the parser module generates is not
super useful by itself, especially when you consider the maintenance costs.

On the other side, as you mention here:

>I am interested in switching CPython's parsing strategy to something else
(what exactly remains to be seen) and any new approach is unlikely to reuse
the current CST technology. (OTOH I think it would be wise to keep the
current AST.)

it is true that changing the parser can influence greatly the hypothetical
CST module so it may complicate the conversion to a new parser solution if
the API does not abstract enough (or it may be close to impractical
depending on the new parser solution).

My original suggestion was based on the fact that the parser module is not
super useful and it has a great maintenance cost, but the "realm" of what
it solves (providing access to the parse trees) could be useful to some use
cases so that is why I was talking about "parser" and lib2to3 in the same
email.

Perhaps we can be more productive if we focus on just deprecating the
"parser" module, but I thought it was an opportunity to solve two (related)
problems at once.


On Mon, 20 May 2019 at 17:28, Guido van Rossum  wrote:

> On Thu, May 16, 2019 at 3:51 PM Pablo Galindo Salgado 
> wrote:
>
>> [Nathaniel Smith]
>>
> >Will the folks using forks be happy to switch to the stdlib version?
>> >For example I can imagine that if black wants to process 3.7 input
>> >code while running on 3.6, it might prefer a parser on PyPI even if
>> >he stdlib version were public, since the PyPI version can be updated
>> >independently of the host Python.
>>
>> The tool can parse arbitrary grammars, the one that is packed into is
>> just one of them.
>>
>> I think it would be useful, among other things because the standard
>> library
>> lacks currently a proper CST solution. The ast module is heavily
>> leveraged for
>> things like formatters,
>>
>
> Actually, I think the `ast` module doesn't work very well for formatters,
> because it loses comments. (Retaining comments and all details of
> whitespace is the specific use case for which I created pgen2.)
>
>
>> static code analyzers...etc but CST can be very useful as
>> Łukasz describes here:
>>
>> https://bugs.python.org/issue7
>>
>> I think is missing an important gap in the stdlib and the closest thing
>> we have
>> (the current parser module) is not useful for any of that. Also, the core
>> to generating
>> the hypothetical new package (with some new API over it may be) is
>> already undocumented
>> as an implementation detail of lib2to3 (and some people are already using
>> it directly).
>>
>
> I wonder if lib2to3 is actually something that would benefit from moving
> out of the stdlib. (Wasn't it on Amber's list?) As Łukasz points out in
> that issue, it is outdated. Maybe if it was out of the stdlib it would
> attract more contributors. Then again, I have recently started exploring
> the idea of a PEG parser for Python. Maybe a revamped version of the core
> of lib2to3 based on PEG instead of pgen would be interesting to some folks.
>
> I do agree that the two versions of tokenize.py should be unified (and the
> result kept in the stdlib). However there are some issues here, because
> tokenize.py is closely tied to the names and numbers of the tokens, and the
> latter information is currently generated based on the contents of the
> Grammar file. This may get in the way of using it to tokenize old Python
> versions.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him/his **(why is my pronoun here?)*
> 

Re: [Python-Dev] Parser module in the stdlib

[Steven D'Aprano]

On Mon, May 20, 2019 at 08:55:59AM -0700, Guido van Rossum wrote:

> I am interested in switching CPython's parsing strategy to something else
> (what exactly remains to be seen)

Are you re-thinking the restriction to LL(1) grammars?


-- 
Steven
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Guido van Rossum]

On Mon, May 20, 2019 at 11:29 AM Steven D'Aprano 
wrote:

> On Mon, May 20, 2019 at 08:55:59AM -0700, Guido van Rossum wrote:
>
> > I am interested in switching CPython's parsing strategy to something else
> > (what exactly remains to be seen)
>
> Are you re-thinking the restriction to LL(1) grammars?
>

Indeed. I think it served us well for the first 10-15 years, but now it is
just a burden.

See a thread I started at Discourse:
https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379

And some followup there:
https://discuss.python.org/t/preparing-for-new-python-parsing/1550

Note that this is very much speculative.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] bpo-36919: Exception from 'compile' reports a newline char not present in input

[Pavel Koneski]

Hi,

The following issue arose when implementing IronPython 3. Normally the
behavior of CPython is authoritative, but in this case I started having
doubts.

Since Python 3.2, input in 'exec' mode  of 'compile' does not have to
end in a newline anymore. However, it creates a surprising behavior when
a 'SyntaxError' is reported:

>>> try: compile('try', '', 'exec')
... except SyntaxError as ex: print(repr(ex))
...
SyntaxError('invalid syntax', ('', 1, 4, 'try\n'))

The 'text' field of the exception thrown contains an additional newline
character that was not present in the input. Is it:

a. Proper Python language behavior?
b. CPython implementation artifact?
c. A bug?

The comments on bpo (https://bugs.python.org/issue36919) suggest Case B,
but inconclusive. If case B it is, since IronPython does not exhibit
this behavior, I will submit a patch to StdLib test(s) to make them
implementation independent.

Thanks for comments,
Pavel

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 587 "Python Initialization Configuration" version 4

[Antoine Pitrou]

Hi,

- Since PyInitError can be "ok", I'd rather call it "PyInitStatus".

- The PyConfig example sets "module_search_paths" but not
  "module_search_paths_set".  Is it an error?

- "PyConfig_InitPythonConfig" vs "PyConfig_InitIsolatedConfig": why not
  "PyConfig_InitConfig" and "PyConfig_InitIsolatedConfig"?

- Why is there an "isolated" field in addition to the "IsolatedConfig"
  functions?

- PyConfig.isolated is not documented.

- "Configuration files are still used with this configuration": why is
  this?  Shouldn't they be ignored in an isolated configuration?

- In "Comparison of Python and Isolated Configurations": what does -1
  mean here?

Overall, this looks like a great improvement.  My only worry is that
interactions between the various options seem complicated and
difficult to understand.  Perhaps we will need some detailed
documentation and examples.

Regards

Antoine.



On Mon, 20 May 2019 14:05:42 +0200
Victor Stinner  wrote:
> Hi,
> 
> I expected the version 3 of my PEP to be complete, but Gregory Szorc
> and Steve Dower spotted a few more issues ;-)
> 
> The main change of the version 4 is the introduction of "Python
> Configuration" and "Isolated Configuration" default configuration
> which are well better defined.
> 
> The new "Isolated Configuration" provides sane default values to
> isolate Python from the system. For example, to embed Python into an
> application. Using the environment are now opt-in options, rather than
> an opt-out options. For example, environment variables, command line
> arguments and global configuration variables are ignored by default.
> 
> Building a customized Python which behaves as the regular Python
> becomes easier using the new Py_RunMain() function. Moreover, using
> the "Python Configuration", PyConfig.argv arguments are now parsed the
> same way the regular Python parses command line arguments, and
> PyConfig.xoptions are handled as -X opt command line options.
> 
> I replaced all macros with functions. Macros can cause issues when
> used from different programming languages, whereas functions are
> always well supported.
> 
> PyPreConfig structure doesn't allocate memory anymore (the allocator
> field becomes an integer, instead of a string). I removed the
> "Constant PyConfig" special case which introduced too many exceptions
> for little benefit.
> 
> See the "Version History" section for the full changes.
> 
> HTML version:
> https://www.python.org/dev/peps/pep-0587/
> 
> Full text below.
> 
> Victor
> 
> 
> PEP: 587
> Title: Python Initialization Configuration
> Author: Victor Stinner , Nick Coghlan 
> 
> BDFL-Delegate: Thomas Wouters 
> Discussions-To: python-dev@python.org
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 27-Mar-2019
> Python-Version: 3.8
> 
> Abstract
> 
> 
> Add a new C API to configure the Python Initialization providing finer
> control on the whole configuration and better error reporting.
> 
> It becomes possible to read the configuration and then override some
> computed parameters before it is applied. It also becomes possible to
> completely override how Python computes the module search paths
> (``sys.path``).
> 
> The new `Isolated Configuration`_ provides sane default values to
> isolate Python from the system. For example, to embed Python into an
> application. Using the environment are now opt-in options, rather than
> an opt-out options. For example, environment variables, command line
> arguments and global configuration variables are ignored by default.
> 
> Building a customized Python which behaves as the regular Python becomes
> easier using the new ``Py_RunMain()`` function. Moreover, using the
> `Python Configuration`_, ``PyConfig.argv`` arguments are now parsed the
> same way the regular Python parses command line arguments, and
> ``PyConfig.xoptions`` are handled as ``-X opt`` command line options.
> 
> This extracts a subset of the API design from the PEP 432 development
> and refactoring work that is now considered sufficiently stable to make
> public (allowing 3rd party embedding applications access to the same
> configuration APIs that the native CPython CLI is now using).
> 
> 
> Rationale
> =
> 
> Python is highly configurable but its configuration evolved organically.
> The initialization configuration is scattered all around the code using
> different ways to set them: global configuration variables (ex:
> ``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``),
> command line arguments (ex: ``-b``), configuration files (ex:
> ``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A
> straightforward and reliable way to configure Python is needed.
> 
> Some configuration parameters are not accessible from the C API, or not
> easily. For example, there is no API to override the default values of
> ``sys.executable``.
> 
> Some options like ``PYTHONPATH`` can only be set using an environment
> variable which has a side effect on Pytho

Re: [Python-Dev] bpo-36919: Exception from 'compile' reports a newline char not present in input

[Guido van Rossum]

I answered (B) in the tracker.

On Mon, May 20, 2019 at 12:32 PM Pavel Koneski 
wrote:

> Hi,
>
> The following issue arose when implementing IronPython 3. Normally the
> behavior of CPython is authoritative, but in this case I started having
> doubts.
>
> Since Python 3.2, input in 'exec' mode  of 'compile' does not have to
> end in a newline anymore. However, it creates a surprising behavior when
> a 'SyntaxError' is reported:
>
> >>> try: compile('try', '', 'exec')
> ... except SyntaxError as ex: print(repr(ex))
> ...
> SyntaxError('invalid syntax', ('', 1, 4, 'try\n'))
>
> The 'text' field of the exception thrown contains an additional newline
> character that was not present in the input. Is it:
>
> a. Proper Python language behavior?
> b. CPython implementation artifact?
> c. A bug?
>
> The comments on bpo (https://bugs.python.org/issue36919) suggest Case B,
> but inconclusive. If case B it is, since IronPython does not exhibit
> this behavior, I will submit a patch to StdLib test(s) to make them
> implementation independent.
>
> Thanks for comments,
> Pavel
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

Hi,

here is the first version of my PEP 594 to deprecate and eventually remove 
modules from the standard library. The PEP started last year with talk during 
the Python Language Summit 2018, https://lwn.net/Articles/755229/.

The PEP can be confirmed in two stages. I'm not planning any code changes for 
3.8. Instead I only like to document a bunch of modules as deprecated. Active 
deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase 
gives us 3 years to change our minds or handle edge cases, too.

Regards,
Christian



PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes 
Status: Active
Type: Process
Content-Type: text/x-rst
Created: 20-May-2019
Post-History:


Abstract


This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats and APIs that
have been superseded a long time ago, e.g. Mac OS 9 and Commodore.


Rationale
=

Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often refrained to as "batteries included"
philosophy and was one of the corner stones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.

Times have changed. The introduction of the cheese shop (PyPI), setuptools,
and later pip, it became simple and straight forward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.

On the other hand, Python's standard library is piling up cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.

* Any additional module increases the maintenance cost for the Python core
  development team. The team has limited resources, reduced maintenance cost
  frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
  de-facto solution for a problem. A majority of users only pick 3rd party
  modules to replace a stdlib module, when they have a compelling reason, e.g.
  lxml instead of `xml`. The removal of an unmaintained stdlib module
  increases the chances of a community contributed module to become widely
  used.
* A lean and mean standard library benefits platforms with limited resources
  like devices with just a few hundred kilobyte of storage (e.g. BBC
  Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
  (e.g. pyodide) also benefit from reduced download size.

The modules in the PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example
least controversial are 30 years old multimedia formats like ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980ties. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.

This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be cooperate environments or class rooms where external
code is not permitted without legal approval.

* The usage of FTP is declining, but some files are still provided over
  the FTP protocol or hosters offer FTP to upload content. Therefore
  ``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` module are widely used. They are mature
  modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
  kids and can make crazy sounds. Making a computer generate crazy sounds is
  powerful and highly motivating exercise for a 9yo aspiring developer. It's
  a fun battery to keep.


Deprecation schedule


3.8
---

This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released
a few months before Python 2.7 will reach its end of lifetime. We expect that
Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and
2020. To reduce churn and to allow a smooth transition from Python 2,
Python 3.8 will neither raise `DeprecationWarning` nor remove any
modules that have been scheduled for removal. Instead deprecated modules will
just be *documented* as deprecated. Optionally modules may emit a
`PendingDeprecationWarning`.

All deprecated modules will also undergo a feature freeze. No additional
features should be added. Bug should still be fixed.

3.9
---

Starting with Python 3.9, deprecated modules will start issuing
`DeprecationWarning`.


3.10


In 3.10

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Andrew Svetlov]

socketserver.py is also questionable

On Mon, May 20, 2019 at 11:15 PM Christian Heimes  wrote:
>
> Hi,
>
> here is the first version of my PEP 594 to deprecate and eventually remove 
> modules from the standard library. The PEP started last year with talk during 
> the Python Language Summit 2018, https://lwn.net/Articles/755229/.
>
> The PEP can be confirmed in two stages. I'm not planning any code changes for 
> 3.8. Instead I only like to document a bunch of modules as deprecated. Active 
> deprecation is planned for 3.9 and removal for 3.10. The long deprecation 
> phase gives us 3 years to change our minds or handle edge cases, too.
>
> Regards,
> Christian
>
>
> 
> PEP: 594
> Title: Removing dead batteries from the standard library
> Author: Christian Heimes 
> Status: Active
> Type: Process
> Content-Type: text/x-rst
> Created: 20-May-2019
> Post-History:
>
>
> Abstract
> 
>
> This PEP proposed a list of standard library modules to be removed from the
> standard library. The modules are mostly historic data formats and APIs that
> have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
>
>
> Rationale
> =
>
> Back in the early days of Python, the interpreter came with a large set of
> useful modules. This was often refrained to as "batteries included"
> philosophy and was one of the corner stones to Python's success story.
> Users didn't have to figure out how to download and install separate
> packages in order to write a simple web server or parse email.
>
> Times have changed. The introduction of the cheese shop (PyPI), setuptools,
> and later pip, it became simple and straight forward to download and install
> packages. Nowadays Python has a rich and vibrant ecosystem of third party
> packages. It's pretty much standard to either install packages from PyPI or
> use one of the many Python or Linux distributions.
>
> On the other hand, Python's standard library is piling up cruft, unnecessary
> duplication of functionality, and dispensable features. This is undesirable
> for several reasons.
>
> * Any additional module increases the maintenance cost for the Python core
>   development team. The team has limited resources, reduced maintenance cost
>   frees development time for other improvements.
> * Modules in the standard library are generally favored and seen as the
>   de-facto solution for a problem. A majority of users only pick 3rd party
>   modules to replace a stdlib module, when they have a compelling reason, e.g.
>   lxml instead of `xml`. The removal of an unmaintained stdlib module
>   increases the chances of a community contributed module to become widely
>   used.
> * A lean and mean standard library benefits platforms with limited resources
>   like devices with just a few hundred kilobyte of storage (e.g. BBC
>   Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
>   (e.g. pyodide) also benefit from reduced download size.
>
> The modules in the PEP have been selected for deprecation because their
> removal is either least controversial or most beneficial. For example
> least controversial are 30 years old multimedia formats like ``sunau``
> audio format, which was used on SPARC and NeXT workstations in the late
> 1980ties. The ``crypt`` module has fundamental flaws that are better solved
> outside the standard library.
>
> This PEP also designates some modules as not scheduled for removal. Some
> modules have been deprecated for several releases or seem unnecessary at
> first glance. However it is beneficial to keep the modules in the standard
> library, mostly for environments where installing a package from PyPI is not
> an option. This can be cooperate environments or class rooms where external
> code is not permitted without legal approval.
>
> * The usage of FTP is declining, but some files are still provided over
>   the FTP protocol or hosters offer FTP to upload content. Therefore
>   ``ftplib`` is going to stay.
> * The ``optparse`` and ``getopt`` module are widely used. They are mature
>   modules with very low maintenance overhead.
> * According to David Beazley [5]_ the ``wave`` module is easy to teach to
>   kids and can make crazy sounds. Making a computer generate crazy sounds is
>   powerful and highly motivating exercise for a 9yo aspiring developer. It's
>   a fun battery to keep.
>
>
> Deprecation schedule
> 
>
> 3.8
> ---
>
> This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released
> a few months before Python 2.7 will reach its end of lifetime. We expect that
> Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and
> 2020. To reduce churn and to allow a smooth transition from Python 2,
> Python 3.8 will neither raise `DeprecationWarning` nor remove any
> modules that have been scheduled for removal. Instead deprecated modules will
> just be *documented* as deprecated. Optionally modu

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 20/05/2019 23.12, Andrew Svetlov wrote:
> socketserver.py is also questionable

I briefly though about the module, but didn't consider it for removal. The 
http.server, xmlrpc.server, and logging configuration server are implemented on 
top of the socketserver. I don't want to remove the socketserver module without 
a suitable replacement for http.server in the standard library.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Antoine Pitrou]

NNTP is still quite used (often through GMane, but probably not only) so
I'd question the removal of nntplib.

cgitb used to be used by some Web frameworks in order to format
exceptions.  Perhaps one should check if that's still the case.

If the wave module depends on the audioop module, and if the wave
module is kept in the stdlib, then the audioop module can't be removed.

Removing the crypt module would remove support for system-standard
password files.  I don't understand the rationale.

Regards

Antoine.


On Mon, 20 May 2019 22:15:22 +0200
Christian Heimes  wrote:
> Hi,
> 
> here is the first version of my PEP 594 to deprecate and eventually remove 
> modules from the standard library. The PEP started last year with talk during 
> the Python Language Summit 2018, https://lwn.net/Articles/755229/.
> 
> The PEP can be confirmed in two stages. I'm not planning any code changes for 
> 3.8. Instead I only like to document a bunch of modules as deprecated. Active 
> deprecation is planned for 3.9 and removal for 3.10. The long deprecation 
> phase gives us 3 years to change our minds or handle edge cases, too.
> 
> Regards,
> Christian
> 
> 
> 
> PEP: 594
> Title: Removing dead batteries from the standard library
> Author: Christian Heimes 
> Status: Active
> Type: Process
> Content-Type: text/x-rst
> Created: 20-May-2019
> Post-History:
> 
> 
> Abstract
> 
> 
> This PEP proposed a list of standard library modules to be removed from the
> standard library. The modules are mostly historic data formats and APIs that
> have been superseded a long time ago, e.g. Mac OS 9 and Commodore.
> 
> 
> Rationale
> =
> 
> Back in the early days of Python, the interpreter came with a large set of
> useful modules. This was often refrained to as "batteries included"
> philosophy and was one of the corner stones to Python's success story.
> Users didn't have to figure out how to download and install separate
> packages in order to write a simple web server or parse email.
> 
> Times have changed. The introduction of the cheese shop (PyPI), setuptools,
> and later pip, it became simple and straight forward to download and install
> packages. Nowadays Python has a rich and vibrant ecosystem of third party
> packages. It's pretty much standard to either install packages from PyPI or
> use one of the many Python or Linux distributions.
> 
> On the other hand, Python's standard library is piling up cruft, unnecessary
> duplication of functionality, and dispensable features. This is undesirable
> for several reasons.
> 
> * Any additional module increases the maintenance cost for the Python core
>   development team. The team has limited resources, reduced maintenance cost
>   frees development time for other improvements.
> * Modules in the standard library are generally favored and seen as the
>   de-facto solution for a problem. A majority of users only pick 3rd party
>   modules to replace a stdlib module, when they have a compelling reason, e.g.
>   lxml instead of `xml`. The removal of an unmaintained stdlib module
>   increases the chances of a community contributed module to become widely
>   used.
> * A lean and mean standard library benefits platforms with limited resources
>   like devices with just a few hundred kilobyte of storage (e.g. BBC
>   Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
>   (e.g. pyodide) also benefit from reduced download size.
> 
> The modules in the PEP have been selected for deprecation because their
> removal is either least controversial or most beneficial. For example
> least controversial are 30 years old multimedia formats like ``sunau``
> audio format, which was used on SPARC and NeXT workstations in the late
> 1980ties. The ``crypt`` module has fundamental flaws that are better solved
> outside the standard library.
> 
> This PEP also designates some modules as not scheduled for removal. Some
> modules have been deprecated for several releases or seem unnecessary at
> first glance. However it is beneficial to keep the modules in the standard
> library, mostly for environments where installing a package from PyPI is not
> an option. This can be cooperate environments or class rooms where external
> code is not permitted without legal approval.
> 
> * The usage of FTP is declining, but some files are still provided over
>   the FTP protocol or hosters offer FTP to upload content. Therefore
>   ``ftplib`` is going to stay.
> * The ``optparse`` and ``getopt`` module are widely used. They are mature
>   modules with very low maintenance overhead.
> * According to David Beazley [5]_ the ``wave`` module is easy to teach to
>   kids and can make crazy sounds. Making a computer generate crazy sounds is
>   powerful and highly motivating exercise for a 9yo aspiring developer. It's
>   a fun battery to keep.
> 
> 
> Deprecation schedule
> 
> 
> 3.8
> ---
> 
> Thi

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 20/05/2019 23.27, Antoine Pitrou wrote:
> NNTP is still quite used (often through GMane, but probably not only) so
> I'd question the removal of nntplib.

Is NNTP support important enough to keep the module in the standard library?

> cgitb used to be used by some Web frameworks in order to format
> exceptions.  Perhaps one should check if that's still the case.

A search on github did not reveal any relevant use of cgitb besides tons of 
copies of test_cgitb and an optional debugging middleware in Paste. I checked 
Django, Plone, CherryPy, flask, and bottle. None uses cgitb.

> If the wave module depends on the audioop module, and if the wave
> module is kept in the stdlib, then the audioop module can't be removed.

No, it can be removed. I explained the situation in the "wave" section of the 
PEP.

> Removing the crypt module would remove support for system-standard
> password files.  I don't understand the rationale.

Applications *must* not access system-standard password files directly. On any 
sanely and securely configured systems, application cannot even access system 
password files like /etc/shadow. Access restrictions and system security 
policies will prevent read access. Also applications cannot assume that users 
are present in any user file. They may come from LDAP, SSSD, ActiveDirectory, 
or other sources.

The correct way to interact with system users is to use the proper APIs, that 
are NSS (name service switch) and PAM (pluggable authentication modules). NSS 
looks up and enumerate users and groups. PAM performs password validation and 
much, much, much more. The pwd and grp modules use the correct APIs to interact 
with NSS. If you need to check or change passwords, you must go through PAM.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Jeff Kintscher]

What replacements are available for NNTP? All I could find was pynntp, 
which had a single release 6 years ago.


https://github.com/greenbender/pynntp

//Jeff

On 5/20/19 2:27 PM, Antoine Pitrou wrote:

NNTP is still quite used (often through GMane, but probably not only) so
I'd question the removal of nntplib.

cgitb used to be used by some Web frameworks in order to format
exceptions.  Perhaps one should check if that's still the case.

If the wave module depends on the audioop module, and if the wave
module is kept in the stdlib, then the audioop module can't be removed.

Removing the crypt module would remove support for system-standard
password files.  I don't understand the rationale.

Regards

Antoine.


On Mon, 20 May 2019 22:15:22 +0200
Christian Heimes  wrote:

Hi,

here is the first version of my PEP 594 to deprecate and eventually remove 
modules from the standard library. The PEP started last year with talk during 
the Python Language Summit 2018, https://lwn.net/Articles/755229/.

The PEP can be confirmed in two stages. I'm not planning any code changes for 
3.8. Instead I only like to document a bunch of modules as deprecated. Active 
deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase 
gives us 3 years to change our minds or handle edge cases, too.

Regards,
Christian



PEP: 594
Title: Removing dead batteries from the standard library
Author: Christian Heimes 
Status: Active
Type: Process
Content-Type: text/x-rst
Created: 20-May-2019
Post-History:


Abstract


This PEP proposed a list of standard library modules to be removed from the
standard library. The modules are mostly historic data formats and APIs that
have been superseded a long time ago, e.g. Mac OS 9 and Commodore.


Rationale
=

Back in the early days of Python, the interpreter came with a large set of
useful modules. This was often refrained to as "batteries included"
philosophy and was one of the corner stones to Python's success story.
Users didn't have to figure out how to download and install separate
packages in order to write a simple web server or parse email.

Times have changed. The introduction of the cheese shop (PyPI), setuptools,
and later pip, it became simple and straight forward to download and install
packages. Nowadays Python has a rich and vibrant ecosystem of third party
packages. It's pretty much standard to either install packages from PyPI or
use one of the many Python or Linux distributions.

On the other hand, Python's standard library is piling up cruft, unnecessary
duplication of functionality, and dispensable features. This is undesirable
for several reasons.

* Any additional module increases the maintenance cost for the Python core
   development team. The team has limited resources, reduced maintenance cost
   frees development time for other improvements.
* Modules in the standard library are generally favored and seen as the
   de-facto solution for a problem. A majority of users only pick 3rd party
   modules to replace a stdlib module, when they have a compelling reason, e.g.
   lxml instead of `xml`. The removal of an unmaintained stdlib module
   increases the chances of a community contributed module to become widely
   used.
* A lean and mean standard library benefits platforms with limited resources
   like devices with just a few hundred kilobyte of storage (e.g. BBC
   Micro:bit). Python on mobile platforms like BeeWare or WebAssembly
   (e.g. pyodide) also benefit from reduced download size.

The modules in the PEP have been selected for deprecation because their
removal is either least controversial or most beneficial. For example
least controversial are 30 years old multimedia formats like ``sunau``
audio format, which was used on SPARC and NeXT workstations in the late
1980ties. The ``crypt`` module has fundamental flaws that are better solved
outside the standard library.

This PEP also designates some modules as not scheduled for removal. Some
modules have been deprecated for several releases or seem unnecessary at
first glance. However it is beneficial to keep the modules in the standard
library, mostly for environments where installing a package from PyPI is not
an option. This can be cooperate environments or class rooms where external
code is not permitted without legal approval.

* The usage of FTP is declining, but some files are still provided over
   the FTP protocol or hosters offer FTP to upload content. Therefore
   ``ftplib`` is going to stay.
* The ``optparse`` and ``getopt`` module are widely used. They are mature
   modules with very low maintenance overhead.
* According to David Beazley [5]_ the ``wave`` module is easy to teach to
   kids and can make crazy sounds. Making a computer generate crazy sounds is
   powerful and highly motivating exercise for a 9yo aspiring developer. It's
   a fun battery to keep.


Deprecation schedule
=

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Antoine Pitrou]

On Tue, 21 May 2019 00:06:35 +0200
Christian Heimes  wrote:
> On 20/05/2019 23.27, Antoine Pitrou wrote:
> > NNTP is still quite used (often through GMane, but probably not only) so
> > I'd question the removal of nntplib.  
> 
> Is NNTP support important enough to keep the module in the standard library?

I'd phrase the question differently: is NNTP dead enough, or nntplib
painful enough to maintain, that it's worth removing it from the stdlib?

If the stdlib didn't have NNTP support, obviously nobody would suggest
adding it nowadays.  But it has that support, and there are certainly
uses of it in the wild, so we must take that into account.

> > If the wave module depends on the audioop module, and if the wave
> > module is kept in the stdlib, then the audioop module can't be removed.  
> 
> No, it can be removed. I explained the situation in the "wave" section of the 
> PEP.

My bad.  I had skipped that.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Simon Cross]

Woot. +100 on this PEP.

> If the stdlib didn't have NNTP support, obviously nobody would suggest
> adding it nowadays.

Perhaps this is a good reason to keep nntplib in the deprecation list?
Another question is "are there any places using nntplib where `pip install
nntplib`" is not an reasonable option? There's quite a lot of time before
3.10 to move nntplib into an outside package.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Antoine Pitrou]

On Tue, 21 May 2019 00:41:19 +0200
Simon Cross  wrote:
> Woot. +100 on this PEP.
> 
> > If the stdlib didn't have NNTP support, obviously nobody would suggest
> > adding it nowadays.  
> 
> Perhaps this is a good reason to keep nntplib in the deprecation list?

No, because the same applies to getopt, optparse and others, which are
not in the deprecation list.

> Another question is "are there any places using nntplib where `pip install
> nntplib`" is not an reasonable option?

You could ask pretty much the same question about most stdlib packages.
Isn't "pip install ftplib" a reasonable option?  How about "pip
install email" or "pip install xmlrpc"?

The PEP is about "dead batteries".  Not "batteries that some people
think would be reasonable to install from PyPI".

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 00.13, Antoine Pitrou wrote:
> On Tue, 21 May 2019 00:06:35 +0200
> Christian Heimes  wrote:
>> On 20/05/2019 23.27, Antoine Pitrou wrote:
>>> NNTP is still quite used (often through GMane, but probably not only) so
>>> I'd question the removal of nntplib.  
>>
>> Is NNTP support important enough to keep the module in the standard library?
> 
> I'd phrase the question differently: is NNTP dead enough, or nntplib
> painful enough to maintain, that it's worth removing it from the stdlib?

The module itself does not create much work. But its tests are a regular source 
of pain and instabilities. The tests for nntplib depend on external NNTP 
servers. These servers are sometimes down, very slow, or don't work over IPv6. 
I'm sure that Pablo and Victor can tell you some war stories. I briefly 
mentioned the issues in the PEP, too.

> If the stdlib didn't have NNTP support, obviously nobody would suggest
> adding it nowadays.  But it has that support, and there are certainly
> uses of it in the wild, so we must take that into account.
> 
>>> If the wave module depends on the audioop module, and if the wave
>>> module is kept in the stdlib, then the audioop module can't be removed.  
>>
>> No, it can be removed. I explained the situation in the "wave" section of 
>> the PEP.
> 
> My bad.  I had skipped that.

The section in audioop was confusing. I have updated the audioop paragraph and 
the crypt paragraph with your feedback. I'll keep the PR 
https://github.com/python/peps/pull/1063 option for a couple of days to collect 
more feedback.

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Terry Reedy]

On 5/20/2019 11:55 AM, Guido van Rossum wrote:
On Thu, May 16, 2019 at 3:57 PM Steve Dower > wrote:


[...]
We still have the policy of not removing modules that exist in the
Python 2 standard library. But 3.9 won't be covered by that :)

I didn't even remember that. Where's that written down?


AFAIK, there has been no BDFL pronouncement or coredev vote.  But it has 
become somewhat of a consensus as a result of discussions in multliple 
threads about removing modules going back to the 3.4 era.


PEP 594, just posted, proposes to codify a 'soft' (or perhaps 'slow') 
version of the policy: doc deprecation, code deprecation, and code 
removal in 3.8, 3.9, 3.10.
FWIW I am strongly in favor of getting rid of the `parser` module, in 
3.8 if we can, otherwise in 3.9 (after strong deprecation in 3.8).


You could request that 'parser' be specifically excluded from the PEP, 
along with 'distutils' (and effectively anything else not specifically 
named).  Or request that it be included, but with an accelerated 
schedule.  I have a vague idea of why you think it harmful to keep it 
around, but a reminder would not hurt ;-).


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Terry Reedy]

On 5/20/2019 6:06 PM, Christian Heimes wrote:


Removing the crypt module would remove support for system-standard
password files.  I don't understand the rationale.


Applications *must* not access system-standard password files directly. On any 
sanely and securely configured systems, application cannot even access system 
password files like /etc/shadow. Access restrictions and system security 
policies will prevent read access. Also applications cannot assume that users 
are present in any user file. They may come from LDAP, SSSD, ActiveDirectory, 
or other sources.

The correct way to interact with system users is to use the proper APIs, that 
are NSS (name service switch) and PAM (pluggable authentication modules). NSS 
looks up and enumerate users and groups. PAM performs password validation and 
much, much, much more. The pwd and grp modules use the correct APIs to interact 
with NSS. If you need to check or change passwords, you must go through PAM.


Add this to the PEP?  It might suggest that crypt should go away sooner.


--
Terry Jan Reedy

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Christian Heimes]

On 16/05/2019 23.12, Pablo Galindo Salgado wrote:
> Hi everyone,
> 
> TLDR
> =
> 
> I propose to remove the current parser module and expose pgen2 as a standard 
> library module.

I like to add this to PEP 594, see https://github.com/python/peps/pull/1063

Terry, thanks for connecting my PEP with Pablo's proposal.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Christian Heimes]

On 21/05/2019 01.06, Terry Reedy wrote:
> On 5/20/2019 6:06 PM, Christian Heimes wrote:
> 
>>> Removing the crypt module would remove support for system-standard
>>> password files.  I don't understand the rationale.
>>
>> Applications *must* not access system-standard password files directly. On 
>> any sanely and securely configured systems, application cannot even access 
>> system password files like /etc/shadow. Access restrictions and system 
>> security policies will prevent read access. Also applications cannot assume 
>> that users are present in any user file. They may come from LDAP, SSSD, 
>> ActiveDirectory, or other sources.
>>
>> The correct way to interact with system users is to use the proper APIs, 
>> that are NSS (name service switch) and PAM (pluggable authentication 
>> modules). NSS looks up and enumerate users and groups. PAM performs password 
>> validation and much, much, much more. The pwd and grp modules use the 
>> correct APIs to interact with NSS. If you need to check or change passwords, 
>> you must go through PAM.
> 
> Add this to the PEP?  It might suggest that crypt should go away sooner.

Yes, I'll do that. I'm currently collecting updates from feedback in PR 
https://github.com/python/peps/pull/1063

Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Inada Naoki]

I use fileinput for several times per year.

fileinput is handy tool to write single script file to analyze log files.

* In such tools, I don't need real argument parser.
* Some log files are compressed and some are not.
  It seems argparse doesn't support transparent decompression.
* I don't want to use 3rd party library for such single script files.

I'd like to add transparent decompression to argparse if fileinput
is deprecated.

Regards,

-- 
Inada Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 594: Removing dead batteries from the standard library

[Guido van Rossum]

Yeah, I was surprised to see fileinput on the list. It's indeed a handy
little tool.

On Mon, May 20, 2019 at 5:19 PM Inada Naoki  wrote:

> I use fileinput for several times per year.
>
> fileinput is handy tool to write single script file to analyze log files.
>
> * In such tools, I don't need real argument parser.
> * Some log files are compressed and some are not.
>   It seems argparse doesn't support transparent decompression.
> * I don't want to use 3rd party library for such single script files.
>
> I'd like to add transparent decompression to argparse if fileinput
> is deprecated.
>
> Regards,
>
> --
> Inada Naoki  
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him/his **(why is my pronoun here?)*

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com