[Python-Dev] Worried about Python release schedule and lack of stable C-API

[Stephen J. Turnbull]

jack.jan...@cwi.nl writes:

 > I’m getting increasingly worried about the future of Python,

You mean about the fact that Python has a plethora (rapidly growing!) 
of powerful, efficient extension packages and you want to use them
all (perhaps recursively)? :-)  I really don't think this bodes ill
for the future of Python.

Perhaps it bodes poorly for your development practices (and I'll try
to tease out some ideas where we can improve the infrastructure for
you), but there are plenty of applications where this plethora works
really well, too.

 > and those worries have been exacerbated by the new yearly release
 > rhythm, and the lack of a stable C-API that is full enough to
 > entice extension writers to use it.

I don't think "full API" is the problem (cf. Phil Thompson's post[1],
where the stable API is good enough for PyQt).  I think the problem is
more likely that in initial development, they simply targeted the .h
files in front of them.[2]  Since then they haven't needed to update,
and *neither has anyone else with resources or pull with with that
package's devs*.  Suggestion: Perhaps the documentation can be
improved to make the stable API easier to conform to from the get-go.
(I think the stable API is already distinguished in the .h files.)

If "full API" *is* the problem, I doubt it's a solvable problem (except
by extension users lobbying extension developers to use the stable
API), because the stable API is a deliberate compromise between
efficient communication between CPython and extensions, and
flexibility for CPython to improve its internal implementation.  If
extension writers *deliberately* eschew the stable API, presumably
they need to access private APIs and if those change, too bad.

 > And I also fully understand that some other developer who creates a
 > new package that is essential to my application only targets the
 > current Python release,

This is where developers in your position can have the most impact, I
think.  When there's a new gotta-have extension, tell the extension's
developers you need support for the stable API or ABI, or you can't
use their package.

 > And that is assuming you have the time and are smart enough to back
 > port the new package to the old Python release, or the old package
 > to the new Python release (and for the latter there’s probably a
 > good reason why the developers haven’t done so already).

Most frequently, as you pointed out, the reason is that they're busy
with other priorities, and it's just not their itch to scratch.

 > I think we really need to come up with some scheme whereby
 > extension packages become more long-lived than a single Python
 > release...

I would guess that a lot of extension packages are quite long-lived.
First, as Phil Thompson points out for PyQt[1], the content of the
wheel may very well work fine, but they have to actually make the
wheels with the appropriate name (or maybe other metadata?), or they
won't install in the new Python version's site-packages.  I think this
is something that Python (distutils-sig) could do something about,
either by adding a way to make multi-version wheels, or providing a
utility to retarget an existing wheel (maybe mv(1) is enough?!)

Second, I'm sure there are plenty of extensions that can be ported to
either new CPython versions or (preferably) to the stable API or ABI
with minimal effort.  Perhaps there could be a SIG or a channel in the
tracker where people could request these ports.  I'm suggesting this
because of the "better things to do problem".  Obviously, if the
extension itself is unmaintained, its maintainer will do nothing.
Even if it is maintained, it seems that its dev community does not
prioritize these ports.  But new developers looking for something to
do might see it in "core" channels.  Probably there are plenty of
people around who can help them build wheels, and I believe a
significant fraction of extensions without current version wheels will
require little or no C changes.  (OK, this doesn't involve a merge
request to CPython, so it doesn't immediately set them on the path to
core dev status, but we can award gold stars.)


Footnotes: 
[1]  


[2]  Maybe that's just me?  OK, I'm busted!

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BKAZWGLTLOZ6BJDC3KAYBWTITFO4BQN4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Ronald Oussoren via Python-Dev]

> On 26 Sep 2021, at 19:03, Christian Heimes  wrote:
> 
> On 26/09/2021 13.07, jack.jan...@cwi.nl wrote:
>> The problem with the stable ABI is that very few developers are targeting 
>> it. I’m not sure why not, whether it has to do with incompleteness of the 
>> ABI, or with issues targeting it easily and your builds and then having 
>> pip/PyPI do the right things with wheels and all that. I’ve been on the 
>> capi-sig mailing list since its inception in 2007, but the discussions are 
>> really going over my head. I don’t understand what the problems are that 
>> keep people from targeting the stable ABI (or the various other attempts at 
>> standardising extensions over Python versions).
> 
> It takes some effort to port old extensions to stable ABI. Several old APIs 
> are not supported in stable ABI extensions. For example developers have to 
> port static type definitions to heap types. It's not complicated, but it 
> takes some effort.

The stable ABI is also not complete, although it should be complete enough for 
a lot of projects.  A, fairly esoteric, issue I ran into is that it is 
currently not possible to define a class with a non-default meta class using 
the type-spec API (AFAIK), see #15870.

And as you write “it takes some effort”, that alone likely reduces the amount 
of projects that migrate to the stable ABI esp. for projects that already have 
a CI/CD setup that creates binary wheels for you (for example using 
cibuildwheel). 

Ronald
—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFLOCVRO2UP2PKWTFUTAQ3EKO6NLMG4E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] The Default for python -X frozen_modules.

[Eric Snow]

We've frozen most of the stdlib modules imported during "python -c
pass" [1][2], to make startup a bit faster.  Import of those modules
is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
to "off" but we'd like to default to "on".  The blocker is the impact
on contributors.  I expect many will make changes to a stdlib module
and then puzzle over why those changes aren't getting used.  That's an
annoyance we can avoid, which is the point of this thread.

Possible solutions:

1. always default to "on" (the annoyance for contributors isn't big enough?)
2. default to "on" if it's a PGO build (and "off" otherwise)
3. default to "on" unless running from the source tree

Thoughts?

-eric


[1] https://bugs.python.org/issue45020
[2] FWIW, we may end up also freezing the modules imported for "python
-m ...", along with some other commonly used modules (like argparse).
That is a separate discussion.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Chris Angelico]

On Tue, Sep 28, 2021 at 2:58 AM Eric Snow  wrote:
>
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?

When exactly does the freezing happen?

It's only a single data point, but when I'm tinkering with the stdlib
itself, I'm always running from the source tree. So option 3 seems
quite viable.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QIIXTPTVOIWUBSEC33QWUO73UNTKCEZA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric Snow]

On Mon, Sep 27, 2021 at 11:09 AM Chris Angelico  wrote:
> When exactly does the freezing happen?

When you build the executable (e.g. "make -j8",
".\PCbuild\build.bat").  So your changes to those .py files wouldn't
show up until then.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IGDYUDVFHDU77OLPP3744FIG3IHZWS4D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric Snow]

On Mon, Sep 27, 2021 at 10:51 AM Eric Snow  wrote:
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree

FWIW, I'm planning on doing (2) (and (3) if it isn't complicated).
Mostly I wanted to verify my assumptions about the possible annoyance
before getting too far.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J656PLVTGTVDCLV2GSZPNV46UTKU4S7M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Chris Angelico]

On Tue, Sep 28, 2021 at 3:21 AM Eric Snow  wrote:
>
> On Mon, Sep 27, 2021 at 11:09 AM Chris Angelico  wrote:
> > When exactly does the freezing happen?
>
> When you build the executable (e.g. "make -j8",
> ".\PCbuild\build.bat").  So your changes to those .py files wouldn't
> show up until then.
>

Ah, gotcha. Then I would say either #1 or #3 would be entirely
acceptable to me, with preference to #3.

Hmm. A thought: What happens if I run "make", then edit one of the
files, then "make install"? Will it notice that the frozen version is
out of date and rebuild it?

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XAFLP6BZYNDGQHJHCWCY7BZW3Z3FXALJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Steve Dower]

On 9/27/2021 5:51 PM, Eric Snow wrote:

Possible solutions:

1. always default to "on" (the annoyance for contributors isn't big enough?)
2. default to "on" if it's a PGO build (and "off" otherwise)


Just to air my concerns regarding option 2 (which I've already spoken to 
Eric about): I feel this option is completely orthogonal to whether PGO 
is used or not, and ought to be discoverable independently.


Essentially, it should be its own configure-time option, and should be 
included somewhere in sysconfig.get_config_var(...).


If we went with #2, there's no reliable way to detect whether 
profile-guided optimisations were used on all CPython builds, which 
means there'd be no reliable way to detect whether frozen modules are 
going to be enabled by default or not. Adding a separate option handles 
this case.


(My overall preference is for #3, FWIW)

Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CDG4LSUXRLKCDPJ4RMVXCAPDJ4PNTN4D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Guido van Rossum]

On Mon, Sep 27, 2021 at 10:40 AM Steve Dower  wrote:

> On 9/27/2021 5:51 PM, Eric Snow wrote:
> > Possible solutions:
> >
> > 1. always default to "on" (the annoyance for contributors isn't big
> enough?)
> > 2. default to "on" if it's a PGO build (and "off" otherwise)
>
> Just to air my concerns regarding option 2 (which I've already spoken to
> Eric about): I feel this option is completely orthogonal to whether PGO
> is used or not, and ought to be discoverable independently.
>
> Essentially, it should be its own configure-time option, and should be
> included somewhere in sysconfig.get_config_var(...).
>
> If we went with #2, there's no reliable way to detect whether
> profile-guided optimisations were used on all CPython builds, which
> means there'd be no reliable way to detect whether frozen modules are
> going to be enabled by default or not. Adding a separate option handles
> this case.
>
> (My overall preference is for #3, FWIW)
>

When I proposed #2, I used "PGO" as a proxy for "best optimization mode".
On UNIX, this is `./configure --enable-optimizations`, which doesn't
mention PGO -- IIUC it turns on PGO and LTO, if they're available.

So my *actual* proposal (call it #2') is to use a separate compile-time
flag, which is set by `./configure --enable-optimizations` regardless of
whether PGO/LTO are possible, and which on Windows can be set by
`PCbuild\build.bat --pgo` (we could add another flag to disable it, but
who'd want to?).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6UAHXEXLTMCJANP4WMO47EI6E5DF76KQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Patrick Reader]

How about checking each non-frozen module's hash and/or and comparing it to 
that of the frozen module? Would that defeat the performance improvement of 
freezing? Is it just a terrible idea?

On 27/09/2021 17:51, Eric Snow wrote:
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?
>
> -eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YPENSVRAT2HKZSBX73ZWZYTLFIFFFDQT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Patrick Reader]

I accidentally a word.

"... module's hash and/or *timestamp* and comparing it ..."

On 27/09/2021 18:38, Patrick Reader wrote:
> How about checking each non-frozen module's hash and/or and comparing it to 
> that of the frozen module? Would that defeat the performance improvement of 
> freezing? Is it just a terrible idea?
>
> On 27/09/2021 17:51, Eric Snow wrote:
>> We've frozen most of the stdlib modules imported during "python -c
>> pass" [1][2], to make startup a bit faster.  Import of those modules
>> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
>> to "off" but we'd like to default to "on".  The blocker is the impact
>> on contributors.  I expect many will make changes to a stdlib module
>> and then puzzle over why those changes aren't getting used.  That's an
>> annoyance we can avoid, which is the point of this thread.
>>
>> Possible solutions:
>>
>> 1. always default to "on" (the annoyance for contributors isn't big enough?)
>> 2. default to "on" if it's a PGO build (and "off" otherwise)
>> 3. default to "on" unless running from the source tree
>>
>> Thoughts?
>>
>> -eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6ALEB4NCYS32EOLR5PT25GYBDN7ZOYEO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Ethan Furman]

On 9/27/21 10:50 AM, Guido van Rossum wrote:

> So my *actual* proposal (call it #2') is to use a separate compile-time flag, 
which is set by `./configure
> --enable-optimizations` regardless of whether PGO/LTO are possible, and which 
on Windows can be set by
> `PCbuild\build.bat --pgo` (we could add another flag to disable it, but who'd 
want to?).

I think a configure-time flag is the way to go, and I'm happy to have it 
included with --enable-optimizations.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FCEJANFK7477B5IEHJ55BCEN5OBL3BTU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Steve Dower]

On 9/27/2021 6:40 PM, Patrick Reader wrote:

I accidentally a word.

"... module's hash and/or *timestamp* and comparing it ..."

On 27/09/2021 18:38, Patrick Reader wrote:

How about checking each non-frozen module's hash and/or and comparing it to 
that of the frozen module? Would that defeat the performance improvement of 
freezing? Is it just a terrible idea?


The filesystem access is what hurts the most, so yeah, that check won't 
even be considered here without a manual opt-in (which is the option 
Eric is discussing).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VOX3HTK3RBD7KLJ46XZQGS7G4SHVDDX3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Steve Dower]

On 9/27/2021 7:15 PM, Ethan Furman wrote:

On 9/27/21 10:50 AM, Guido van Rossum wrote:

 > So my *actual* proposal (call it #2') is to use a separate 
compile-time flag, which is set by `./configure
 > --enable-optimizations` regardless of whether PGO/LTO are possible, 
and which on Windows can be set by
 > `PCbuild\build.bat --pgo` (we could add another flag to disable it, 
but who'd want to?).


I think a configure-time flag is the way to go, and I'm happy to have it 
included with --enable-optimizations.


Having it be implied by an "--enable-optimizations" option is totally 
fine (and we'd add one to build.bat for this), but I still think it 
needs to be discoverable later whether the frozen modules build option 
was used or not, independent of other build options.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QU6H2GS6S4MD7WTDI2Y6BLP46MXJRVFG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric Snow]

On Mon, Sep 27, 2021 at 12:40 PM Steve Dower  wrote:
> Having it be implied by an "--enable-optimizations" option is totally
> fine (and we'd add one to build.bat for this), but I still think it
> needs to be discoverable later whether the frozen modules build option
> was used or not, independent of other build options.

That's reasonable.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KKLMPEB6SI2EC34MUPLSTFBJJYG4O4WE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Brett Cannon]

On Sun, Sep 26, 2021 at 3:51 AM Phil Thompson via Python-Dev <
python-dev@python.org> wrote:

> On 26/09/2021 05:21, Steven D'Aprano wrote:
>
> [snip]
>


> > These are not rhetorical questions, I genuinely do not know. I *think*
> > that there was an attempt to make a stable C API back in 3.2 days:
> >
> > https://www.python.org/dev/peps/pep-0384/
> >
> > but I don't know what difference it has made to extension writers in
> > practice. From your description, it sounds like perhaps not as big a
> > difference as we would have liked.
> >
> > Maybe extension writers are not using the stable C API? Is that even
> > possible? Please excuse my ignorance.
>
> PyQt has used the stable ABI for many years. The main reason for using
> it is to reduce the number of wheels. The PyQt ecosystem currently
> contains 15 PyPI projects across 4 platforms supporting 5 Python
> versions (including v3.10). Without the stable ABI a new release would
> require 300 wheels. With the stable ABI it is a more manageable 60
> wheels.
>
> However the stable ABI is still a second class citizen as it is still
> not possible (AFAIK) to specify a wheel name that doesn't need to
> explicitly include each supported Python version (rather than a minimum
> stable ABI version).
>

Actually you can do this. The list of compatible wheels for a platform
starts at CPython 3.2 when the stable ABI was introduced and goes forward
to the version of Python you are running. So you can build a wheel file
that targets the oldest version of CPython that you are targeting and its
version of the stable ABI and it is considered forward compatible. See
`python -m pip debug --verbose` for the complete list of wheel tags that
are supported for an interpreter.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/E4EA6GW6N5NREEBM4QOA7GTLBGTGD4AV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Brett Cannon]

On Mon, Sep 27, 2021 at 9:54 AM Eric Snow 
wrote:

> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big
> enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?
>

What about opting out when `--with-pydebug` is used? I'm not sure how many
people actively develop in a non-debug build other than testing something,
but at that point I would be having to run `make` probably anyway for
whatever I'm mucking with if it's *that* influenced by a debug build.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PYX65WVG5ICXEBIY5ZDLDFAJHDU56HLA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Barry Warsaw]

A couple of questions.

If you’re planning a runtime -X option, then does that mean that the modules 
will be frozen at build time but Python will decide at runtime whether to use 
the frozen modules or the unfrozen ones?

Are you planning on including the currently frozen importlib modules in that 
same mechanism?

Will `make test` and/or CI run Python with both options?  How will we make sure 
that frozen modules (or not) don’t break Python?

Option #3 seems like the most reasonable one to me, with the ability to turn it 
on when running from the source tree.

-Barry

> On Sep 27, 2021, at 09:51, Eric Snow  wrote:
> 
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
> 
> Possible solutions:
> 
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
> 
> Thoughts?
> 
> -eric
> 
> 
> [1] https://bugs.python.org/issue45020
> [2] FWIW, we may end up also freezing the modules imported for "python
> -m ...", along with some other commonly used modules (like argparse).
> That is a separate discussion.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
> Code of Conduct: http://python.org/psf/codeofconduct/



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VH6ZFP72H5NZDOTMNOFCUHSPCGERBRDM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Victor Stinner]

Hi Eric,

Which stdlib modules are currently frozen? If I really want to hack
site.py or os.py for whatever reason, I just have to use "python3 -X
frozen_modules=off"?

> 1. always default to "on" (the annoyance for contributors isn't big enough?)

What is the annoyance? What is different between frozen and not frozen?

Victor

On Mon, Sep 27, 2021 at 6:58 PM Eric Snow  wrote:
>
> We've frozen most of the stdlib modules imported during "python -c
> pass" [1][2], to make startup a bit faster.  Import of those modules
> is controlled by "-X frozen_modules=[on|off]".  Currently it defaults
> to "off" but we'd like to default to "on".  The blocker is the impact
> on contributors.  I expect many will make changes to a stdlib module
> and then puzzle over why those changes aren't getting used.  That's an
> annoyance we can avoid, which is the point of this thread.
>
> Possible solutions:
>
> 1. always default to "on" (the annoyance for contributors isn't big enough?)
> 2. default to "on" if it's a PGO build (and "off" otherwise)
> 3. default to "on" unless running from the source tree
>
> Thoughts?
>
> -eric
>
>
> [1] https://bugs.python.org/issue45020
> [2] FWIW, we may end up also freezing the modules imported for "python
> -m ...", along with some other commonly used modules (like argparse).
> That is a separate discussion.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/4ESW3NNOX43DRFKLEW3IMDXDKPDMNRGR/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CLODS7B5Z3UEZTQ7QIALG2DWB4H37EWP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric Snow]

On Mon, Sep 27, 2021 at 2:59 PM Brett Cannon  wrote:
> What about opting out when `--with-pydebug` is used? I'm not sure how many 
> people actively develop in a non-debug build other than testing something, 
> but at that point I would be having to run `make` probably anyway for 
> whatever I'm mucking with if it's that influenced by a debug build.

Yeah, that's an option too.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/C2IMGPQMJQAFCH26SYHE4JE4WJRCPDBM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric Snow]

On Mon, Sep 27, 2021 at 3:04 PM Barry Warsaw  wrote:
> If you’re planning a runtime -X option, then does that mean that the modules 
> will be frozen at build time but Python will decide at runtime whether to use 
> the frozen modules or the unfrozen ones?

Correct.  FYI, this was already done.

> Are you planning on including the currently frozen importlib modules in that 
> same mechanism?

No.  They must always be frozen.  See is_essential_frozen_module() in
Python/import.c.

> Will `make test` and/or CI run Python with both options?  How will we make 
> sure that frozen modules (or not) don’t break Python?

If "configure --with-optimizations" always sets the default to "on"
and the default is "off" otherwise, then the PGO buildbots will
exercise the frozen path.  Likewise if "--with-pydebug" (or
in-source-tree) makes the default "off" and otherwise it's "on".

Without a build-time option already handled by one of the buildbots,
we'd need to either add a dedicated buildbot or run it both ways (like
we do with importlib).  I expect that won't be necessary.

> Option #3 seems like the most reasonable one to me, with the ability to turn 
> it on when running from the source tree.

It's definitely the one that fits most naturally for me.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I2JTBQSFFA2GFMSGRDGHDARUPSZTLMQ2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Eric Snow]

On Mon, Sep 27, 2021 at 3:31 PM Victor Stinner  wrote:
> Which stdlib modules are currently frozen? If I really want to hack
> site.py or os.py for whatever reason, I just have to use "python3 -X
> frozen_modules=off"?

The single-source-of-truth is Tools/scripts/freeze_modules.py.  After
running "make regen-frozen" you'll find a cleaner list in
Python/frozen_modules/MANIFEST.  You can also look at the generated
code in Makefile.pre.in or Python/frozen.c.  Finally, you can run
"./python -X frozen_modules=on -c 'import _imp;
print(_imp._frozen_module_names())'"

> > 1. always default to "on" (the annoyance for contributors isn't big enough?)
>
> What is the annoyance?

The annoyance of changes to the .py files not getting used (at least
not until after running "make all"

> What is different between frozen and not frozen?

They have a different loader and repr.  Also, frozen modules do not
have __file__ set (and __path__ is always []).

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J5IINOU6JBHNBA4ZOTXWDCBC3QIQT2EF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: The Default for python -X frozen_modules.

[Victor Stinner]

On Tue, Sep 28, 2021 at 12:52 AM Eric Snow  wrote:
> On Mon, Sep 27, 2021 at 3:31 PM Victor Stinner  wrote:
> > Which stdlib modules are currently frozen? If I really want to hack
> > site.py or os.py for whatever reason, I just have to use "python3 -X
> > frozen_modules=off"?
>
> The single-source-of-truth is Tools/scripts/freeze_modules.py.  After
> running "make regen-frozen" you'll find a cleaner list in
> Python/frozen_modules/MANIFEST.  You can also look at the generated
> code in Makefile.pre.in or Python/frozen.c.  Finally, you can run
> "./python -X frozen_modules=on -c 'import _imp;
> print(_imp._frozen_module_names())'"

Ok, so compared to Python 3.10, the following 13 stdlib modules can
now be frozen:

* _collections_abc
* _sitebuiltins
* abc
* codecs
* genericpath
* io
* ntpath
* os
* os.path
* posixpath
* site
* stat
* zipimport

(I tested on Linux. I guess that the list is the same on other
operating systems.)

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RYRZ7GLC6ULGXZJ4KHRIGH4XCIMW5M4H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Worried about Python release schedule and lack of stable C-API

[Nathaniel Smith]

On Sun, Sep 26, 2021 at 3:38 AM  wrote:
> Open3D is an example. Will finally move to Python 3.9 some time the coming 
> month. Its dependency graph contains about 70 other packages.
>
> In this specific case, the underlying problem was that TensorFlow was stuck 
> at 3.8. The TensorFlow codebase got ported in November 2020, then released 
> early 2021. Then Open3D included the new Tensorflow (plus whatever else 
> needed to be adapted) in their codebase in May. They’re now going through 
> their release schedule, and their 0.14 release should be up on PyPI soon.

I took a minute to look up the release dates to fill in this timeline:

Python 3.9 released: October 2020
Tensorflow adds 3.9 support: November 2020
Tensorflow v2.5.0 released with the new 3.9 support: May 2021
Open3d adds 3.9 support: May 2021
First Open3d release to include the new 3.9 support: ~October 2021

So it seems like in this case at least, the year long delay consists
of ~1 month of porting work, and ~11 months of projects letting the
finished code sit in their repos without shipping to users.

It seems like the core problem here is that these projects don't
consider it important to keep up with the latest Python release. I'm
not sure what CPython upstream can do about that. Maybe you could
lobby these projects to ship releases more promptly?

By contrast, to pick a random library that uses the unstable C API
extensively, NumPy is already shipping wheels for 3.10 -- and 3.10
isn't even out yet. So it's certainly possible to do, even for
projects with a tiny fraction of Tensorflow's engineering budget.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OGBANTGGJAI2ZM5SYSLCWFGRIML5VWUL/
Code of Conduct: http://python.org/psf/codeofconduct/