[Python-Dev] A fast startup patch (was: Python startup time)

2018-05-03 Thread Carl Shapiro
Hello, Yesterday Neil Schemenauer mentioned some work that a colleague of mine (CCed) and I have done to improve CPython start-up time. Given the recent discussion, it seems timely to discuss what we are doing and whether it is of interest to other people hacking on the CPython runtime. There ar

Re: [Python-Dev] A fast startup patch (was: Python startup time)

2018-05-04 Thread Carl Shapiro
On Fri, May 4, 2018 at 5:14 AM, Nick Coghlan wrote: > This definitely seems interesting, but is it something you'd be seeing us > being able to take advantage of for conventional Python installations, or > is it more something you'd expect to be useful for purpose-built > interpreter instances? (

Re: [Python-Dev] A fast startup patch (was: Python startup time)

2018-05-07 Thread Carl Shapiro
On Fri, May 4, 2018 at 6:58 PM, Nathaniel Smith wrote: > What are the obstacles to including "preloaded" objects in regular .pyc > files, so that everyone can take advantage of this without rebuilding the > interpreter? > The system we have developed can create a shared object file for each comp

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-17 Thread Carl Shapiro
On Sun, Sep 16, 2018 at 1:24 PM, Antoine Pitrou wrote: > I think it's of limited interest if it only helps with modules used > during the startup sequence, not arbitrary stdlib or third-party > modules. > This should help any use-case that is already using the freeze module already bundled with

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-18 Thread Carl Shapiro
On Tue, Sep 18, 2018 at 1:31 AM, Antoine Pitrou wrote: > No idea. In my previous experiments with module import speed, I > concluded that executing module bytecode generally was the dominating > contributor, but that doesn't mean loading bytecode is costless. > My observations might not be so d

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-18 Thread Carl Shapiro
On Tue, Sep 18, 2018 at 5:55 AM, Fabio Zadrozny wrote: > During the import process, Python can already deal with folders and .zip > files in sys.path... now, instead of having special handling for a new > concept with a custom command line, etc, why not just say that this is a > special file (e.g

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-18 Thread Carl Shapiro
On Tue, Sep 18, 2018 at 11:38 AM, Steve Dower wrote: > The primary benefit of the importlib hook approach is that it would not > require rebuilding CPython each time you make a change. Since we need to > consider a wide range of users across a wide range of platforms, having the > ability to load

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-20 Thread Carl Shapiro
On Wed, Sep 19, 2018 at 12:32 AM, Stefan Behnel wrote: > Also, one thing that would be interesting to find out is whether constant > Python data structures can actually be pre-allocated in the data segment > (and I mean their object structs) . Then things like tuples of strings > (argument lists

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-21 Thread Carl Shapiro
On Tue, Sep 18, 2018 at 3:00 PM, Neil Schemenauer wrote: > The users of Python are pretty diverse so it depends on who you ask. > Some would like a giant executable that includes everything they > need (so of like the Go approach). Other people want an executable > that has just importlib inside

Re: [Python-Dev] Store startup modules as C structures for 20%+ startup speed improvement?

2018-09-21 Thread Carl Shapiro
On Thu, Sep 20, 2018 at 11:20 PM, Stefan Behnel wrote: > What about the small integers cache? The C serialisation generates several > PyLong objects that would normally reside in the cache. Is this handled > somewhere? I guess the cache could entirely be loaded from the data > segment. And the sa

[Python-Dev] Re: PEP 554 for 3.9 or 3.10?

2020-04-26 Thread Carl Shapiro
On Tue, Apr 21, 2020 at 7:33 AM Victor Stinner wrote: > IMHO it's worth it to explore the subinterpreters approach. I don't > think that it's going to be a major maintenance burden: the code is > already written and tests. The PEP is only about adding a public > Python API. While this PEP may not

[Python-Dev] Re: PEP 620: Hide implementation details from the C API

2020-06-25 Thread Carl Shapiro
Thank you very much for putting this PEP together. It would be very helpful to broaden the objective of avoiding functions returning PyObject** to other types of pointers. I have in mind several functions in the C-API that return a char* pointer to the contents of an object. While these function

[Python-Dev] Re: (PEP 620) C API for efficient loop iterating on a sequence of PyObject** or other C types

2020-06-26 Thread Carl Shapiro
On Tue, Jun 23, 2020 at 7:52 AM Stefan Behnel wrote: > I agree that this is more explicit when it comes to resource management, > but there is nothing that beats direct native data structure access when it > comes to speed. >From the perspective of the function that wants to get access to the c

[Python-Dev] Re: Memory address vs serial number in reprs

2020-07-19 Thread Carl Shapiro
On Sun, Jul 19, 2020 at 1:34 PM Thomas Moreau wrote: > While it would be nice to have simpler identifiers for objects, it would > be hard to make it work for multiprocessing, as objects in different > interpreter would end up having the same repr. Shared objects (locks) might > also have differen

[Python-Dev] Re: PEP 626: Precise line numbers for debugging and other tools.

2020-07-21 Thread Carl Shapiro
On Tue, Jul 21, 2020 at 11:46 AM Mark Shannon wrote: > On 18/07/2020 9:20 am, Inada Naoki wrote: > > And I am worrying precise tracing blocks future advanced bytecode > optimization. > > Can we omit precise tracing and line number information when > > optimization (`-O`) is enabled? > > I don't t