[Python-Dev] Two new environment variables to debug Python 2.7

[Victor Stinner]

Hi,

FYI I just merged two pull requests into Python 2.7, each add a new
environment variable changing the behaviour in debug mode:

bpo-31733: don't dump "[xxx refs]" into stderr by default anymore
Set PYTHONSHOWREFCOUNT=1
commit 3c082a7fdb472f02bcac7a7f8fe1e3a34a11b70b

bpo-31692, bpo-19527: don't dump allocations counts by default anymore
Set PYTHONSHOWALLOCCOUNT=1
commit 7b4ba62e388474e811268322b47f80d464933541

I never ever used "[xxx refs]" to detect a reference leak. I only use
"./python -m test -R 3:3 test_xxx" to detect reference leaks. To be
honest, usually I only run such test explicitly when one of our
"Refleaks" buildbot starts to comlain :-)

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

[Victor Stinner]

> Since the ``time.clock()`` function was deprecated in Python 3.3, no
> ``time.clock_ns()`` is added.

FYI I just proposed a change to *remove* time.clock() from Python 3.7:
https://bugs.python.org/issue31803

This change is not required by, nor directly related to, the PEP 564.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Guido van Rossum]

On Mon, Oct 16, 2017 at 10:02 PM, Nick Coghlan  wrote:

> On 17 October 2017 at 14:31, Guido van Rossum  wrote:
>
>> No, that version just defers to magic in ContextVar.get/set, whereas what
>> I'd like to see is that the latter are just implemented in terms of
>> manipulating the mapping directly. The only operations for which speed
>> matters would be __getitem__ and __setitem__; most other methods just defer
>> to those. __delitem__ must also be a primitive, as must __iter__ and
>> __len__ -- but those don't need to be as speedy (however __delitem__ must
>> really work!).
>>
>
> To have the mapping API at the base of the design, we'd want to go back to
> using the ContextKey version of the API as the core primitive (to ensure we
> don't get name conflicts between different modules and packages), and then
> have ContextVar be a convenience wrapper that always accesses the currently
> active context:
>
> class ContextKey:
> ...
> class ExecutionContext:
> ...
>
> class ContextVar:
> def __init__(self, name):
> self._key = ContextKey(name)
>
> def get(self):
> return get_execution_context()[self._key]
>
> def set(self, value):
> get_execution_context()[self._key] = value
>
> def delete(self, value):
> del get_execution_context()[self._key]
>

Why would we need this extra layer? I would assume that the key can just be
the ContextVar object itself, e.g.

return get_execution_context()[self]

or

get_execution_context()[self] = value


> While I'd defer to Yury on the technical feasibility, I'd expect that
> version could probably be made to work *if* you were amenable to some of
> the mapping methods on the execution context raising RuntimeError in order
> to avoid locking ourselves in to particular design decisions before we're
> ready to make them.
>
> The reason I say that is because one of the biggest future-proofing
> concerns when it comes to exposing a mapping as the lowest API layer is
> that it makes the following code pattern possible:
>
> ec = get_execution_context()
> # Change to a different execution context
> ec[key] = new_value
>
> The appropriate semantics for that case (modifying a context that isn't
> the currently active one) are *really* unclear, which is why PEP 550
> structures the API to prevent it (context variables can only manipulate the
> active context, not arbitrary contexts).
>

But why on earth would you want to prevent that? If there's some caching
involved that I have overlooked (another problem with the complexity of the
design, or perhaps the explanation), couldn't mutating the non-context
simply set a dirty bit to ensure that if it ever gets made the current
context again the cache must be considered invalidated?


> However, even with a mapping at the lowest layer, a similar API constraint
> could still be introduced via a runtime guard in the mutation methods:
>
> if get_execution_context() is not self:
> raise RuntimeError("Cannot modify an inactive execution context")
>
> That way, to actually mutate a different context, you'd still have to
> switch contexts, just as you have to switch threads in C if you want to
> modify another thread's thread specific storage.
>

But that sounds really perverse. If anything, modifying an EC that's not
any thread's current context should be *simpler* than modifying the current
context. (I'm okay with a prohibition on modifying another *thread's*
current context.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Yury Selivanov]

On Tue, Oct 17, 2017 at 1:02 AM, Nick Coghlan  wrote:
> On 17 October 2017 at 14:31, Guido van Rossum  wrote:
>>
>> No, that version just defers to magic in ContextVar.get/set, whereas what
>> I'd like to see is that the latter are just implemented in terms of
>> manipulating the mapping directly. The only operations for which speed
>> matters would be __getitem__ and __setitem__; most other methods just defer
>> to those. __delitem__ must also be a primitive, as must __iter__ and __len__
>> -- but those don't need to be as speedy (however __delitem__ must really
>> work!).
>
>
> To have the mapping API at the base of the design, we'd want to go back to
> using the ContextKey version of the API as the core primitive (to ensure we
> don't get name conflicts between different modules and packages), and then
> have ContextVar be a convenience wrapper that always accesses the currently
> active context:
>
> class ContextKey:
> ...
> class ExecutionContext:
> ...
>
> class ContextVar:
> def __init__(self, name):
> self._key = ContextKey(name)
>
> def get(self):
> return get_execution_context()[self._key]
>
> def set(self, value):
> get_execution_context()[self._key] = value
>
> def delete(self, value):
> del get_execution_context()[self._key]

ContextVar itself will be hashable, we don't need ContextKeys.

>
> While I'd defer to Yury on the technical feasibility, I'd expect that
> version could probably be made to work *if* you were amenable to some of the
> mapping methods on the execution context raising RuntimeError in order to
> avoid locking ourselves in to particular design decisions before we're ready
> to make them.
>
> The reason I say that is because one of the biggest future-proofing concerns
> when it comes to exposing a mapping as the lowest API layer is that it makes
> the following code pattern possible:
>
> ec = get_execution_context()
> # Change to a different execution context
> ec[key] = new_value

I *really* don't want to make ECs behave like 'locals()'.  That will
make everything way more complicated.

My way of thinking about this: "get_execution_context()" returns you a
shallow copy of the current EC (at least conceptually).  So making any
modifications on it won't affect the current environment.  The only
way to actually apply the modified EC object to the environment will
be its 'run(callable)' method.

Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup optimization: script vs. service

[Neil Schemenauer]

Christian Heimes  wrote:
> That approach could work, but I think that it is the wrong
> approach. I'd rather keep Python optimized for long-running
> processes and introduce a new mode / option to optimize for
> short-running scripts.

Another idea is to run a fake transasaction through the process
before forking.  That will "warm up" things so that most of the lazy
init is already done.

After returning from the core sprint, I have gotten over my initial
enthusiam for my "lazy module defs" idea.  It is just too big of a
change for Python to accept that this point.  I still hope there
would be a way to make LOAD_NAME/LOAD_GLOBAL trigger something like
__getattr__().  That would allow libraries that want to aggressively
do lazy-init to do so in the clean way.

The main reason that Python startup is slow is that we do far too
much work on module import (e.g. initializing data structures that
never get used).  Reducing that work will almost necessarily impact
pre-fork model programs (e.g. they expect the init to be done before
the fork).

As someone who uses that model heavily, I would still be okay with
the "lazification" as I think there are many more programs that
would be helped vs the ones hurt.  Initializing everything that your
program might possibibly need right at startup time doesn't seem
like a goal to strive for.  I can understand if you have a different
opinion though.

A third approach would be to do more init work at compile time.
E.g. for re.compile, if the compiled result could be stored in the
.pyc, that would eliminate a lot of time for short scripts and for
long-running programs.  Some Lisp systems have "compiler macros".
They are basically a hook to allow programs to do some work before
the code is sent to the compiler.  If something like that existed in
Python, it could be used by re.compile to generate a compiled
representation of the regex to store in the .pyc file.  That kind of
behavior is pretty different than the "there is only runtime" model
that Python generally tries to follow.

Spit-ball idea, thought up just now:

PAT = __compiled__(re.compile(...))

The expression in __compiled__(..) would be evaluated by the
compiler and the resulting value would become the value to store in
th .pyc.  If you are running the code as the script, __compiled__
just returns its argument unchanged.

Cheers,

  Neil

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reorganize Python categories (Core, Library, ...)?

[Neil Schemenauer]

Antoine Pitrou  wrote:
> There is no definite "correct category" when you're mixing different
> classification schemes (what kind of bug it is --
> bug/security/enhancement/etc. --, what functional domain it pertains
> to -- networking/concurrency/etc. --, which stdlib API it affects).

I think there should be a set of tags rather than a single category.
In the blurb entry, you could apply all the tags that are relevant.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Guido van Rossum]

On Tue, Oct 17, 2017 at 8:54 AM, Yury Selivanov 
wrote:

> On Tue, Oct 17, 2017 at 1:02 AM, Nick Coghlan  wrote:
> > The reason I say that is because one of the biggest future-proofing
> concerns
> > when it comes to exposing a mapping as the lowest API layer is that it
> makes
> > the following code pattern possible:
> >
> > ec = get_execution_context()
> > # Change to a different execution context
> > ec[key] = new_value
>
> I *really* don't want to make ECs behave like 'locals()'.  That will
> make everything way more complicated.
>

At least some of the problems with locals() have more to do with the legacy
of that function than with inherent difficulties. And local variables might
be optimized by a JIT in a way that context vars never will be (or at least
if we ever get to that point we will be able to redesign the API first).


> My way of thinking about this: "get_execution_context()" returns you a
> shallow copy of the current EC (at least conceptually).  So making any
> modifications on it won't affect the current environment.  The only
> way to actually apply the modified EC object to the environment will
> be its 'run(callable)' method.
>

I understand that you don't want to throw away the implementation work
you've already done. But I find that the abstractions you've introduced are
getting in the way of helping people understand what they can do with
context variables, and I really want to go back to a model that is *much*
closer to understanding how instance variables are just self.__dict__.
(Even though there are possible complications due to __slots__ and
@property.)

In short, I really don't think there's a need for context variables to be
faster than instance variables.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Guido van Rossum]

Also, IMO this is all the interface we should need to explain to users
(even framework authors):
https://github.com/gvanrossum/pep550/blob/master/simpler.py

On Tue, Oct 17, 2017 at 11:25 AM, Guido van Rossum  wrote:

> On Tue, Oct 17, 2017 at 8:54 AM, Yury Selivanov 
> wrote:
>
>> On Tue, Oct 17, 2017 at 1:02 AM, Nick Coghlan  wrote:
>> > The reason I say that is because one of the biggest future-proofing
>> concerns
>> > when it comes to exposing a mapping as the lowest API layer is that it
>> makes
>> > the following code pattern possible:
>> >
>> > ec = get_execution_context()
>> > # Change to a different execution context
>> > ec[key] = new_value
>>
>> I *really* don't want to make ECs behave like 'locals()'.  That will
>> make everything way more complicated.
>>
>
> At least some of the problems with locals() have more to do with the
> legacy of that function than with inherent difficulties. And local
> variables might be optimized by a JIT in a way that context vars never will
> be (or at least if we ever get to that point we will be able to redesign
> the API first).
>
>
>> My way of thinking about this: "get_execution_context()" returns you a
>> shallow copy of the current EC (at least conceptually).  So making any
>> modifications on it won't affect the current environment.  The only
>> way to actually apply the modified EC object to the environment will
>> be its 'run(callable)' method.
>>
>
> I understand that you don't want to throw away the implementation work
> you've already done. But I find that the abstractions you've introduced are
> getting in the way of helping people understand what they can do with
> context variables, and I really want to go back to a model that is *much*
> closer to understanding how instance variables are just self.__dict__.
> (Even though there are possible complications due to __slots__ and
> @property.)
>
> In short, I really don't think there's a need for context variables to be
> faster than instance variables.
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [RELEASE] Python 3.7.0a2 is now available for testing

[Ned Deily]

Python 3.7.0a2 is the second of four planned alpha previews of Python 3.7,
the next feature release of Python.  During the alpha phase, Python 3.7
remains under heavy development: additional features will be added
and existing features may be modified or deleted.  Please keep in mind
that this is a preview release and its use is not recommended for
production environments.  The next preview, 3.7.0a3, is planned for
2017-11-27.  You can find Python 3.7.0a2 and more information here:

   https://www.python.org/downloads/release/python-370a2/

--
  Ned Deily
  n...@python.org -- []

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Nathaniel Smith]

On Oct 17, 2017 11:25 AM, "Guido van Rossum"  wrote:


In short, I really don't think there's a need for context variables to be
faster than instance variables.


There really is: currently the cost of looking up a thread local through
the C API is a dict lookup, which is faster than instance variable lookup,
and decimal and numpy have both found that that's already too expensive.

Or maybe you're just talking about the speed when the cache misses, in
which case never mind :-).

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Yury Selivanov]

On Tue, Oct 17, 2017 at 2:25 PM, Guido van Rossum  wrote:
> On Tue, Oct 17, 2017 at 8:54 AM, Yury Selivanov 
[..]
>> My way of thinking about this: "get_execution_context()" returns you a
>> shallow copy of the current EC (at least conceptually).  So making any
>> modifications on it won't affect the current environment.  The only
>> way to actually apply the modified EC object to the environment will
>> be its 'run(callable)' method.
>
>
> I understand that you don't want to throw away the implementation work
> you've already done. But I find that the abstractions you've introduced are
> getting in the way of helping people understand what they can do with
> context variables, and I really want to go back to a model that is *much*
> closer to understanding how instance variables are just self.__dict__. (Even
> though there are possible complications due to __slots__ and @property.)

I don't really care about the implementation work that has already
been done, it's OK if I write it from scratch again.

I actually like what you did in
https://github.com/gvanrossum/pep550/blob/master/simpler.py, it seems
reasonable.  The only thing that I'd change is to remove "set_ctx"
from the public API and add "Context.run(callable)".  This makes the
API more flexible to potential future changes and amendments.


> In short, I really don't think there's a need for context variables to be
> faster than instance variables.

Well, even our current idea about the API, doesn't really prohibit us
from adding a cache to ContextVar.get().  That would be an
implementation detail, right?  Same as our latest optimization for
creating bound methods (CALL_METHOD & LOAD_METHOD opcodes), which
avoids creating bound method instances when it's OK not to.


Yury
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 510 (function specialization) rejected

[Victor Stinner]

Hi,

I rejected my own PEP 510 "Specialize functions with guards" that I
wrote in January 2016:

https://github.com/python/peps/commit/c99fb8bf5b5c16c170e1603a1c66a74e93a4ae84

"This PEP was rejected by its author since the design didn't show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations."

I stopped working on my FAT Python project almost one year ago:
https://faster-cpython.readthedocs.io/fat_python.html

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Guido van Rossum]

On Tue, Oct 17, 2017 at 12:51 PM, Nathaniel Smith  wrote:

> On Oct 17, 2017 11:25 AM, "Guido van Rossum"  wrote:
>
>
> In short, I really don't think there's a need for context variables to be
> faster than instance variables.
>
>
> There really is: currently the cost of looking up a thread local through
> the C API is a dict lookup, which is faster than instance variable lookup,
> and decimal and numpy have both found that that's already too expensive.
>

(At first I found this hard to believe, but then I realized that decimal
and numpy presumably access these from C code where a typical operation
like Decimal.__add__ is much faster than a dict lookup. So point taken.)


> Or maybe you're just talking about the speed when the cache misses, in
> which case never mind :-).
>

I'm happy to support caching the snot out of this, but it seems we agree
that the "semantics" can be specified without taking the caching into
account, and that's what I'm after. I presume that each ContextVar object
will have one cached value? Because that's what PEP 550 currently
specifies. Surely it wouldn't be hard for a direct __setitem__ (or
__delitem__) call to cause the cache to be invalidated, regardless of
whether the affected context is a *current* context or not (a bunch of
different approaches suggest themselves).Oother mutating methods can be
implemented in terms of __setitem__ (or __delitem__), and I don't care for
them to be as fast as a cached lookup.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 511 (code transformers) rejected

[Victor Stinner]

Hi,

I rejected my own PEP 511 "API for code transformers" that I wrote in
January 2016:

https://github.com/python/peps/commit/9d8fd950014a80324791d7dae3c130b1b64fdace

Rejection Notice:

"""
This PEP was rejected by its author.

This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.

This PEP was also seen as a nice tool to experiment new Python features,
but it is already possible to experiment them without the PEP, only with
importlib hooks. If a feature becomes useful, it should be directly part
of Python, instead of depending on an third party Python module.

Finally, this PEP was driven was the FAT Python optimization project
which was abandonned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.
"""

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 510 (function specialization) rejected

[Victor Stinner]

2017-10-17 22:25 GMT+02:00 Guido van Rossum :
> It takes courage to admit failures like this! I think this is a good call.
> It echoes the experiences with Unladen Swallow and Pyston. Despite what
> people may think, CPython really isn't slow, given the large set of
> constraints on the implementation.

Oh, I still have a long queue of optimization ideas that I want to try
:-) But first, I would like to fix the issue blocking all significant
optimizations: make the stable ABI usable to allow to change major
CPython design choices without breaking C extensions.

https://haypo.github.io/new-python-c-api.html

Sadly, I didn't find time yet to write a proper PEP for that.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 510 (function specialization) rejected

[Guido van Rossum]

It takes courage to admit failures like this! I think this is a good call.
It echoes the experiences with Unladen Swallow and Pyston. Despite what
people may think, CPython really isn't slow, given the large set of
constraints on the implementation.

On Tue, Oct 17, 2017 at 1:10 PM, Victor Stinner 
wrote:

> Hi,
>
> I rejected my own PEP 510 "Specialize functions with guards" that I
> wrote in January 2016:
>
> https://github.com/python/peps/commit/c99fb8bf5b5c16c170e1603a1c66a7
> 4e93a4ae84
>
> "This PEP was rejected by its author since the design didn't show any
> significant speedup, but also because of the lack of time to implement
> the most advanced and complex optimizations."
>
> I stopped working on my FAT Python project almost one year ago:
> https://faster-cpython.readthedocs.io/fat_python.html
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup optimization: script vs. service

[Michel Desmoulin]

Maybe it's time to bring back the debate on the "lazy" keyword then ?
Rendering any statement arbitrarily lazy could help with perfs. It would
also make hacks like ugettext_lazy in Django useless. And would render
moot the extensions of f-strings for lazily rendered ones. And bring
lazy imports in the mix.

Le 17/10/2017 à 19:39, Neil Schemenauer a écrit :
> Christian Heimes  wrote:
>> That approach could work, but I think that it is the wrong
>> approach. I'd rather keep Python optimized for long-running
>> processes and introduce a new mode / option to optimize for
>> short-running scripts.
> 
> Another idea is to run a fake transasaction through the process
> before forking.  That will "warm up" things so that most of the lazy
> init is already done.
> 
> After returning from the core sprint, I have gotten over my initial
> enthusiam for my "lazy module defs" idea.  It is just too big of a
> change for Python to accept that this point.  I still hope there
> would be a way to make LOAD_NAME/LOAD_GLOBAL trigger something like
> __getattr__().  That would allow libraries that want to aggressively
> do lazy-init to do so in the clean way.
> 
> The main reason that Python startup is slow is that we do far too
> much work on module import (e.g. initializing data structures that
> never get used).  Reducing that work will almost necessarily impact
> pre-fork model programs (e.g. they expect the init to be done before
> the fork).
> 
> As someone who uses that model heavily, I would still be okay with
> the "lazification" as I think there are many more programs that
> would be helped vs the ones hurt.  Initializing everything that your
> program might possibibly need right at startup time doesn't seem
> like a goal to strive for.  I can understand if you have a different
> opinion though.
> 
> A third approach would be to do more init work at compile time.
> E.g. for re.compile, if the compiled result could be stored in the
> .pyc, that would eliminate a lot of time for short scripts and for
> long-running programs.  Some Lisp systems have "compiler macros".
> They are basically a hook to allow programs to do some work before
> the code is sent to the compiler.  If something like that existed in
> Python, it could be used by re.compile to generate a compiled
> representation of the regex to store in the .pyc file.  That kind of
> behavior is pretty different than the "there is only runtime" model
> that Python generally tries to follow.
> 
> Spit-ball idea, thought up just now:
> 
> PAT = __compiled__(re.compile(...))
> 
> The expression in __compiled__(..) would be evaluated by the
> compiler and the resulting value would become the value to store in
> th .pyc.  If you are running the code as the script, __compiled__
> just returns its argument unchanged.
> 
> Cheers,
> 
>   Neil
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/desmoulinmichel%40gmail.com
> 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup optimization: script vs. service

[Guido van Rossum]

Let's kick this part of the discussion back to python-ideas.

On Tue, Oct 17, 2017 at 1:36 PM, Michel Desmoulin  wrote:

> Maybe it's time to bring back the debate on the "lazy" keyword then ?
> Rendering any statement arbitrarily lazy could help with perfs. It would
> also make hacks like ugettext_lazy in Django useless. And would render
> moot the extensions of f-strings for lazily rendered ones. And bring
> lazy imports in the mix.
>
> Le 17/10/2017 à 19:39, Neil Schemenauer a écrit :
> > Christian Heimes  wrote:
> >> That approach could work, but I think that it is the wrong
> >> approach. I'd rather keep Python optimized for long-running
> >> processes and introduce a new mode / option to optimize for
> >> short-running scripts.
> >
> > Another idea is to run a fake transasaction through the process
> > before forking.  That will "warm up" things so that most of the lazy
> > init is already done.
> >
> > After returning from the core sprint, I have gotten over my initial
> > enthusiam for my "lazy module defs" idea.  It is just too big of a
> > change for Python to accept that this point.  I still hope there
> > would be a way to make LOAD_NAME/LOAD_GLOBAL trigger something like
> > __getattr__().  That would allow libraries that want to aggressively
> > do lazy-init to do so in the clean way.
> >
> > The main reason that Python startup is slow is that we do far too
> > much work on module import (e.g. initializing data structures that
> > never get used).  Reducing that work will almost necessarily impact
> > pre-fork model programs (e.g. they expect the init to be done before
> > the fork).
> >
> > As someone who uses that model heavily, I would still be okay with
> > the "lazification" as I think there are many more programs that
> > would be helped vs the ones hurt.  Initializing everything that your
> > program might possibibly need right at startup time doesn't seem
> > like a goal to strive for.  I can understand if you have a different
> > opinion though.
> >
> > A third approach would be to do more init work at compile time.
> > E.g. for re.compile, if the compiled result could be stored in the
> > .pyc, that would eliminate a lot of time for short scripts and for
> > long-running programs.  Some Lisp systems have "compiler macros".
> > They are basically a hook to allow programs to do some work before
> > the code is sent to the compiler.  If something like that existed in
> > Python, it could be used by re.compile to generate a compiled
> > representation of the regex to store in the .pyc file.  That kind of
> > behavior is pretty different than the "there is only runtime" model
> > that Python generally tries to follow.
> >
> > Spit-ball idea, thought up just now:
> >
> > PAT = __compiled__(re.compile(...))
> >
> > The expression in __compiled__(..) would be evaluated by the
> > compiler and the resulting value would become the value to store in
> > th .pyc.  If you are running the code as the script, __compiled__
> > just returns its argument unchanged.
> >
> > Cheers,
> >
> >   Neil
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> desmoulinmichel%40gmail.com
> >
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

[Victor Stinner]

Antoine Pitrou:
> Why not ``time.process_time_ns()``?

I measured the minimum delta between two clock reads, ignoring zeros.
I tested time.process_time(), os.times(), resource.getrusage(), and
their nanosecond variants (with my WIP implementation of the PEP 564).

Linux:

* process_time_ns(): 1 ns
* process_time(): 2 ns
* resource.getrusage(): 1 us
   ru_usage structure uses timeval, so it makes sense
* clock(): 1 us
   CLOCKS_PER_SECOND = 1,000,000 => res = 1 us
* times_ns().elapsed, times().elapsed: 10 ms
   os.sysconf("SC_CLK_TCK") == HZ = 100 => res = 10 ms
* times_ns().user, times().user: 10 ms
   os.sysconf("SC_CLK_TCK") == HZ = 100 => res = 10 ms

Windows:

* process_time(), process_time_ns(): 15.6 ms
* os.times().user, os.times_ns().user: 15.6 ms

Note: I didn't test os.wait3() and os.wait4(), but they also use the
ru_usage structure and so probably also have a resolution of 1 us.

It looks like *currently*, only time.process_time() has a resolution
in nanoseconds (smaller than 1 us). I propose to only add
time.process_time_ns(), as you proposed.

We might add nanosecond variant for the other functions once operating
systems will add new functions with better resolution.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

[Victor Stinner]

I updated my PEP 564 to add time.process_time_ns():
https://github.com/python/peps/blob/master/pep-0564.rst

The HTML version should be updated shortly:
https://www.python.org/dev/peps/pep-0564/

I better explained why some functions got a new nanosecond variant,
whereas others don't. The rationale is the precision loss affecting
only a few functions in practice.

I completed the "Annex: Clocks Resolution in Python" with more
numbers, again, to explain why some functions don't need a nanosecond
variant.

Thanks Antoine, the PEP now looks better to me :-)

Victor

2017-10-18 0:05 GMT+02:00 Victor Stinner :
> Antoine Pitrou:
>> Why not ``time.process_time_ns()``?
>
> I measured the minimum delta between two clock reads, ignoring zeros.
> I tested time.process_time(), os.times(), resource.getrusage(), and
> their nanosecond variants (with my WIP implementation of the PEP 564).
>
> Linux:
>
> * process_time_ns(): 1 ns
> * process_time(): 2 ns
> * resource.getrusage(): 1 us
>ru_usage structure uses timeval, so it makes sense
> * clock(): 1 us
>CLOCKS_PER_SECOND = 1,000,000 => res = 1 us
> * times_ns().elapsed, times().elapsed: 10 ms
>os.sysconf("SC_CLK_TCK") == HZ = 100 => res = 10 ms
> * times_ns().user, times().user: 10 ms
>os.sysconf("SC_CLK_TCK") == HZ = 100 => res = 10 ms
>
> Windows:
>
> * process_time(), process_time_ns(): 15.6 ms
> * os.times().user, os.times_ns().user: 15.6 ms
>
> Note: I didn't test os.wait3() and os.wait4(), but they also use the
> ru_usage structure and so probably also have a resolution of 1 us.
>
> It looks like *currently*, only time.process_time() has a resolution
> in nanoseconds (smaller than 1 us). I propose to only add
> time.process_time_ns(), as you proposed.
>
> We might add nanosecond variant for the other functions once operating
> systems will add new functions with better resolution.
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Nick Coghlan]

On 18 October 2017 at 05:55, Yury Selivanov  wrote:

> On Tue, Oct 17, 2017 at 2:25 PM, Guido van Rossum 
> wrote:
> > On Tue, Oct 17, 2017 at 8:54 AM, Yury Selivanov  >
> [..]
> >> My way of thinking about this: "get_execution_context()" returns you a
> >> shallow copy of the current EC (at least conceptually).  So making any
> >> modifications on it won't affect the current environment.  The only
> >> way to actually apply the modified EC object to the environment will
> >> be its 'run(callable)' method.
> >
> >
> > I understand that you don't want to throw away the implementation work
> > you've already done. But I find that the abstractions you've introduced
> are
> > getting in the way of helping people understand what they can do with
> > context variables, and I really want to go back to a model that is *much*
> > closer to understanding how instance variables are just self.__dict__.
> (Even
> > though there are possible complications due to __slots__ and @property.)
>
> I don't really care about the implementation work that has already
> been done, it's OK if I write it from scratch again.
>
> I actually like what you did in
> https://github.com/gvanrossum/pep550/blob/master/simpler.py, it seems
> reasonable.  The only thing that I'd change is to remove "set_ctx"
> from the public API and add "Context.run(callable)".  This makes the
> API more flexible to potential future changes and amendments.
>

Yep, with that tweak, I like Guido's suggested API as well.


Attempting to explain why I think we want "Context.run(callable)" rather
"context_vars.set_ctx()" by drawing an analogy to thread local storage:

1. In C, the compiler & CPU work together to ensure you can't access
another thread's thread locals.
2. In Python's thread locals API, we do the same thing: you can only get
access to the running thread's thread locals, not anyone else's

At the Python API layer, we don't expose the ability to switch explicitly
to another thread state while remaining within the current function.
Instead, we only offer two options: starting a new thread, and waiting for
a thread to finish execution. The lifecycle of the thread local storage is
then intrinsically linked to the lifecycle of the thread it belongs to.

That intrinsic link makes various aspects of thread local storage easier to
reason about, since the active thread state can't change in the middle of a
running function - even if the current thread gets suspended by the OS,
resuming the function also implies resuming the original thread.

Including a "contextvars.set_ctx" API would be akin to making
PyThreadState_Swap a public Python-level API, rather than only exposing
_thread.start_new_thread the way we do now.

One reason we *don't* do that is because it would make thread locals much
harder to reason about - every function call could have an implicit side
effect of changing the active thread state, which would mean the thread
locals at the start of the function could differ from those at the end of
the function, even if the function itself didn't do anything to change them.

Only offering Context.run(callable) provides a similar "the only changes to
the execution context will be those this function, or a function it called,
explicitly initiates" protection for context variables, and Guido's
requested API simplifications make this aspect even easier to reason about:
after any given function call, you can be certain of being back in the
context you started in, because we wouldn't expose any Python level API
that allowed an execution context switch to persist beyond the frame that
initiated it.



The above is my main rationale for preferring contextvars.Context.run() to
contextvars.set_ctx(), but it's not the only reason I prefer it.

At a more abstract design philosophy level, I think the distinction between
symmetric and asymmetric coroutines is relevant here [2]:

* in symmetric coroutines, there's a single operation that says "switch to
running this other coroutine"
* in asymmetric coroutines, there are separate operations for starting or
resuming coroutine and for suspending the currently running one

Python's native coroutines are asymmetric - we don't provide a "switch to
this coroutine" primitive, we instead provide an API for starting or
resuming a coroutine (via cr.__next__(), cr.send() & cr.throw()), and an
API for suspending one (via await).

The contextvars.set_ctx() API would be suitable for symmetric coroutines,
as there's no implied notion of parent context/child context, just a notion
of switching which context is active.

The Context.run() API aligns better with asymmetric coroutines, as there's
a clear distinction between the parent frame (the one initiating the
context switch) and the child frame (the one running in the designated
context).

As a practical matter, Context.run also composes nicely (in combination
with functools.partial) for use with any existing API based on submitting
functions for delayed execution, or execution in

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Ethan Furman]

On 10/17/2017 09:40 PM, Nick Coghlan wrote:

On 18 October 2017 at 05:55, Yury Selivanov wrote:



I actually like what you did in
https://github.com/gvanrossum/pep550/blob/master/simpler.py
, it seems
reasonable.  The only thing that I'd change is to remove "set_ctx"
from the public API and add "Context.run(callable)".  This makes the
API more flexible to potential future changes and amendments.


Yep, with that tweak, I like Guido's suggested API as well.


Attempting to explain why I think we want "Context.run(callable)" rather 
"context_vars.set_ctx()" by drawing an analogy
to thread local storage:

1. In C, the compiler & CPU work together to ensure you can't access another 
thread's thread locals.
2. In Python's thread locals API, we do the same thing: you can only get access 
to the running thread's thread locals,
not anyone else's

At the Python API layer, we don't expose the ability to switch explicitly to 
another thread state while remaining within
the current function. Instead, we only offer two options: starting a new 
thread, and waiting for a thread to finish
execution. The lifecycle of the thread local storage is then intrinsically 
linked to the lifecycle of the thread it
belongs to.


I seem to remember mention about frameworks being able to modify contexts for various tasks/coroutines; if the framework 
cannot create and switch to a new context how will it set them up?  Or did I misunderstand?


--
~Ethan~
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 510 (function specialization) rejected

[Nick Coghlan]

On 18 October 2017 at 06:25, Guido van Rossum  wrote:

> It takes courage to admit failures like this! I think this is a good call.
> It echoes the experiences with Unladen Swallow and Pyston.
>

And Armin Rigo's experience with psyco before that.

Despite what people may think, CPython really isn't slow, given the large
> set of constraints on the implementation.
>

Antonio Cuni had a good PyPy presentation at EuroPython indirectly talking
about the fact that when folks say "Python is slow", what they often mean
is "Many of Python's conceptual abstractions come at a high runtime cost in
the reference implementation":
https://speakerdeck.com/antocuni/the-joy-of-pypy-jit-abstractions-for-free

That means the general language level performance pay-offs for alternative
implementations come from working out how to make the abstraction layers
cheaper, as experience shows that opt-in ahead-of-time techniques like
Cython, vectorisation, and binary extension modules can do a much better
job of dealing with the clearly identifiable low level performance
bottlenecks (Readers that aren't familiar with the concept may be
interested in [1] as a good recent example of the effectiveness of the
latter approach).

Cheers,
Nick.

[1]
https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Timeout for PEP 550 / Execution Context discussion

[Nick Coghlan]

On 18 October 2017 at 16:25, Ethan Furman  wrote:

> On 10/17/2017 09:40 PM, Nick Coghlan wrote:
>
>> At the Python API layer, we don't expose the ability to switch explicitly
>> to another thread state while remaining within
>> the current function. Instead, we only offer two options: starting a new
>> thread, and waiting for a thread to finish
>> execution. The lifecycle of the thread local storage is then
>> intrinsically linked to the lifecycle of the thread it
>> belongs to.
>>
>
> I seem to remember mention about frameworks being able to modify contexts
> for various tasks/coroutines; if the framework cannot create and switch to
> a new context how will it set them up?  Or did I misunderstand?


That's what Context.run() will handle.

>From a usage perspective, what Yury and I are suggesting is that switching
execution to a new context should always implicitly switch back when the
called operation returns:

def run_in_isolated_ec(operation):
ec = contextvars.new_context()
return ec.run(operation)

That's safe and simple to use, and integrates nicely with the APIs for
resuming coroutines (next(cr), cr.__next__(), cr.send(), cr.throw()).

By contrast, exposing set_ctx() directly puts the responsibility for
reverting the active context back to the initial one on the caller:

def run_in_isolated_ec(operation):
ec = contextvars.new_context()
initial_ec = contextvar.get_ctx()
contextvars.set_ctx(ec)
try:
return operation()
finally:
# What happens if we forget to revert back to the previous
context here?
contextvars.set_ctx(initial_ec)

While the contextvars implementation is going to need a context switching
capability like that internally in order to implement ec.run() (just as the
threading implementation's C API includes PyThreadState_Swap), we don't
currently have a use case for exposing it as a Python level API.

And if we don't expose an API that allows it, then we can delay specifying
what effect the following code should have all the calling function or
coroutine (or whether it's a runtime error):

def buggy_run_in_isolated_ec(operation):
ec = contextvars.new_context()
contextvars.set_ctx(ec)
return operation()
# Oops, forgot to revert the active context to the one we were
called with

So if we start out only exposing "Context.run()" at the Python layer
(covering all currently known use cases), then any subsequent addition of
an in-place context switching API can be guided by specific examples of
situations where Context.run() isn't sufficient.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com