[Python-Dev] Best Python API for exposing posix_spawn

[Pablo Galindo Salgado]

Hi,

I'm currently working on exposing posix_spawn in the posix module (and by
extension in the os module). You can find the initial implementation in
this PR:

https://github.com/python/cpython/pull/5109

As pointed out by Gregory P. Smith, some changes are needed in the way the
file_actions arguments is passed from Python. For context, posix_spawn has
the following declaration:

int posix_spawn(pid_t *pid, const char *path,
const posix_spawn_file_actions_t *file_actions,
const posix_spawnattr_t *attrp,
char *const argv[], char *const envp[]);

Here, file_actions is an object that represents a list of file actions
(open, close or dup2) that is populated using helper functions on the C API.

The question is: what is the best way to deal with this argument?

Following Gregory's comment on the PR I understand that he is proposing to
have three objects in the os module representing each action and pass a
sequence of these objects to the Python API. What I am not sure about this
is that there is no previous example of such classes in the os module for
other similar APIs and therefore I am not sure if there is a better
approach.

Thanks you very much for your time!

Pablo Galindo
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Pablo Galindo Salgado]

I think I really like Antoine's suggestion so I'm going to finish
implementing it that way. I think this keeps the API simple, does not bring
in the os module new dependencies, keeps the C implementation clean and is
consistent with the rest of the posix module. I will post an update when is
ready.

Thank you everyone for sharing your view and advice!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Best Python API for exposing posix_spawn

[Pablo Galindo Salgado]

Thank you everyone that commented in this thread about the best interface
for posix_spawn. I have finished implementing Antoine's suggestion in the
PR:

https://github.com/python/cpython/pull/5109

I think it would be good if we can have this merged before the feature lock
at the end of the month if possible.

Thanks you very much everyone for your time and suggestions!

On Wed, 10 Jan 2018, 09:17 Pablo Galindo Salgado, 
wrote:

> I think I really like Antoine's suggestion so I'm going to finish
> implementing it that way. I think this keeps the API simple, does not bring
> in the os module new dependencies, keeps the C implementation clean and is
> consistent with the rest of the posix module. I will post an update when is
> ready.
>
> Thank you everyone for sharing your view and advice!
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Exposing different versions of a system call in Python

[Pablo Galindo Salgado]

Hello everyone,

In today's episode of exposing useful Linux system calls I am exposing
preadv2 in this PR:

https://github.com/python/cpython/pull/5239

as requested in this issue:

https://bugs.python.org/issue31368

As njsmith has commented in the PR, preadv2 only exists because regular
preadv was missing a flags argument, and in C the only way to add an
argument is to make a new function. In Python we have already exposed
preadv2 and a possible solution would be add a optional argument that
passes the new parametera to preadv and calls preadv2 if this happens.

On the other side, we have pipe and pipe2 as an example of exposing two
versions when this situation happens.

The question is:

What is preferable, exposing both functions or augment the old one?

Thank you everyone for your time!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] sys.settrace does not produce events for C functions

[Pablo Galindo Salgado]

The docs for sys.settrace mention that:

>> event is a string: 'call', 'line', 'return', 'exception', >> 'c_call',
'c_return', or 'c_exception'

But in the code for ceval.c the only point where call_trace is invoked with
PyTrace_C_CALL or PyTrace_C_RETURN is under the C_TRACE macro. In this
macro this line prevents any function set up using sys.settrace to call
call_trace with the mentioned arguments:

if (tstate->use_tracing && tstate->c_profilefunc)

Notice that from the code of PyEval_SetTrace and PyEval_SetProfile, only
the later sets tstate->c_profilefunc and therefore only functions installed
using sys.setprofile will recieve a c_call for the event.

Xiang Zhan has suggested me to ask here what is the best course of action:

1) Document this behavior.

2) Fix the code.

This question is related to this issue:
https://bugs.python.org/issue17799

Thanks everyone for your time!

Pablo
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] building extensions as builtins is broken in 3.7

[Pablo Galindo Salgado]

Hi,

I am working on the release blocker https://bugs.python.org/issue32232. I
tried to apply the patch proposed by Matthias Klose and I found that it
works on Unix but it fails to build on Windows (probably because circular
imports). I tried to add some tests but after some work on the problem and
some comments by Ned Deily I am starting to think that the test for this is
basically compiling the whole interpreter with static linking of the
built-ins and run the whole test suite. This has to be done as a new build
target on Travis. I did some experiments with this following:

https://wiki.python.org/moin/BuildStatically

But I run into multiple linking errors. The Travis build hangs forever and
then fails in the best case.

Here is an example of what I am talking:
https://github.com/pablogsal/cpython/pull/1

In this PR you can see the patch and the static linking configured as per
above and the Travis output.

I am happy to work on this as long as someone can tell me what is the
appropriate course of action.

Thank you very much for your time!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

[Pablo Galindo Salgado]

> > Pablo's issue35378 evolved to add a weak reference in iterators to try
> > to detect when the Pool is destroyed: raise an exception from the
> > iterator, if possible.

> That's an ok fix for me.

I am playing with weakreferences inside the iterator and result objects,
but this may not be enough/a complete solution. For example, take the
code of ApplyResult.get:

def get(self, timeout=None):
if self._pool() is None:   # self._pool is a weakref
raise RuntimeError("The pool is dead. Aborting") <--- new code
self.wait(timeout)

It can be that the pool is alive when we check for it (self._pool() is
None) but while
the code is waiting with no timeout (timeout=None) the pool dies,
effectively leaving the
program deadlocked with no error.

--

I agree that misusage of the pool should not be encouraged but in this
situation the fact that
this code hangs:

import multiprocessing

for x in multiprocessing.Pool().imap(int, ["4", "3"]):
print(x)


is a bit worriying because although is incorrect and an abuse of the
API, users can do this easily with
no error message other than a misterious hang. I have found this on
several places and people were
very confused because usually the interpreter throws some kind of
error indication. In my humble opinion,
we should try to avoid hanging as a consequence of the misusage, whatever we do.

Pablo
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

[Pablo Galindo Salgado]

>
> Your original solution would have added a strong reference back to the
> pool from the iterator. At first glance, that seems like a reasonable
> solution to me. Victor is worried about the "risk of new reference
> cycles". But reference cycles are not a problem - we have the cyclic
> GC precisely to deal with them. So I'd like to see a better
> justification for rejecting that solution than "there might be
> reference cycles". But in response to that, you made the iterator have
> a weak reference back to the pool. That's flawed because it doesn't
> prevent the pool being terminated - as you say, the deadlock is still
> present.


Just to be clear: I am in favour of the strong reference, but I also
understand the "danger" (leaking the pool until the pool's generation
reaches the threshold and the gc runs) and that is the reason I was
experimenting with the weakreference.

On Tue, 11 Dec 2018 at 18:37, Paul Moore  wrote:

> On Tue, 11 Dec 2018 at 17:50, Pablo Galindo Salgado 
> wrote:
> > I agree that misusage of the pool should not be encouraged but in this
> situation the fact that
> > this code hangs:
> >
> > import multiprocessing
> >
> > for x in multiprocessing.Pool().imap(int, ["4", "3"]):
> > print(x)
> >
> >
> > is a bit worriying because although is incorrect and an abuse of the
> API, users can do this easily with
> > no error message other than a misterious hang.
>
> OK, so the first problem here (to me, at least) is that it's not
> obvious *why* this code is incorrect and an abuse of the API. It takes
> a reasonable amount of thinking about the problem to notice that the
> Pool object isn't retained, but the iterator returned from imap is.
> And when the pool is collected, the worker processes are terminated,
> causing the hang, as the worker never sends a result back to the main
> process. But it's not obvious to me why the pool is collected before
> the imap method has completed.
>
> As I understand it, originally the code worked because the pool
> *didn't* call terminate() when collected. Now it does, and we have a
> problem. I'm not *entirely* sure why, if the pool is terminated, the
> wait in the iterator doesn't terminate immediately with some sort of
> "process being waited on died" error, but let's assume there are good
> reasons for that (as I mentioned before, I'm not an expert in
> multiprocessing, so I'm OK with assuming that the original design,
> done by people who *are* experts, is sound :-))
>
> Your original solution would have added a strong reference back to the
> pool from the iterator. At first glance, that seems like a reasonable
> solution to me. Victor is worried about the "risk of new reference
> cycles". But reference cycles are not a problem - we have the cyclic
> GC precisely to deal with them. So I'd like to see a better
> justification for rejecting that solution than "there might be
> reference cycles". But in response to that, you made the iterator have
> a weak reference back to the pool. That's flawed because it doesn't
> prevent the pool being terminated - as you say, the deadlock is still
> present.
>
> > I have found this on several places and people were
> > very confused because usually the interpreter throws some kind of error
> indication. In my humble opinion,
> > we should try to avoid hanging as a consequence of the misusage,
> whatever we do.
>
> I agree with this. But that implies to me that we should be holding a
> strong reference to the pool,
>
> As a (somewhat weak) analogy, consider
>
> for n in map(int, ["1", "2"]):
> print(n)
>
> That won't fail if the list gets collected, because map keeps a
> reference to the list. My intuition would be that the Pool().imap
> example would hold a reference to the pool on essentially the same
> basis.
>
> The more I think about this, the more I struggle to see Victor's logic
> for rejecting your original solution. And I *certainly* don't see why
> this issue should justify changing the whole API to require users to
> explicitly manage pool lifetimes.
>
> Paul
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 570

[Pablo Galindo Salgado]

Hi everyone,

We have submitted PEP 570 for consideration to the steering council:

https://github.com/python/steering-council/issues/4

The discussion is happening on discourse:

https://discuss.python.org/t/pep-570-python-positional-only-parameters/1078

To eliminate splitting the discussion into two forums (discourse and
python-dev),
we kindly ask you to go to discourse if you want to participate in the
debate :)

Here is the full document of the PEP:

https://www.python.org/dev/peps/pep-0570/

Pablo Galindo Salgado
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Parser module in the stdlib

[Pablo Galindo Salgado]

Hi everyone,

TLDR
=

I propose to remove the current parser module and expose pgen2 as a
standard library module.

Some context
===

The parser module has been "deprecated" (technically we recommend to prefer
the ast module instead) since Python2.5 but is still in the standard
library.
Is a 1222-line C module that needs to be kept in sync with the grammar and
the Python parser sometimes by hand. It has also been broken for several
years:
I recently fixed a bug that was introduced in Python 3.5 that caused the
parse module to not being able to parse if-else blocks (
https://bugs.python.org/issue36256).

The interface it provides is a very raw view of the CST. For instance:

>>> parser.sequence2st(parser.suite("def f(x,y,z): pass").totuple())

provides an object with methods compile, isexpr, issuite, tolist, totuple.
The last two produce containers with the numerical values of the grammar
elements (tokens, dfas...etc)

>>> parser.suite("def f(x,y,z): pass").tolist()
[257, [269, [295, [263, [1, 'def'], [1, 'f'], [264, [7, '('], [265, [266,
[1, 'x']], [12, ','], [266, [1, 'y']], [12, ','], [266, [1, 'z']]], [8,
')']], [11, ':'], [304, [270, [271, [277, [1, 'pass']]], [4, '']], [4,
''], [0, '']]

This is a very raw interface and is very tied to the particularities of
CPython without almost any abstraction.

On the other hand, there is a Python implementation of the Python parser
and parser generator in lib2to3 (pgen2). This module is not documented and
is usually considered an implementation
detail of lib2to3 but is extremely useful. Several 3rd party packages
(black, fissix...) are using it directly or have their own forks due to the
fact that it can get outdated with respect to the Python3
grammar as it was originally used only for Python2 to Python3 migration. It
has the ability to consume LL1 grammars in EBNF form and produces an LL1
parser for them (by creating parser tables
that the same module can consume). Many people use the module currently to
support or parse supersets of Python (like Python2/3 compatible grammars,
cython-like changes...etc).

Proposition


I propose to remove finally the parser module as it has been "deprecated"
for a long time, is almost clear that nobody uses it and has very limited
usability and replace it (maybe with a different name)
with pgen2 (maybe with a more generic interface that is detached to lib2to3
particularities). This will not only help a lot current libraries that are
using forks or similar solutions but also will help to keep
synchronized the shipped grammar (that is able to parse Python2 and Python3
code) with the current Python one (as now will be more justified to keep
them in sync).

What do people think about? Do you have any concerns? Do you think is a
good/bad idea?

Regards from sunny London,
Pablo Galindo
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Pablo Galindo Salgado]

> Will the folks using forks be happy to switch to the stdlib version?
>For example I can imagine that if black wants to process 3.7 input
>code while running on 3.6, it might prefer a parser on PyPI even if
>he stdlib version were public, since the PyPI version can be updated
>independently of the host Python.
The tool can parse arbitrary grammars, the one that is packed into is just
one of them.

I think it would be useful, among other things because the standard library
lacks currently a proper CST solution. The ast module is heavily leveraged
for
things like formatters, static code analyzers...etc but CST can be very
useful as
Łukasz describes here:

https://bugs.python.org/issue7

I think is missing an important gap in the stdlib and the closest thing we
have
(the current parser module) is not useful for any of that. Also, the core
to generating
the hypothetical new package (with some new API over it may be) is already
undocumented
as an implementation detail of lib2to3 (and some people are already using
it directly).



On Thu, 16 May 2019 at 23:41, Nathaniel Smith  wrote:

> On Thu, May 16, 2019 at 2:13 PM Pablo Galindo Salgado
>  wrote:
> > I propose to remove finally the parser module as it has been
> "deprecated" for a long time, is almost clear that nobody uses it and has
> very limited usability and replace it (maybe with a different name)
> > with pgen2 (maybe with a more generic interface that is detached to
> lib2to3 particularities). This will not only help a lot current libraries
> that are using forks or similar solutions but also will help to keep
> > synchronized the shipped grammar (that is able to parse Python2 and
> Python3 code) with the current Python one (as now will be more justified to
> keep them in sync).
>
> Will the folks using forks be happy to switch to the stdlib version?
> For example I can imagine that if black wants to process 3.7 input
> code while running on 3.6, it might prefer a parser on PyPI even if
> the stdlib version were public, since the PyPI version can be updated
> independently of the host Python.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Parser module in the stdlib

[Pablo Galindo Salgado]

> Actually, I think the `ast` module doesn't work very well for formatters,
because it loses comments. (Retaining comments and all details of
whitespace is the specific use case for which I created pgen2.)

Some uses I have seen include using it to check that the code before and
after the formatting has no functional changes (both have the same ast) or
to augment the information obtained with other sources. But yeah, I agree
that static code analyzers and linters are a much bigger target.

>I wonder if lib2to3 is actually something that would benefit from moving
out of the stdlib. (Wasn't it on Amber's list?) As Łukasz points out in
that issue, it is outdated. Maybe if it was out of the stdlib it would
attract more contributors. Then again, I have recently started exploring
the idea of a PEG parser for Python. Maybe a revamped version of the core
of lib2to3 based on PEG instead of pgen would be interesting to some folks.

I was thinking more on the line of leveraging some parts lib2to3 having
some CST-related solution similar to the ast module, not exposing the whole
functionality of lib2to3. Basically, it would be a more high-level
abstraction to substitute the current parser module. Technically you should
be able to reconstruct some primitives that lib2to3 uses on top of the
output that the parser module generates (modulo some extra information from
the grammar), but the raw output that the parser module generates is not
super useful by itself, especially when you consider the maintenance costs.

On the other side, as you mention here:

>I am interested in switching CPython's parsing strategy to something else
(what exactly remains to be seen) and any new approach is unlikely to reuse
the current CST technology. (OTOH I think it would be wise to keep the
current AST.)

it is true that changing the parser can influence greatly the hypothetical
CST module so it may complicate the conversion to a new parser solution if
the API does not abstract enough (or it may be close to impractical
depending on the new parser solution).

My original suggestion was based on the fact that the parser module is not
super useful and it has a great maintenance cost, but the "realm" of what
it solves (providing access to the parse trees) could be useful to some use
cases so that is why I was talking about "parser" and lib2to3 in the same
email.

Perhaps we can be more productive if we focus on just deprecating the
"parser" module, but I thought it was an opportunity to solve two (related)
problems at once.


On Mon, 20 May 2019 at 17:28, Guido van Rossum  wrote:

> On Thu, May 16, 2019 at 3:51 PM Pablo Galindo Salgado 
> wrote:
>
>> [Nathaniel Smith]
>>
> >Will the folks using forks be happy to switch to the stdlib version?
>> >For example I can imagine that if black wants to process 3.7 input
>> >code while running on 3.6, it might prefer a parser on PyPI even if
>> >he stdlib version were public, since the PyPI version can be updated
>> >independently of the host Python.
>>
>> The tool can parse arbitrary grammars, the one that is packed into is
>> just one of them.
>>
>> I think it would be useful, among other things because the standard
>> library
>> lacks currently a proper CST solution. The ast module is heavily
>> leveraged for
>> things like formatters,
>>
>
> Actually, I think the `ast` module doesn't work very well for formatters,
> because it loses comments. (Retaining comments and all details of
> whitespace is the specific use case for which I created pgen2.)
>
>
>> static code analyzers...etc but CST can be very useful as
>> Łukasz describes here:
>>
>> https://bugs.python.org/issue7
>>
>> I think is missing an important gap in the stdlib and the closest thing
>> we have
>> (the current parser module) is not useful for any of that. Also, the core
>> to generating
>> the hypothetical new package (with some new API over it may be) is
>> already undocumented
>> as an implementation detail of lib2to3 (and some people are already using
>> it directly).
>>
>
> I wonder if lib2to3 is actually something that would benefit from moving
> out of the stdlib. (Wasn't it on Amber's list?) As Łukasz points out in
> that issue, it is outdated. Maybe if it was out of the stdlib it would
> attract more contributors. Then again, I have recently started exploring
> the idea of a PEG parser for Python. Maybe a revamped version of the core
> of lib2to3 based on PEG instead of pgen would be interesting to some folks.
>
> I do agree that the two versions of tokenize.py should be unified (and the
> result kept in the stdlib). However there are some issues here, because
> tokenize.py is closely tied to the 

Re: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures

[Pablo Galindo Salgado]

>
> I propose to make co_argcount meaning the number of positional
> parameters (i.e. positional-only + positional-or-keyword). This would
> remove the need of changing the code that uses co_argcount.
>

I like the proposal, it will certainly make handling normal cases
downstream much easier because
if you do not care about positional-only arguments you can keep
inspecting co_argcount
and that
will give you what you expect. Note that if we choose to do this, it has to
be done now-ish IMHO to
avoid making the change painful because it will change the semantics of
co_argcount.


> As for the code object constructor, I propose to make posonlyargcount an
> optional parameter (default 0) added after existing parameters.
> PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or
> PyCode_NewEx() with different signature.


I am not convinced about having a default argument in the code constructor.
The code constructor
is kept with all arguments positional for efficiency and adding defaults
will make it slower or having
a more confusing an asymmetrical interface. Also, this will be misaligned
on how keyword-only
parameters are provided. This is by far not the first time this constructor
has changed.

On the Python side, the new code.replace should cover most of the
Python-side use cases regarding
creating code objects from the Python side.


license.dash-license
Description: Binary data
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures

[Pablo Galindo Salgado]

Opened https://bugs.python.org/issue37122 to track this in the bug tracker.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Recent buildbot reports and asyncio test failures

[Pablo Galindo Salgado]

Hi everyone,

Just a heads-up regarding some messages you will see in your pull requests.
There is an intermittent failure on some buildbots
regarding asyncio:

https://buildbot.python.org/all/#/builders/21

As the builds do not fail all the time, the systems understand that if your
(merged) commits fail to build, they may be the cause
of the failure and then it does a report into the pull request.

I am working on investigating a way to improve the report mechanism to make
it less noisy in this case, but bear in mind that
the correct way to solve this is fixing the asyncio bug in the test suite
and this won't likely go away completely until is solved.

We are doing all that we can to solve all the recent leaks and failures on
the test suite, but there is a noticeable increase in the
number of merged pull requests because of the imminent feature freeze and
because this happens across several timezones
is very difficult to get them all.

Thanks to everyone that is helping solving these bugs :)

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

Hi,

Recently, we moved the optimization for the removal of dead code of the form

if 0:
  

to the ast so we use JUMP bytecodes instead (being completed in PR14116).
The reason is that
currently, any syntax error in the block will never be reported. For
example:

if 0:
  return

if 1:
pass
else:
return

while 0:
return


at module level do not raise any syntax error (just some examples), In
https://bugs.python.org/issue37500 it was reported
that after that, code coverage will decrease as coverage.py sees these new
bytecodes (even if they are not executed). In general,
the code object is a bit bigger and the optimization now it requires an
JUMP instruction to be executed, but syntax errors are reported.

The discussion on issue 37500 is about if we should prioritize the
optimization or the correctness of reporting syntax errors. In my opinion,
SyntaxErrors should be reported with independence of the value of variables
(__debug__) or constant as is a property of the code being written
not of the code being executed. Also, as CPython is the reference
implementation of Python, the danger here is that it could be interpreted
that
this optimization is part of the language and its behavior should be
mirrored in every other Python implementation. Elsewhere we have always
prioritize correctness over speed or optimizations.

I am writing this email to know what other people think. Should we revert
the change and not report Syntax Errors on optimized blocks? Someone
sees a viable way of reporting the errors and not emitting the bytecode for
these block?

Regards,
Pablo
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RHP4LTM4MSTAPJHGMQGGHERYI4PZR23R/

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

> Until a solution which makes everyone happy can be found, I suggest to
move back to the status quo: revert the change.

>More people seems to expect "if 0: ..." to be removed, than people who
care of syntax errors on "if 0".

I don't think this is that clear. As Paul wrote on the issue this is the
result of fixing a bug that has been open since
2008 (11 years), which itself was noticed independently, also in 2008
(#1875 and #1920, respectively).
He also independently discovered the same issue last year when writing some
tests for IPython.

https://bugs.python.org/msg347394

On Fri, 5 Jul 2019 at 23:10, Victor Stinner  wrote:

> Hi,
>
> Until a solution which makes everyone happy can be found, I suggest to
> move back to the status quo: revert the change.
>
> More people seems to expect "if 0: ..." to be removed, than people who
> care of syntax errors on "if 0".
>
> --
>
> Would it be possible to detect if the "if 0" block would raise a
> syntax error, and only remove it if it doesn't raise a syntax error?
>
> That's the approach I chose in my fatoptimizer project which is
> implemented as an AST optimizer:
>
> https://github.com/vstinner/fatoptimizer/blob/master/fatoptimizer/dead_code.py
>
> See the tests to see which cases are *not* optimized:
>
> https://github.com/vstinner/fatoptimizer/blob/master/test_fatoptimizer.py#L2428
>
> Some examples (the "dead code elimitaiton" is not only about "if 0",
> but also "while 0", dead code after return, etc.):
>
> self.check_dont_optimize("""
> def func():
> if 0:
> yield
> """)
>
> self.check_dont_optimize("while 1: x = 1")
>
> self.check_dont_optimize("""
> def func(obj):
> return
> if 0:
> yield from obj
> """)
>
> self.check_dont_optimize("""
> try:
> pass
> except Exception:
> yield 3
> """)
>
> See also the doc:
> https://fatoptimizer.readthedocs.io/en/latest/optimizations.html#dead-code
>
> --
>
> About code coverage, it seems like -X noopt would help:
> https://github.com/python/cpython/pull/13600
>
> But I'm not sure anymore after Ned Batchelder wrote:
>
> "The real-word implications from my world are this: if your code has
> "if 0:" clauses in it, and you measure its coverage, then because the
> lines have not been optimized away, coverage.py will think the lines
> are a possible execution path, and will be considered a miss because
> they are not executed.  This will reduce your coverage percentage."
> https://bugs.python.org/issue37500#msg347362
>
> Does it mean that coverage.py will report even more "false positive"
> using -X noopt?
>
> Victor
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LWZPZQ2IL67DPX3RC342TTOVCLKDSBTJ/

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

> You started your msg by saying "we moved the optimization", but that's
not so:  the optimization was eliminated.  So just finish "moving" it
;-)

PR14116 was "finishing" the moving by adding JUMPS (a weaker optimization
but still an optimization).

> We should revert the change until someone (not me ;-) ) thinks harder
about whether it's possible to get both.  There doesn't seem to be,
e.g., a conceptual problem with simply throwing away "if 0:" subtrees
from the AST before generating code, or from snipping the "if 1:"
guard off an "if 1:" block.

Ok, these were very good points. I am convinced. I agree that the better
thing to do at this pointis to revert this until we can think this a bit
more :)

I have made a PR to revert the change.

On Fri, 5 Jul 2019 at 23:27, Tim Peters  wrote:

> [Pablo Galindo Salgado ]
> > Recently, we moved the optimization for the removal of dead code of the
> form
> >
> > if 0:
> >   
> >
> > to the ast so we use JUMP bytecodes instead (being completed in
> PR14116). The
> > reason is that currently, any syntax error in the block will never be
> reported.
> > For example:
>
> "Any syntax error" is way overstated.  "Almost all" syntax errors will
> be reported.  The ones that won't be aren't "really" about syntax (as
> most people view it). but more restrictions on the context in which
> certain statements can appear:
>
> > if 0:
> >   return
> >
> > if 1:
> > pass
> > else:
> > return
> >
> > while 0:
> > return
> >
> >
> > at module level do not raise any syntax error (just some examples),
>
> Whereas in function scope, those are all fine, with "0" or "1".  It's
> the "module level" context that matters to those.  Regardless of
> context, a syntax error that's actually about syntax ;-) is reported:
>
> if 0:
> x +
>
>
> > In https://bugs.python.org/issue37500 it was reported
> > that after that, code coverage will decrease as coverage.py sees these
> > new bytecodes (even if they are not executed). In general,
> > the code object is a bit bigger and the optimization now it
> > requires an JUMP instruction to be executed, but syntax errors
> > are reported.
>
> And I added other gripes to the report.
>
> I have one file (temp.py) using "if 0:" over 400 times.  When I need
> to write a small program or example (for, e.g. a StackOverflow
> answer), I add a new "if 1:" block at the end, fiddle with it until
> it's ready, then change the "1:" to "0:".  So the snippets stay around
> forever, but are effectively commented out so have no effect
> (unless/until they're needed again).
>
> There are, of course, other ways to comment them out, but none so
> convenient.  For example, under the "if 1:" block, the code is already
> indented 4 spaces, so can be pasted exactly as-is into a StackOverflow
> answer.
>
> But it doesn't really matter what I happen to do:  I'm just one of
> millions of users, who have come to rely on this stuff for way over a
> decade.
>
> > The discussion on issue 37500 is about if we should prioritize the
> optimization
> > or the correctness of reporting syntax errors. In my opinion,
>
> Which doesn't really make sense unless it's logically _necessary_ to
> "pick just one - you can't have both".
>
> > SyntaxErrors should be reported with independence of the value of
> variables
> > (__debug__) or constant as is a property of the code being written not
> of the
> > code being executed. Also, as CPython is the reference implementation of
> Python,
> > the danger here is that it could be interpreted that this optimization
> is part of the
> > language and its behavior should be mirrored in every other Python
> implementation.
>
> It's been that way for at least 15 years (since Python 2.4, so it's
> hard to be worried about that now.  Indeed, Armin Rigo posted the
> original example, and he wasn't fooled a bit about intent ;-)
>
> > Elsewhere we have always prioritize correctness over speed or
> optimizations.
>
> When they're irreconcilably in conflict, and the cost of correctness
> isn't "just too much", sure.
>
>
> > I am writing this email to know what other people think. Should we
> revert the change
> > and not report Syntax Errors on optimized blocks? Someone sees a viable
> way of
> > reporting the errors and not emitting the bytecode for these 

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

I think this summarizes the situation very well :)

https://xkcd.com/1172/

On Fri, 5 Jul 2019 at 22:28, Pablo Galindo Salgado 
wrote:

> Hi,
>
> Recently, we moved the optimization for the removal of dead code of the
> form
>
> if 0:
>   
>
> to the ast so we use JUMP bytecodes instead (being completed in PR14116).
> The reason is that
> currently, any syntax error in the block will never be reported. For
> example:
>
> if 0:
>   return
>
> if 1:
> pass
> else:
> return
>
> while 0:
> return
>
>
> at module level do not raise any syntax error (just some examples), In
> https://bugs.python.org/issue37500 it was reported
> that after that, code coverage will decrease as coverage.py sees these new
> bytecodes (even if they are not executed). In general,
> the code object is a bit bigger and the optimization now it requires an
> JUMP instruction to be executed, but syntax errors are reported.
>
> The discussion on issue 37500 is about if we should prioritize the
> optimization or the correctness of reporting syntax errors. In my opinion,
> SyntaxErrors should be reported with independence of the value of
> variables (__debug__) or constant as is a property of the code being written
> not of the code being executed. Also, as CPython is the reference
> implementation of Python, the danger here is that it could be interpreted
> that
> this optimization is part of the language and its behavior should be
> mirrored in every other Python implementation. Elsewhere we have always
> prioritize correctness over speed or optimizations.
>
> I am writing this email to know what other people think. Should we revert
> the change and not report Syntax Errors on optimized blocks? Someone
> sees a viable way of reporting the errors and not emitting the bytecode
> for these block?
>
> Regards,
> Pablo
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/M4SMPVBNGTOFLTQP6LKO5C34OSSVWVUR/

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

Ok, I think I found a way to make everyone happy. I have updated the issue
and the PR with the solution.

Basically the idea is to add an attribute to the compiler that is checked
when emitting the bytecode
and if is false, we stop don't emit anything but we do check for errors.
Unless I am fundamentally wrong
(which is possible as is late here) is very simple and covers all the
cases.

It would be great if someone could review the PR to check if I missed
anything with this argument.

On Fri, 5 Jul 2019 at 22:28, Pablo Galindo Salgado 
wrote:

> Hi,
>
> Recently, we moved the optimization for the removal of dead code of the
> form
>
> if 0:
>   
>
> to the ast so we use JUMP bytecodes instead (being completed in PR14116).
> The reason is that
> currently, any syntax error in the block will never be reported. For
> example:
>
> if 0:
>   return
>
> if 1:
> pass
> else:
> return
>
> while 0:
> return
>
>
> at module level do not raise any syntax error (just some examples), In
> https://bugs.python.org/issue37500 it was reported
> that after that, code coverage will decrease as coverage.py sees these new
> bytecodes (even if they are not executed). In general,
> the code object is a bit bigger and the optimization now it requires an
> JUMP instruction to be executed, but syntax errors are reported.
>
> The discussion on issue 37500 is about if we should prioritize the
> optimization or the correctness of reporting syntax errors. In my opinion,
> SyntaxErrors should be reported with independence of the value of
> variables (__debug__) or constant as is a property of the code being written
> not of the code being executed. Also, as CPython is the reference
> implementation of Python, the danger here is that it could be interpreted
> that
> this optimization is part of the language and its behavior should be
> mirrored in every other Python implementation. Elsewhere we have always
> prioritize correctness over speed or optimizations.
>
> I am writing this email to know what other people think. Should we revert
> the change and not report Syntax Errors on optimized blocks? Someone
> sees a viable way of reporting the errors and not emitting the bytecode
> for these block?
>
> Regards,
> Pablo
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CFOVHXD5LIQ7J7KJABWYYLYLDA2R53EH/

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

Thanks Terry for your comments!

> I presume that most of us agree that the last option, doing both, would be 
> best, or at least agreeable. 

I created a PR reverting the change but I (may) have found a way of doing both 
things :)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FZKYJZXVSIDJPB5EIAC2KKVE42JXG55P/

[Python-Dev] Re: Removing dead bytecode vs reporting syntax errors

[Pablo Galindo Salgado]

>. If it doesn't, could this one optimization
be left in the peephole optimizer at bytecode level?

Sadly no, because at that time there is not enough information left to do 
things correctly. The problem manifest with constructs like

if something or __debug__:
  ...

You cannot just look at the bytecode before the POP_JUMP_IF_FALSE (or similar) 
because you don't know reliably which bytecodes correspond to the entire 
condition (at least that I know of).

My proposal (it seem that works, but is being reviewed still) is basically 
doing a compiler pass that just check for errors over the blocks that we don't 
want to generate bytecode instead of not visiting them at all. I think is the 
less intrusive and faster solution that I can think off.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/22SLXBQI6RSVUSCKW25YGQFESD5OCEBV/

[Python-Dev] Re: Modified parser does not seem to work. What am I doing wrong?

[Pablo Galindo Salgado]

Hi Stefano,

One of the problems you have is that the rule for slices has a negative
lookahead for the comma:

 slices[expr_ty]:
| a=slice !',' { a }

IIRC the reason that is there is to allow "x[3,]" to be parsed.

Also, to allow "a[k=3]" you need to to create a rule that allows skipping
the "slices" part of the subscript and allow the 2nd argument of
_Py_Subscript  to be NULL. For example:

| a=t_primary '[' k=kwargs ']' !t_lookahead { _Py_Subscript(a, NULL, k,
Store, EXTRA) }

Regards,
Pablo

On Fri, 16 Oct 2020 at 22:07, Stefano Borini 
wrote:

> Hello,
>
> I am trying to implement PEP-637, and I started modifying the parser
> and the grammar, but I don't know what I am missing.
>
> The PR is here
>
>
> https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-implementation-attempt-1?expand=1
>
> It includes other stuff but the core is that I extended the Subscript
> in the asdl to accept the keyword args
>
> | Subscript(expr value, expr slice, keyword* keywords, expr_context ctx)
>
> which seems to work:
>
> >>> ast.parse('a[3]').body[0].value.keywords
> []
>
> I also added a few "productions" (I believe they are called like this,
> my compiler theory is very approximated), one of which is now
>
> primary[expr_ty]:
> 
> | a=primary '[' b=slices c=[',' k=kwargs {k}]']' {
> _Py_Subscript(a, b, c, Load, EXTRA) }
>
> I also tried with additional formats like
>
> | a=primary '[' b=slices c=kwargs ']' { _Py_Subscript(a, b, c,
> Load, EXTRA) }
>
> or even just
>
> | a=primary '[' c=kwargs ']' { _Py_Subscript(a, NULL, c, Load, EXTRA) }
>
> but I always get a SyntaxError:
>
> >>> ast.parse('a[k=3]').body[0].value.keywords
> Traceback (most recent call last):
>   File "", line 1, in 
>   File "/Users/sbo/Work/Projects/stefanoborini/cpython/Lib/ast.py",
> line 50, in parse
> return compile(source, filename, mode, flags,
>   File "", line 1
> a[k=3]
>^
> SyntaxError: invalid syntax
>
> Note that I always recreated ast parser and pegen.c with 'make
> regen-all' and recompiled with make.
> I tried to enable debug and print the pegen.c debug, but it's a bit
> heavy and I could not immediately spot the issue. I suspect I am
> missing something somewhere.
>
> Thanks
>
>
> --
> Kind regards,
>
> Stefano Borini
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RZYDHYZPTPGLBX4VCR6HTYGJ3GL2CWIX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKEGBZW7WVBTNSLEM44AYIV6CWLSK5GO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Accepting PEP 626

[Pablo Galindo Salgado]

On behalf of the steering council, I am happy to announce that as
BDFL-Delegate I am
accepting PEP 626 -- Precise line numbers for debugging and other tools.
I am confident this PEP will result in a better experience for debuggers,
profilers and tools
that rely on tracing functions. All the existing concerns regarding
out-of-process debuggers
and profilers have been addressed by Mark in the latest version of the PEP.
The acceptance of
the PEP comes with the following requests:

* The PEP must be updated to explicitly state that the API functions
described in the
   "Out of process debuggers and profilers" must remain self-contained in
any potential
future modifications or enhancements.
* The PEP states that the "f_lineno" attribute of the code object will be
updated to point to
   the current line being executed even if tracing is off. Also, there were
some folks concerned with
   possible performance implications. Although in my view there is no
reason to think this will impact
   performance negatively, I would like us to confirm that indeed this is
the case before merging the
   implementation (with the pyperformance test suite, for example).

Congratulations Mark Shannon!

Thanks also to everyone else who provided feedback on this PEP!

Regards from rainy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5ETSBH4GP2UAJTAIQ575DHCUA35H64IR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 626

[Pablo Galindo Salgado]

> Performance compared to what?

Compared before the patch. The comparison that I mentioned is before and
after the PR with the PEP implementation.

> The current behavior of `f_lineno` is ill-defined, so mimicking it would
be tricky

Maybe I failed to express myself: that's fine, we don't need to mimick the
current behaviour of f_lineno or change anything in the PEP regarding that.
I just want to check that the new semantics do not slow down anything in a
subtle way.

> What's the reason for supposing that it will be slower?

There is no real concern, but as there were some conversations about
performance and the pep mentions that "the "f_lineno" attribute of the code
object will be updated to point the current line being executed" I just
want to make sure that updating that field on every bytecode line change
does not affect anything. Again, I am pretty sure it will be negligible
impact and the performance check should be just a routine confirmation.

Cheers,
Pablo

On Thu, 29 Oct 2020, 09:47 Mark Shannon,  wrote:

> Hi,
>
> That's great. Thanks Pablo.
>
> On 29/10/2020 1:32 am, Pablo Galindo Salgado wrote:
> > On behalf of the steering council, I am happy to announce that as
> > BDFL-Delegate I am
> > accepting PEP 626 -- Precise line numbers for debugging and other tools.
> > I am confident this PEP will result in a better experience for
> > debuggers, profilers and tools
> > that rely on tracing functions. All the existing concerns regarding
> > out-of-process debuggers
> > and profilers have been addressed by Mark in the latest version of the
> > PEP. The acceptance of
> > the PEP comes with the following requests:
> >
> > * The PEP must be updated to explicitly state that the API functions
> > described in the
> > "Out of process debuggers and profilers" must remain self-contained
> > in any potential
> >  future modifications or enhancements.
> > * The PEP states that the "f_lineno" attribute of the code object will
> > be updated to point to
> > the current line being executed even if tracing is off. Also, there
> > were some folks concerned with
> > possible performance implications. Although in my view there is no
> > reason to think this will impact
> > performance negatively, I would like us to confirm that indeed this
> > is the case before merging the
> > implementation (with the pyperformance test suite, for example).
>
> Performance compared to what?
> The current behavior of `f_lineno` is ill-defined, so mimicking it would
> be tricky.
>
> What's the reason for supposing that it will be slower?
>
> Cheers,
> Mark.
>
> >
> > Congratulations Mark Shannon!
> >
> > Thanks also toeveryone else who provided feedback on this PEP!
> >
> > Regards from rainy London,
> > Pablo Galindo Salgado
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BIUZGR4YSCTV5FFNSN2RSCGB6MC3U2RB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 626

[Pablo Galindo Salgado]

> The new semantics may well result in some slowdowns. That's stated in
the PEP. I don't think I can reliably isolate the effects of the (very
slight)
change in the behavior of f_lineno.

Ok, then let's make at least we measure the general slowdowns.

> When you say updating "field", are you thinking of a C struct?
That's an implementation detail.
The PEP states that the f_lineno *attribute* of the code object will be
updated.

Any of them, but as you mentioned isolating the effect of that it may be
not worth
it. Let's measure the impact of the whole change as a whole then.

Cheers,
Pablo


On Thu, 29 Oct 2020 at 10:55, Mark Shannon  wrote:

> Hi Pablo,
>
> On 29/10/2020 9:56 am, Pablo Galindo Salgado wrote:
> >  > Performance compared to what?
> >
> > Compared before the patch. The comparison that I mentioned is before and
> > after the PR with the PEP implementation.
>
> PEP 626 changes the line number table and the compiler.
> Such a comparison would not test the performance impact of the change to
> `f_lineno`, as it will likely be swamped by the other changes.
>
> >
> >  > The current behavior of `f_lineno` is ill-defined, so mimicking it
> would
> > be tricky
> >
> > Maybe I failed to express myself: that's fine, we don't need to mimick
> > the current behaviour of f_lineno or change anything in the PEP
> > regarding that. I just want to check that the new semantics do not slow
> > down anything in a subtle way.
>
> The new semantics may well result in some slowdowns. That's stated in
> the PEP.
> I don't think I can reliably isolate the effects of the (very slight)
> change in the behavior of f_lineno.
>
> >
> >  > What's the reason for supposing that it will be slower?
> >
> > There is no real concern, but as there were some conversations about
> > performance and the pep mentions that "the "f_lineno" attribute of the
> > code object will be updated to point the current line being executed" I
> > just want to make sure that updating that field on every bytecode line
> > change does not affect anything. Again, I am pretty sure it will be
> > negligible impact and the performance check should be just a routine
> > confirmation.
>
> When you say updating "field", are you thinking of a C struct?
> That's an implementation detail.
> The PEP states that the f_lineno *attribute* of the code object will be
> updated.
>
> Note that the behavior of 3.9 is weird in some cases:
>
> test.py:
>  import sys
>
>  def print_line():
>  print(sys._getframe(1).f_lineno)
>
>  def test():
>  print_line()
>  sys._getframe(0).f_trace = True
>  print_line()
>  print_line()
>
>      test()
>
>
> $ python3.9 ~/test/test.py
> 7
> 8
> 8
>
> With PEP 626 this is required to print:
> 7
> 9
> 10
>
>
> Cheers,
> Mark.
>
> >
> > Cheers,
> > Pablo
> >
> > On Thu, 29 Oct 2020, 09:47 Mark Shannon,  > <mailto:m...@hotpy.org>> wrote:
> >
> > Hi,
> >
> > That's great. Thanks Pablo.
> >
> > On 29/10/2020 1:32 am, Pablo Galindo Salgado wrote:
> >  > On behalf of the steering council, I am happy to announce that as
> >  > BDFL-Delegate I am
> >  > accepting PEP 626 -- Precise line numbers for debugging and other
> > tools.
> >  > I am confident this PEP will result in a better experience for
> >  > debuggers, profilers and tools
> >  > that rely on tracing functions. All the existing concerns
> regarding
> >  > out-of-process debuggers
> >  > and profilers have been addressed by Mark in the latest version
> > of the
> >  > PEP. The acceptance of
> >  > the PEP comes with the following requests:
> >  >
> >  > * The PEP must be updated to explicitly state that the
> API functions
> >  > described in the
> >  > "Out of process debuggers and profilers" must remain
> > self-contained
> >  > in any potential
> >  >  future modifications or enhancements.
> >  > * The PEP states that the "f_lineno" attribute of the code object
> > will
> >  > be updated to point to
> >  > the current line being executed even if tracing is off. Also,
> > there
> >  > were some folks concerned with
> >  > possible performance implications. Although in my view there
> 

[Python-Dev] Re: Accepting PEP 626

[Pablo Galindo Salgado]

> Except that we can't measure the performance of a specification.
> We can only measure the performance of entire implementations.
> I can make an implementation that conforms to PEP 626 that is slower
> than master, or I can make one that's faster :)
> It doesn't change the value of the PEP itself.

Oh, I see now where the confusion is. Regarding the benchmarks, unless
is catastrophically worse and there is no way to make it competitive, the
PEP acceptance is not conditional to the specific result of the benchmarks:
my request is just to *have* a benchmark on the implementation :)

On Thu, 29 Oct 2020 at 13:29, Mark Shannon  wrote:

> Hi Pablo,
>
> On 29/10/2020 11:08 am, Pablo Galindo Salgado wrote:
> >  > The new semantics may well result in some slowdowns. That's stated in
> > the PEP.I don't think I can reliably isolate the effects of the (very
> > slight)
> > change in the behavior of f_lineno.
> >
> > Ok, then let's make at least we measure the general slowdowns.
>
> Except that we can't measure the performance of a specification.
> We can only measure the performance of entire implementations.
>
> I can make an implementation that conforms to PEP 626 that is slower
> than master, or I can make one that's faster :)
> It doesn't change the value of the PEP itself.
>
> Let me give you a toy example.
>
> def f():
>  while 1:
>  body()
>
> 3.9 compiles this to:
>
> (The trailing, implicit return has been stripped for clarity)
>
>3 >>0 LOAD_GLOBAL  0 (body)
>2 CALL_FUNCTION0
>4 POP_TOP
>6 JUMP_ABSOLUTE0
>
> A naive implementation that conforms to PEP 626 would this compile to:
>
>2 >>0 NOP
>3   2 LOAD_GLOBAL  0 (body)
>4 CALL_FUNCTION0
>6 POP_TOP
>8 JUMP_ABSOLUTE0
>
> But a better implementation could produce this:
>
>2   0 NOP
>3 >>2 LOAD_GLOBAL  0 (body)
>4 CALL_FUNCTION0
>6 POP_TOP
>2   8 JUMP_ABSOLUTE2
>
> Which has the same bytecodes as 3.9 in the loop, and has the correct
> line numbers.
>
> Cheers,
> Mark.
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JCW4JZH6LNWDYXGPXWQ4MPWDZ27KROEK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

[Pablo Galindo Salgado]

Regarding having a Solaris buildbot: if someone provides a Solaris buildbot
then the deal is that that someone or some other party must look after that
buildbot and fix problems that appear in it in a timely manner. Broken
buildbots stop releases and I don't want to be in a situation in which I
need to halt a release because the Solaris buildbot is broken and there is
no-one to fix it in time.

In the past, we had to dealt with similar situations and not only is
stressful but normally ends in me or Victor having to login in a buildbot
for a platform that we are not experts on to try to fix it in time.

On Fri, 30 Oct 2020, 09:29 Ronald Oussoren via Python-Dev, <
python-dev@python.org> wrote:

>
> > On 29 Oct 2020, at 22:43, Victor Stinner  wrote:
> >
> > Hi,
> >
> > I propose to drop the Solaris support in Python to reduce the Python
> > maintenance burden:
> >
> >   https://bugs.python.org/issue42173
> >
> > I wrote a draft PR to show how much code could be removed (around 700
> > lines in 65 files):
> >
> >   https://github.com/python/cpython/pull/23002/files
>
> A significant fraction of that is in comments and documentation. A number
> of the changes in documentation would be good to go in regardless of the
> resolution of this proposal.
>
> >
> > In 2016, I asked if we still wanted to maintain the Solaris support in
> > Python, because Solaris buildbots were failing for longer than 6
> > months and nobody was able to fix them. It was requested to find a
> > core developer volunteer to fix Solaris issues and to set up a Solaris
> > buildbot.
> >
> >
> https://mail.python.org/archives/list/python-dev@python.org/thread/NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS/#NOT2RORSNX72ZLUHK2UUGBD4GTPNKBUS
> >
> > Four years later, nothing has happened. Moreover, in 2018, Oracle laid
> > off the Solaris development engineering staff. There are around 25
> > open Python bugs specific to Solaris.
>
> As another data point: There is someone on BPO that files issues about
> Solaris on BPO, including PRs. It might be worthwhile to ask that person if
> they can provide a buildbot (while making clear that this results in the
> assumption that they’d look after Solaris port).
>
> If Solaris would get dropped I’d prefer option 2
>
> Ronald
> —
>
> Twitter / micro.blog: @ronaldoussoren
> Blog: https://blog.ronaldoussoren.net/
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XFRQ2VEBK6NKUWMC6HXOJDLAIOQHORCP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A7Q7GW7ZI2EZDQSVYRCOI3S45MCS4WRZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Drop Solaris, OpenSolaris, Illumos and OpenIndiana support in Python

[Pablo Galindo Salgado]

>Two volunteer core developers and at least one buildbot would help a
> lot to ensure that Python is working on Solaris for real, and reduce
> the number of open Solaris issues. If it happens, I'm perfectly fine
> with keeping Solaris support.
> I also hope that more people will contribute to maintain the code, not
> only Ronald and Jesús. Many people wrote to me that Python is a key
> component of Illumos (the package manager is written in Python). So
> maybe Python on Illumos deserves some love?

If we decide to take the proposal seriously, I think it may be beneficial
to establish a deadline for having the buildbot green and the issues fixed,
so the promise of having "real" support at some point in the future does
actually manifest itself or it does not block following Victor's proposal
if it
actually does not happen.


On Fri, 30 Oct 2020 at 12:54, Victor Stinner  wrote:

> Hi Ronald,
>
> Le ven. 30 oct. 2020 à 12:59, Ronald Oussoren  a
> écrit :
> > I agree. That’s what I tried to write, its not just providing a buildbot
> but also making sure that it keeps working and stays green.
>
> This is really great!
>
> Jesús Cea Avión is also a volunteer to maintain the Solaris (see the bpo).
>
> Moreover, it seems like some people would like to provide servers to
> run a Solaris buildbot. Example:
> https://bugs.python.org/issue42173#msg379895
>
> Two volunteer core developers and at least one buildbot would help a
> lot to ensure that Python is working on Solaris for real, and reduce
> the number of open Solaris issues. If it happens, I'm perfectly fine
> with keeping Solaris support.
>
> I also hope that more people will contribute to maintain the code, not
> only Ronald and Jesús. Many people wrote to me that Python is a key
> component of Illumos (the package manager is written in Python). So
> maybe Python on Illumos deserves some love?
>
> There are many ways to contribute to the Solaris support of Python:
>
> * Comment Solaris issues (bugs.python.org, search for "Solaris" in the
> title)
> * Propose PRs to fix issues or implement Solaris specific features
> * Review Solaris PRs
> * Provide Solaris servers accessible to Python core developers (SSH access)
> * Donate to the CPython project:
>
>   * https://www.python.org/psf/donations/python-dev/
>   * https://github.com/sponsors/python
>
> * etc.
>
> See also the https://devguide.python.org/ if you would like to
> contribute to Python.
>
> By the way, thanks Jakub Kulík (CC-ed to this email) who fixed
> multiple Solaris issues in the last 2 years ;-)
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/732XMDFCTUNDP2GPFQWWUTLWBQFLLINR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.10.0a2 available for testing

[Pablo Galindo Salgado]

The engines of the secret release manager machine have finished producing a
new pre-release. Go get it here:

https://www.python.org/downloads/release/python-3100a2/


*Major new features of the 3.10 series, compared to 3.9*

Python 3.10 is still in development. This releasee, 3.10.0a2 is the second
of six planned alpha releases. Alpha releases are intended to make it
easier to test the current state of new features and bug fixes and to test
the release process. During the alpha phase, features may be added up until
the start of the beta phase (2021-05-03) and, if necessary, may be modified
or deleted up until the release candidate phase (2021-10-04). Please keep
in mind that this is a preview release and its use is not recommended for
production environments.

Many new features for Python 3.10 are still being planned and written.
Among the new major new features and changes so far:

PEP 623 -- Remove wstr from Unicode
PEP 604 -- Allow writing union types as X | Y
PEP 612 -- Parameter Specification Variables
PEP 626 -- Precise line numbers for debugging and other tools.
(Hey, fellow core developer, if a feature you find important is missing
from this list, let Pablo know.)
The next pre-release of Python 3.10 will be 3.10.0a3, currently scheduled
for 2020-12-07.

*And now for something completely different *

The cardinality (the number of elements) of infinite sets can be one of the
most surprising results of set theory. For example, there are the same
amount of even natural numbers than natural numbers (which can be even or
odd). There is also the same amount of rational numbers than natural
numbers. But on the other hand, there are more real numbers between 0 and 1
than natural numbers! All these sets have infinite cardinality but turn out
that some of these infinities are bigger than others. These infinite
cardinalities normally are represented using aleph numbers. Infinite sets
are strange beasts indeed.

Regards from cold London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EYPALIF2V25SI4LBH5THTY2IM4VWDGXJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.9.1 is now available, together with 3.10.0a3 and 3.8.7rc1

[Pablo Galindo Salgado]

It's starting to get very cold (at least on the Northern hemisphere) so we
have been carefully packaging a total of three new Python releases to keep
you warm these days!

Python 3.9.1 is the first maintenance release of Python 3.9, and also the
first version of Python to support macOS 11 Big Sur natively on Apple
Silicon. Go get it here:

https://www.python.org/downloads/release/python-391/

Maintenance releases for the 3.9 series will continue at regular bi-monthly
intervals, with **3.9.2** planned for Monday, 2021-02-08.

Python 3.10a3 is the third alpha release of Python 3.10. You can get it
here:

https://www.python.org/downloads/release/python-3100a3/

Python 3.10a3 is the release preview of the next maintenance release of
Python 3.8. You can get it here:

https://www.python.org/downloads/release/python-387rc1/

Assuming no critical problems are found prior to **2020-12-21** , the
currently scheduled release date for **3.8.7** , no code changes are
planned between this release candidate and the final release. That being
said, please keep in mind that this is a pre-release of 3.8.7 and as such
its main purpose is testing.

Your friendly release team,
Ned Deily, Steve Dower, Pablo Galindo, Łukasz Langa
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MCDBXKUBRCWL6M5W6A6MME7FGMBZMVR6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] CPython codebase plots

[Pablo Galindo Salgado]

Hi,

The year is almost over and plenty of internet services are generating the
“2020 in review” summary so I hope everyone is in the mood
for some extra summary plots. I have generated some “CPython lifetime in
review” plots that I hope you enjoy.

You can check them out here:

https://discuss.python.org/t/cpython-codebase-plots/6267

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/O6M3QBPUDCIBV3FF2VGRRHGCZJLJVC3T/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.10.0a4 is now available

[Pablo Galindo Salgado]

Happy new year to all of you. I hope you all have a great start of the
year! And how to best celebrate that we have left 2020 behind that with a
new Python alpha release? :)  Go get it here:

https://www.python.org/downloads/release/python-3100a4/


*Major new features of the 3.10 series, compared to 3.9*
Python 3.10 is still in development. This releasee, 3.10.0a4 is the second
of six planned alpha releases.
Alpha releases are intended to make it easier to test the current state of
new features and bug fixes and to test the release process.
During the alpha phase, features may be added up until the start of the
beta phase (2021-05-03) and, if necessary, may be modified or deleted up
until the release candidate phase (2021-10-04). Please keep in mind that
this is a preview release and its use is not recommended for production
environments.

Many new features for Python 3.10 are still being planned and written.
Among the new major
new features and changes so far:

* PEP 623 – Remove wstr from Unicode
* PEP 604 – Allow writing union types as X | Y
* PEP 612 – Parameter Specification Variables
* PEP 626 – Precise line numbers for debugging and other tools.
* bpo-38605: from __future__ import annotations (PEP 563) is now the
default.
* PEP 618 – Add Optional Length-Checking To zip.

(Hey, fellow core developer, if a feature you find important is missing
from this list, let me know.)
The next pre-release of Python 3.10 will be 3.10.0a5, currently scheduled
for 2021-02-01.


*And now for something completely different*
The Majumdar–Papapetrou spacetime
<https://en.wikipedia.org/wiki/Sudhansu_Datta_Majumdar#Majumdar%E2%80%93Papapetrou_solution>
is
one surprising solution of the coupled Einstein-Maxwell equations that
describe a cluster of static charged black holes with the gravitational and
the electrostatic forces cancelling each other out. Each one of these many
black holes of the multi-black holes system has a spherical topology and
follows the Reissner–Nordström metric
<https://en.wikipedia.org/wiki/Reissner%E2%80%93Nordstr%C3%B6m_metric>.
Unsurprisingly, the movement of a test particle in such spacetime is not
only a very chaotic system but also has some fractals
<https://arxiv.org/abs/gr-qc/9502014> hiding the complexity of its movement.

Regards from cold London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HPZ7Z62WGFZKSFT2Z2G4ZGWGONWFJ6B7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: 3.10 change (?) for __bool__

[Pablo Galindo Salgado]

This may be related to the changes in https://bugs.python.org/issue42246.
Could you open a new issue and add Mark Shannon to it if that turns to be
the case?

Pablo

On Mon, 11 Jan 2021 at 19:36, Chris Angelico  wrote:

> On Tue, Jan 12, 2021 at 6:00 AM Mats Wichmann  wrote:
> >
> >
> > On 1/8/21 4:31 PM, Mats Wichmann wrote:
> > >
> > >
> > > Someone reported a testsuite break on stuff I work on (scons) with
> > > 3.10a4, and it looks similar to this which appears in the changelog at
> > >
> > > https://docs.python.org/3.10/whatsnew/changelog.html#changelog
> > >
> > > bpo-23898: Fix inspect.classify_class_attrs() to support attributes
> with
> > > overloaded __eq__ and __bool__. Patch by Mike Bayer.
> > >
> > > Except when I go look at that BPO issue, it's old - closed 2015.  Is
> > > this appearing in the 3.10 changelog in error?  Sorry - confused !!!
> >
> > okay, that was silly, I didn't realize the changelog was cumulative over
> > many versions, so that entry was not for 3.10 at all (teach me to do
> > searching in browser window, where it just flies right past any section
> > headings so I miss it was for a different version :) ).
> >
> > > The test in question does indeed touch a class which overrides __bool_
> > > in order to raise an exception (to say "don't do that"), and in the
> test
> > > run the (expected) exception is not raised.
> >
> > So updated information: the test in question is checking if a class (A)
> > has an attribute using a truth test, where the attribute's value is an
> > instance of another class (B) and expecting that that will cause the
> > __bool__ method to be called. [aside: this test is done to validate that
> > a class which really doesn't want this kind of test indeed rejects it]
> > That apparently no longer happens, if it's wrapped in a try block ???
> > Distilled down to simple case:
> >
> > class A:
> >  pass
> >
> > class B:
> >  def __bool__(self):
> >  raise AttributeError("don't do that!")
> >
> > a = A()
> > b = B()
> > a.b = b
> > # expect this to cause b.__bool__ to be called
> > if a.b:
> >  print("Found it!")
> >
> > and it raises the exception.  But when protected:
> >
> > try:
> >  if a.b:
> >  pass
> > except AttributeError:
> >  print("Got expected exception")
> > else:
> >  print("Missed expected exception")
> >
> > it won't trigger. But if I add a "real" statement in the block following
> > the "if", then it's back to the pre-3.10 behavior of calling __bool__:
> >
> >   try:
> >  if a.b:
> >  dummy = True
> > except AttributeError:
> >  print("Got expected exception")
> > else:
> >  print("Missed expected exception")
> >
> >
> > Any thoughts on this?
>
> Oooh interesting. I tried on a build of 3.10 from October and:
> 1) The unguarded version bombed out with an exception
> 2) The "if... pass" version reported that it got the exception
> 3) The "if... dummy" version reported that it got the exception
>
> ie every one of them did indeed raise. But on a fresh build from the
> master branch, I got the same results you did. That means the change
> happened some time between commit 497126f7ea and commit ace008c531, an
> 800ish commit span.
>
> I'll start bisecting to try to track this down. It looks like "if a.b:
> pass" is getting partially optimized out; the disassembly shows a
> being loaded, its attribute b being looked up, and then it just jumps
> to the else - there's no POP_JUMP_IF_FALSE as there is when there's a
> bit of actual code in there.
>
> ChrisA
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/K2LD2L5RF2ZFUYEXQ3Z5U4TY5QBRFPCQ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7DMQXD3EF6CIAUIHFORMXEOUDZGUO2YW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Heap types (PyType_FromSpec) must fully implement the GC protocol

[Pablo Galindo Salgado]

One worry that I have in general with this move
is the usage of _PyType_GetModuleByDef to get the type object
from the module definition. This normally involves getting a TLS in every
instance creation,
which can impact notably performance for some perf-sensitive types or types
that are created a lot.

On Tue, 12 Jan 2021 at 18:21, Neil Schemenauer 
wrote:

> On 2021-01-12, Victor Stinner wrote:
> > It seems like a safer approach is to continue the work on
> > bpo-40077: "Convert static types to PyType_FromSpec()".
>
> I agree that trying to convert static types is a good idea.  Another
> possible bonus might be that we can gain some performance by
> integrating garbage collection with the Python object memory
> allocator.  Static types frustrate that effort.
>
> Could we have something easier to use than PyType_FromSpec(), for
> the purposes of coverting existing code?  I was thinking of
> something like:
>
> static PyTypeObject Foo_TypeStatic = {
> }
> static PyTypeObject *Foo_Type;
>
> PyInit_foo(void)
> {
> Foo_Type = PyType_FromStatic(&Foo_TypeStatic);
> }
>
>
> The PyType_FromStatic() would return a new heap type, created by
> copying the static type.  The static type could be marked as being
> unusable (e.g. with a type flag).
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RPG2TRQLONM2OCXKPVCIDKVLQOJR7EUU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HOCGUW3S6AXBSQ5BWX5KYPFVXEGWQJ6H/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

[Pablo Galindo Salgado]

Hi Julien,

Isn't this a similar problem that you have with regular malloc? When you
call malloc() with
some size, malloc actually will reserve more than that for
alignment purposes and for
bookkeeping and apart from some platform-specific APIs
like malloc_usable_size()
you cannot query that value.

Unless I am missing something here, there is also some problem of what you
propose: the Python
allocators deal with memory and they do not know what that memory is going
to be used so when you
say:

> This is a bummer as there then no safe way that I can think of to know ifan
allocated memory space is gc-tracked or gc-untracked.

My answer would be that that's because memory itself cannot be gc tracked,
only objects can and those belonging to different
categories. For example, notice that the tracemalloc module does not report
objects, it only reports memory blocks and you
cannot ask tracemalloc to give you a list of all the objects that were
allocated because it does not have the notion of what
an object is.

Adding the functionality that you suggest would couple the allocator with
other parts of the VM and it also won't give you the
the answer that you need to because what's underneath (pymalloc or libc
malloc or whatever else) may actually allocate even more
than what you asked.

Regards from cloudy London,
Pablo Galindo Salgado




On Thu, 14 Jan 2021 at 16:10, Julien Danjou  wrote:

> Hi there,
>
> I've spent quite some time on memory profiling for Python now. I'm
> struggling to get more information from the allocated memory right now
> for what looks like a sad reason. :(
>
> Supposedly PyObject_Malloc() returns some memory space to store a
> PyObject. If that was true all the time, that would allow anyone to
> introspect the allocated memory and understand why it's being used.
>
> Unfortunately, this is not the case. Objects whose types are tracked by
> the GC go through _PyObject_GC_Alloc() which changes the underlying
> memory structure to be (PyGC_HEAD + PyObject).
>
> This is a bummer as there then no safe way that I can think of to know
> if an allocated memory space is gc-tracked or gc-untracked. It makes it
> therefore impossible to introspect the memory allocated by
> PyObject_Malloc().
>
> There are multiple ways to solve this, but I don't see any way right now
> of doing this without slightly changing CPython.
>
> Has anyone any idea on how to workaround this or the kind of change that
> could be acceptable to mitigate the issue?
>
> Thanks!
>
> Cheers,
> --
> Julien Danjou
> # Free Software hacker
> # https://julien.danjou.info
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QRX6U5XBXMHMT6YKIXERS3UT64ALYV27/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ISXWWDMPY2N4UAAEHC4VYLCRW2XJLWDM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

[Pablo Galindo Salgado]

> Exactly, which is a bit a bummer. Considering Python provides 3
> different memory allocator, it'd be great if there was some ability to
> be sure that PyObject_Malloc pointer are actually PyObject, not
> Py_GC_HEAD.

The allocators are specialized based on the allocation strategy
and efficiency, not based on what are you going to use the memory
for. If you want to allocate a buffer using the object allocation
strategy because  then nobody is preventing you
to use PyObject_Malloc(). Even if we sanitize the whole stdlib to
be conforming to "only objects are allocated using PyObejct_Malloc()",
3rd party extension modules and other bests can do whatever, so you
can still crash if you decide to interpreter the output as an object.


On Fri, 15 Jan 2021 at 08:43, Julien Danjou  wrote:

> On Thu, Jan 14 2021, Tim Peters wrote:
>
> > I'm not clear on exactly what it is you're after, but CPython faces
> > the same question all the time: _given_ a pointer to an object, is
> > there, or is there not, a GC header prepended?  That's answered by
> > this C API function:
> >
> > """
> > int PyObject_IS_GC(PyObject *obj)
> >
> > Returns non-zero if the object implements the garbage collector
> > protocol, otherwise returns 0.
> >
> > The object cannot be tracked by the garbage collector if this function
> > returns 0.
> > """
> >
> > FYI, the implementation usually resolves it by looking at whether
> > obj's type object has the Py_TPFLAGS_HAVE_GC flag set.
>
> Right, but that only works if you have the PyObject address.
> If you all got is a pointer returned by PyObject_Malloc(), you don't
> know if the PyObject is at this pointer address, or after the PyGC_HEAD
> header that is prepended. :(
>
> --
> Julien Danjou
> /* Free Software hacker
>https://julien.danjou.info */
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/Z6GJUNKOFE2IN6EKQKDBBMX4SUZ36ITU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KIBIVCKSXVE77YE3PWPPK7QPBAFQFMOU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PyGC and PyObject_Malloc introspection

[Pablo Galindo Salgado]

> Then the correct endpoint would more likely to be PyObject_New(), but
> there's no way to intercept such calls for statistical analysis
> currently. And as you wrote, if some code decide to use PyMalloc()
> directly, then that memory won't be tracked.

The one that CPython uses in debug mode to track all references for
sys.getobjects() is PyObject_Init(), which is heavily inlined for
performance
reasons (the same as many of the other calls in the allocation chain), so
unfortunately
is not possible to intercept using LD_PRELOAD or even GOT patching.

> t sounds like the provided C API is a bit too low level for this,
> preventing any kind of statistical analysis of the allocation patterns.
> :(

Yes, allowing interception or customization will prevent inlining or other
optimizations
and therefore will involve a considerable performance hit. As an experiment
I forced
PyObject_Init and _PyObject_Init to not be inlined and that made a 7-13%
speed
impact overall in the performance test suite.

If you want to track all objects creation, your best bet IMHO is a debug
build and to use sys.getobjects().

Regards from sunny London,
Pablo Galindo Salgado


On Fri, 15 Jan 2021 at 12:17, Julien Danjou  wrote:

> On Fri, Jan 15 2021, Pablo Galindo Salgado wrote:
>
> >> Exactly, which is a bit a bummer. Considering Python provides 3
> >> different memory allocator, it'd be great if there was some ability to
> >> be sure that PyObject_Malloc pointer are actually PyObject, not
> >> Py_GC_HEAD.
> >
> > The allocators are specialized based on the allocation strategy
> > and efficiency, not based on what are you going to use the memory
> > for. If you want to allocate a buffer using the object allocation
> > strategy because  then nobody is preventing you
> > to use PyObject_Malloc(). Even if we sanitize the whole stdlib to
> > be conforming to "only objects are allocated using PyObejct_Malloc()",
> > 3rd party extension modules and other bests can do whatever, so you
> > can still crash if you decide to interpreter the output as an object.
>
> Agreed.
>
> Then the correct endpoint would more likely to be PyObject_New(), but
> there's no way to intercept such calls for statistical analysis
> currently. And as you wrote, if some code decide to use PyMalloc()
> directly, then that memory won't be tracked.
>
> It sounds like the provided C API is a bit too low level for this,
> preventing any kind of statistical analysis of the allocation patterns.
> :(
>
> --
> Julien Danjou
> # Free Software hacker
> # https://julien.danjou.info
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A6FCIFGSWL6POYZNIEYPZVCQMJDOES7P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Pablo Galindo Salgado]

Hi Mark,

Thanks for gathering this proposal! Looks very interesting. I have some 
preliminary questions: how is this going to affect
the "py-bt" command of the gdb helpers 
(https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897)
and other similar tools that produce a unified Python-C backtrace? There are 
several debuggers that
rely on the fact that they can merge the C stack and the Python stack by 
substituting every call to "_PyEval_EvalFrameDefault"
and friends for the respective Python frame obtained separately or by 
inspecting the frame object in the stack (like gdb does).

Is this change going to affect these tools or they will continue working as 
before? In case this change will affect this tools,
is there any workaround to produce the unified C/Python call stack given the 
Python stack and the C stack?

Kind regards,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Pablo Galindo Salgado]

>It depends on what you mean by "similar tools".

Any 3rd party tool or debugger that is printing merged stacks. (There are
many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)

> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it is at
> the moment :)

I agree with that, but we need to have in mind that this will break
these tools and therefore we need to be aware on how difficult will
be to overcome the new situation. Is not the same that a small or
medium change is required than the case where is virtually impossible
to overcome. If we were designing the interpreter from scratch that would
not matter, but with an existing ecosystem of tools, it does. Breaking tools
in an established ecosystem has always been a major complaint of users when
we do major releases.

Notice as well that the gdb helpers *are* an out-of-process tool.

Of course, I don't imply that this should be a show stopper or similar, but
it should be taken into consideration along all the other pros and cons.

Regards from cloudy London,
Pablo Galindo Salgado

On Wed, 20 Jan 2021 at 15:12, Mark Shannon  wrote:

> Hi Pablo,
>
> On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> > Hi Mark,
> >
> > Thanks for gathering this proposal! Looks very interesting. I have some
> preliminary questions: how is this going to affect
> > the "py-bt" command of the gdb helpers (
> https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897
> )
> > and other similar tools that produce a unified Python-C backtrace? There
> are several debuggers that
> > rely on the fact that they can merge the C stack and the Python stack by
> substituting every call to "_PyEval_EvalFrameDefault"
> > and friends for the respective Python frame obtained separately or by
> inspecting the frame object in the stack (like gdb does).
>
> I don't see any problem fixing up the gdb helpers. The code will need to
> be aware that C frames will be one-to-many to Python frames, but there's
> nothing tricky there.
>
> >
> > Is this change going to affect these tools or they will continue working
> as before? In case this change will affect this tools,
> > is there any workaround to produce the unified C/Python call stack given
> the Python stack and the C stack?
>
> It depends on what you mean by "similar tools".
> For in-process tools, then the API will continue to work.
> For out-of-process debuggers, then the author's are on their own.
> But that's always been the case.
> The source code is public, and it won't be any more opaque than it is at
> the moment :)
>
> Cheers,
> Mark.
>
> >
> > Kind regards,
> > Pablo Galindo Salgado
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/27DON4D7Y3WMFMOT7OV6D4LD6QUXXQRB/
> > Code of Conduct: http://python.org/psf/codeofconduct/
> >
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZNCDQCYPATMZRU7SHQM4OTMVXUI6VX5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 651 -- Robust Overflow Handling

[Pablo Galindo Salgado]

> Do you have any specific examples?

>From my previous message: gdb helpers, lldb helpers, TotalView debugger,
py-spy and I know of at least 3 closed source tools.

>I feel the responsibility for this belongs with the tool authors.
> If they aren't using the C-API, or requesting an alternative interface
> should the API be insufficient, then they can't really complain.

The responsibility to fix it is on the authors, yes, but is not that back
and white. In the worst case, what users will see is that tools that worked
now don't and in the worst-worst case users will see that those tools will
not work again because is virtually impossible with the new system. And
that, as I said is not a show stopper, but it matters.

In any case, if we are able to fix the gdb helpers in the stdlib, then I am
not afraid because it means that everyone could fix their own thing.

Cheers,
Pablo Galindo Salgado


On Wed, 20 Jan 2021 at 16:06, Mark Shannon  wrote:

> Hi Pablo,
>
> On 20/01/2021 3:18 pm, Pablo Galindo Salgado wrote:
> >  >It depends on what you mean by "similar tools".
> >
> > Any 3rd party tool or debugger that is printing merged stacks. (There
> > are many: gdb helpers, lldb helpers, TotalView debugger, py-spy, ...)
> >
> >> For in-process tools, then the API will continue to work.
> >> For out-of-process debuggers, then the author's are on their own.
> >> But that's always been the case.
> >> The source code is public, and it won't be any more opaque than it is at
> >> the moment :)
> >
> > I agree with that, but we need to have in mind that this will break
> > these tools and therefore we need to be aware on how difficult will
> > be to overcome the new situation. Is not the same that a small or
> > medium change is required than the case where is virtually impossible
> > to overcome. If we were designing the interpreter from scratch that would
> > not matter, but with an existing ecosystem of tools, it does. Breaking
> tools
> > in an established ecosystem has always been a major complaint of users
> > when we do major releases.
>
> Do you have any specific examples?
>
> I feel the responsibility for this belongs with the tool authors.
> If they aren't using the C-API, or requesting an alternative interface
> should the API be insufficient, then they can't really complain.
>
> >
> > Notice as well that the gdb helpers *are* an out-of-process tool.
>
> True, but they are in the standard library, so they are our
> responsibility to fix.
>
> Cheers,
> Mark.
>
> >
> > Of course, I don't imply that this should be a show stopper or similar,
> > but it should be taken into consideration along all the other pros and
> cons.
> >
> > Regards from cloudy London,
> > Pablo Galindo Salgado
> >
> > On Wed, 20 Jan 2021 at 15:12, Mark Shannon  > <mailto:m...@hotpy.org>> wrote:
> >
> > Hi Pablo,
> >
> > On 19/01/2021 6:46 pm, Pablo Galindo Salgado wrote:
> >  > Hi Mark,
> >  >
> >  > Thanks for gathering this proposal! Looks very interesting. I
> > have some preliminary questions: how is this going to affect
> >  > the "py-bt" command of the gdb helpers
> > (
> https://github.com/python/cpython/blob/master/Tools/gdb/libpython.py#L1876-L1897
> )
> >  > and other similar tools that produce a unified Python-C
> > backtrace? There are several debuggers that
> >  > rely on the fact that they can merge the C stack and the Python
> > stack by substituting every call to "_PyEval_EvalFrameDefault"
> >  > and friends for the respective Python frame obtained separately
> > or by inspecting the frame object in the stack (like gdb does).
> >
> > I don't see any problem fixing up the gdb helpers. The code will
> > need to
> > be aware that C frames will be one-to-many to Python frames, but
> > there's
> > nothing tricky there.
> >
> >  >
> >  > Is this change going to affect these tools or they will continue
> > working as before? In case this change will affect this tools,
> >  > is there any workaround to produce the unified C/Python call
> > stack given the Python stack and the C stack?
> >
> > It depends on what you mean by "similar tools".
> > For in-process tools, then the API will continue to work.
> > For out-of-process debuggers, then the author's are on their own.
> > But that's always been the case.
> > 

[Python-Dev] [ Release ] Python 3.10a5 and release blockers

[Pablo Galindo Salgado]

Hi everyone,

I am prepared to start the release process for Python 3.10 a5 but there are
several
issues marked as release blockers that affect 3.10:

* https://bugs.python.org/issue38302
* https://bugs.python.org/issue42634
* https://bugs.python.org/issue41490
* https://bugs.python.org/issue42967
* https://bugs.python.org/issue42899

Although release blockers mainly apply to important releases, there are two
are security issues and some
of the rest involve changes in bytecode or the tracing machinery, so I
would prefer if these issues are addressed
before making a new release as many maintainers are waiting for the next
alpha to test again the bugs that they
reported.

Please, if you are involved in any of these issues try to see if is
possible to address them or if you think
is ok to release the alpha without a fix, please, drop me an email stating
so.

Thanks,

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZVOFYB5K6UZZLQXQCCCWAJNLTMBF5Z63/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python 3.10.0a5 is now available

[Pablo Galindo Salgado]

Well, this one took a bit more time due to some surprise last time
reference leaks and release blockers to fix, but now Python 3.10.0a5 it’s
here. Will this be the first release announcement of the 3.10 series
without copy-paste typos? Go get it here:

https://www.python.org/downloads/release/python-3100a5/

*Major new features of the 3.10 series, compared to 3.9*

Python 3.10 is still in development. This release, 3.10.0a5 is the fifth of
seven planned alpha releases. Alpha releases are intended to make it easier
to test the current state of new features and bug fixes and to test the
release process. During the alpha phase, features may be added up until the
start of the beta phase (2021-05-03) and, if necessary, may be modified or
deleted up until the release candidate phase (2021-10-04). Please keep in
mind that this is a preview release and its use is not recommended for
production environments.

Many new features for Python 3.10 are still being planned and written.
Among the new major new features and changes so far:

   - PEP 623 <https://www.python.org/dev/peps/pep-0623/> – Remove wstr from
   Unicode
   - PEP 604 <https://www.python.org/dev/peps/pep-0604/> – Allow writing
   union types as X | Y
   - PEP 612 <https://www.python.org/dev/peps/pep-0612/> – Parameter
   Specification Variables
   - PEP 626 <https://www.python.org/dev/peps/pep-0626/> – Precise line
   numbers for debugging and other tools.
   - bpo-38605 <https://bugs.python.org/issue38605>: from __future__ import
   annotations (PEP 563 <https://www.python.org/dev/peps/pep-0563/>) is now
   the default.
   - PEP 618  <https://www.python.org/dev/peps/pep-0618/>– Add Optional
   Length-Checking To zip.
   - bpo-12782 <https://bugs.python.org/issue12782>: Parenthesized context
   managers are now officially allowed.
   - (Hey, fellow core developer, if a feature you find important is
   missing from this list, let Pablo know .)

The next pre-release of Python 3.10 will be 3.10.0a6, currently scheduled
for 2021-03-01.
And now for something completely different

The Chandrasekhar limit is the maximum mass of a stable white dwarf star.
White dwarfs resist gravitational collapse primarily through electron
degeneracy pressure, compared to main sequence stars, which resist collapse
through thermal pressure. The Chandrasekhar limit is the mass above which
electron degeneracy pressure in the star’s core is insufficient to balance
the star’s own gravitational self-attraction. Consequently, a white dwarf
with a mass greater than the limit is subject to further gravitational
collapse, evolving into a different type of stellar remnant, such as a
neutron star or black hole. Those with masses up to the limit remain stable
as white dwarfs. The currently accepted value of the Chandrasekhar limit is
about 1.4 M☉ (2.765×1030 kg). So we can be safe knowing that our sun is not
going to become a black hole!

Regards from cloudy London,

Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UV72JXDKJYTETYQUCU36RXQCGEXGSIP2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.10.0a6 is available, now with 100% more pattern matching

[Pablo Galindo Salgado]

Remember us? It's your friendly CPython release team and we have something
we think you may like: The new alpha release of Python 3.10 is here, now
with 100% more pattern matching. If I were you, I would download it and
start playing with it. Extra points if you report us any bugs you find
along the way! Are you confused about all this pattern matching business?
Fear not, this release also includes some fantastic documentation and some
shiny new "What's new" entries.

Check them here and tell us how we can improve it:

What's new: https://docs.python.org/3.10/whatsnew/3.10.html
Tutorial:
https://docs.python.org/3.10/tutorial/controlflow.html#match-statements

Go get the new alpha here:

https://www.python.org/downloads/release/python-3100a6/


*Note: The alpha was officially released yesterday, but my email client
failed to deliver this email to the mailing lists so I am reposting it.*
**This is an early developer preview of Python 3.10**

# Major new features of the 3.10 series, compared to 3.9

Python 3.10 is still in development.  This release, 3.10.0a6 is the sixth
of seven planned alpha releases.
Alpha releases are intended to make it easier to test the current state of
new features and bug fixes and to test the release process.
During the alpha phase, features may be added up until the start of the
beta phase (2021-05-03) and, if necessary, may be modified or deleted up
until the release candidate phase (2021-10-04).  Please keep in mind that
this is a preview release and its use is **not** recommended for production
environments.

Many new features for Python 3.10 are still being planned and written.
Among the new major
new features and changes so far:

* [PEP 623](https://www.python.org/dev/peps/pep-0623/) -- Remove wstr from
Unicode
* [PEP 604](https://www.python.org/dev/peps/pep-0604/) -- Allow writing
union types as X | Y
* [PEP 612](https://www.python.org/dev/peps/pep-0612/) -- Parameter
Specification Variables
* [PEP 626](https://www.python.org/dev/peps/pep-0626/) -- Precise line
numbers for debugging and other tools.
* [bpo-38605](https://bugs.python.org/issue38605): `from __future__ import
annotations` ([PEP 563](https://www.python.org/dev/peps/pep-0563/)) is now
the default.
* [PEP 618 ](https://www.python.org/dev/peps/pep-0618/) -- Add Optional
Length-Checking To zip.
* [bpo-12782](https://bugs.python.org/issue12782): Parenthesized context
managers are now officially allowed.
* [PEP 632 ](https://www.python.org/dev/peps/pep-0632/) -- Deprecate
distutils module.
* [PEP 613 ](https://www.python.org/dev/peps/pep-0613/) -- Explicit Type
Aliases
* [PEP 634 ](https://www.python.org/dev/peps/pep-0634/) -- Structural
Pattern Matching: Specification
* [PEP 635 ](https://www.python.org/dev/peps/pep-0635/) -- Structural
Pattern Matching: Motivation and Rationale
* [PEP 636 ](https://www.python.org/dev/peps/pep-0636/) -- Structural
Pattern Matching: Tutorial

* (Hey, **fellow core developer,** if a feature you find important
is missing from this list, [let Pablo know](mailto:pablog...@python.org
).)

The next pre-release of Python 3.10 will be 3.10.0a7 ( last alpha release),
currently scheduled for Monday, 2021-04-05.

# More resources

* [Online Documentation](https://docs.python.org/3.10/)
* [PEP 619](https://www.python.org/dev/peps/pep-0619/), 3.10 Release
Schedule
* Report bugs at [https://bugs.python.org](https://bugs.python.org).
* [Help fund Python and its community](/psf/donations/).

# And now for something completely different
Schwarzschild wormholes, also known as Einstein–Rosen bridges (named after
Albert Einstein and Nathan Rosen), are connections between areas of space
that can be modelled as vacuum solutions to the Einstein field equations,
and that are now understood to be intrinsic parts of the maximally extended
version of the Schwarzschild metric describing an eternal black hole with
no charge and no rotation. Here, "maximally extended" refers to the idea
that the spacetime should not have any "edges": it should be possible to
continue this path arbitrarily far into the particle's future or past for
any possible trajectory of a free-falling particle (following a geodesic in
the spacetime).

Although Schwarzschild wormholes are not traversable in both directions,
their existence inspired Kip Thorne to imagine traversable wormholes
created by holding the "throat" of a Schwarzschild wormhole open with
exotic matter (material that has negative mass/energy).

Regards from rainy London,

Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VGZDQ5TFMSWEQZS6HNDLDJQ7GAASGUUB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Python 3.10.0a6 is available, now with 100% more pattern matching

[Pablo Galindo Salgado]

Remember us? It's your friendly CPython release team and we have something
we think you may like: The new alpha release of Python 3.10 is here, now
with 100% more pattern matching. If I were you, I would download it and
start playing with it. Extra points if you report us any bugs you find
along the way! Are you confused about all this pattern matching business?
Fear not, this release also includes some fantastic documentation and some
shiny new "What's new" entries.

Check them here and tell us how we can improve it:

What's new: https://docs.python.org/3.10/whatsnew/3.10.html
Tutorial:
https://docs.python.org/3.10/tutorial/controlflow.html#match-statements

Go get the new alpha here:

https://www.python.org/downloads/release/python-3100a6/

**This is an early developer preview of Python 3.10**

# Major new features of the 3.10 series, compared to 3.9

Python 3.10 is still in development.  This release, 3.10.0a6 is the sixth
of seven planned alpha releases.
Alpha releases are intended to make it easier to test the current state of
new features and bug fixes and to test the release process.
During the alpha phase, features may be added up until the start of the
beta phase (2021-05-03) and, if necessary, may be modified or deleted up
until the release candidate phase (2021-10-04).  Please keep in mind that
this is a preview release and its use is **not** recommended for production
environments.

Many new features for Python 3.10 are still being planned and written.
Among the new major
new features and changes so far:

* [PEP 623](https://www.python.org/dev/peps/pep-0623/) -- Remove wstr from
Unicode
* [PEP 604](https://www.python.org/dev/peps/pep-0604/) -- Allow writing
union types as X | Y
* [PEP 612](https://www.python.org/dev/peps/pep-0612/) -- Parameter
Specification Variables
* [PEP 626](https://www.python.org/dev/peps/pep-0626/) -- Precise line
numbers for debugging and other tools.
* [bpo-38605](https://bugs.python.org/issue38605): `from __future__ import
annotations` ([PEP 563](https://www.python.org/dev/peps/pep-0563/)) is now
the default.
* [PEP 618 ](https://www.python.org/dev/peps/pep-0618/) -- Add Optional
Length-Checking To zip.
* [bpo-12782](https://bugs.python.org/issue12782): Parenthesized context
managers are now officially allowed.
* [PEP 632 ](https://www.python.org/dev/peps/pep-0632/) -- Deprecate
distutils module.
* [PEP 613 ](https://www.python.org/dev/peps/pep-0613/) -- Explicit Type
Aliases
* [PEP 634 ](https://www.python.org/dev/peps/pep-0634/) -- Structural
Pattern Matching: Specification
* [PEP 635 ](https://www.python.org/dev/peps/pep-0635/) -- Structural
Pattern Matching: Motivation and Rationale
* [PEP 636 ](https://www.python.org/dev/peps/pep-0636/) -- Structural
Pattern Matching: Tutorial


* (Hey, **fellow core developer,** if a feature you find important
is missing from this list, [let Pablo know](mailto:pablog...@python.org
).)

The next pre-release of Python 3.10 will be 3.10.0a7 ( last alpha release),
currently scheduled for Monday, 2021-04-05.

# More resources

* [Online Documentation](https://docs.python.org/3.10/)
* [PEP 619](https://www.python.org/dev/peps/pep-0619/), 3.10 Release
Schedule
* Report bugs at [https://bugs.python.org](https://bugs.python.org).
* [Help fund Python and its community](/psf/donations/).

# And now for something completely different
Schwarzschild wormholes, also known as Einstein–Rosen bridges (named after
Albert Einstein and Nathan Rosen), are connections between areas of space
that can be modelled as vacuum solutions to the Einstein field equations,
and that are now understood to be intrinsic parts of the maximally extended
version of the Schwarzschild metric describing an eternal black hole with
no charge and no rotation. Here, "maximally extended" refers to the idea
that the spacetime should not have any "edges": it should be possible to
continue this path arbitrarily far into the particle's future or past for
any possible trajectory of a free-falling particle (following a geodesic in
the spacetime).

Although Schwarzschild wormholes are not traversable in both directions,
their existence inspired Kip Thorne to imagine traversable wormholes
created by holding the "throat" of a Schwarzschild wormhole open with
exotic matter (material that has negative mass/energy).

Regards from rainy London,

Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LZV62UZIHZGM3Y33ZQC2H3NPUZBOOGV2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Steering Council update for February

[Pablo Galindo Salgado]

Hi everyone,

The Steering Council just published the community update for February:

https://github.com/python/steering-council/blob/main/updates/2021-02-steering-council-update.md

As a reminder, we'll are trying to keep this up monthly. As you can see we
are mainly focusing on clearing the PEP backlog and to address ongoing
efforts and time-sensitive issues. Here you can find the text for the
update:

February 01

   -

   After more deliberation and consideration of all available information,
   the Steering Council accepted the Pattern Matching PEPs. The Steering
   Council agreed that high-quality documentation is crucial in this feature
   and therefore its presence should be required before the release. A
   response in the name of the Steering Council will be sent to python-dev
   addressing the decision and the Steering Council view of the process.
   -

   The Steering Council discussed Debian's Python distro issues. The
   Steering Council will collect and communicate where the concerns are to the
   Debian maintainers. Thomas started a draft of the communication that the
   Steering Council will review, complete and sign-off.

<https://github.com/python/steering-council/blob/main/updates/2021-02-steering-council-update.md#february-08>February
08

   -

   The Steering Council started the discussion of PEP 651. It was decided
   that the group will leave this open for next week to better deliberate and
   consider more information.
   -

   The Steering Council discussed the ongoing work on porting types in the
   standard library to heap-types. It was decided that through Pablo, the
   Steering Council will ask the core developers driving those changes to
   create an informational PEP and not to make any more changes in this area
   after beta 1, as per our general policy.
   -

   The Steering Council discussed adding a TOML module to the standard
   library. It was decided that maintainers should write a PEP. On the topic
   of the standard library, it was agreed that the Steering Council should
   present at the PyCon US 2021 Language Summit on the future of the standard
   library and Brett is going to gather data on what PRs are tied to which
   module, which modules are imported and used the most, commit churn to help
   start the planning so we can make data-driven discussions and decisions.
   -

   The acceptance notice for PEP 634 will be sent from the Steering Council
   email, and Thomas will send rejection notices to other competing PEPs that
   are therefore rejected.

<https://github.com/python/steering-council/blob/main/updates/2021-02-steering-council-update.md#february-15>February
15

[ The SC could not meet this week due to holidays in the US and Canada, and
the impossibility to find a common replacement date in the week ]
<https://github.com/python/steering-council/blob/main/updates/2021-02-steering-council-update.md#february-22>February
22

   -

   The Steering Council discussed PEP 651 request and has decided to reject
   it. An email with the rationale will be sent to the author and python-dev.
   -

   Ezio joined the Steering Council meeting today to give an update on the
   GitHub Migration. GitHub was dormant during the holiday months. Ezio and GH
   had a productive meeting a couple of weeks ago. He is working on the tool
   that will convert bpo issues into a format that the GH tool will accept. He
   will also work on a community update once the test repo is ready. Another
   sync up has been scheduled between Ezio and the SC on March 29th.
   -

   The Steering Council discussed renaming the master branch to main and
   the consensus was that we should do that. The group is going to discuss the
   timeline and how to approach this with other repositories under the Python
   organization in some future meeting.
   -

   The Steering Council discussed the Debian situation. The group discussed
   ways to approach the situation and how to coordinate with Debian
   maintainers so it can benefit the Python community at large. It was decided
   that the steering council will email the Debian maintainers to try to reach
   some common ground and discuss how to proceed.

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XCUXKL5LMQHO7GDQUABS4KE6JSMRVUQN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [Release management] schedule for renaming the default branch

[Pablo Galindo Salgado]

Hi,

I am writing on behalf of the Python Release Management team. The Steering
Council has requested the RM team to schedule
and perform the necessary changes to rename the default branch from master
to main.

# The changes

What follows is the technical description of the changes and the timeline.
In order to keep this thread focused on this particular
aspect, if you wish to discuss anything related to the change itself,
please, open a new email thread or reuse an existing one.

   - The change will be executed ***when beta 1 is released***, as the beta
   1 release requires some branching engineering already
   to create the 3.10 branch and point 3.11 to the new one as well as
   changing CI, buildbots...etc **
*This is scheduled for Monday, 2021-05-03**.*
   - The CI will be adapted to work with the new "main" branch.
   - The buildbots will be adapted to work with the new "main" branch.
   - Branch protection rules will be adapted.
   - The different bots will be adapted by the respective bot maintainer
   teams.
   - All the URLs that point to master in the README and other places will
   be adapted to point to main instead (notice this is
   not strictly necessary because GitHub redirects automatically).

Notice that the renaming will automatically:


   - Re-target any open pull requests
   - Update any draft releases based on the branch
   - Move any branch protection rules that explicitly reference the old name
   - Show a notice to repository contributors, maintainers, and admins on
   the repository homepage with instructions to update local copies of the
   repository
   - Show a notice to contributors who git push to the old branch
   - Redirect web requests for the old branch name to the new branch name
   - Return a "Moved Permanently" response in API requests for the old
   branch name

Check this <https://github.com/github/renaming> for more information.

# What you need to do?

You just need to update your local clone after the branch name changes.
>From the local clone of the repository on a computer,
run the following commands to update the name of the default branch.

$ git branch -m master main
$ git fetch origin
$ git branch -u origin/main main

Apart from that, you should update any local script or command that uses
the name "master" to use the name "main".

Regards from windy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [Release management] schedule for renaming the default branch

[Pablo Galindo Salgado]

I am answering this as a member of the release management team, not as an
official response from the SC.

> and understand who the identified impacted parties are

All developers that have clones of the repository and any party maintaining
any script that interacts with the default CPython branch.

> and what the plan is to notify them

These messages on python-dev and python-comitters + messages in the
official release announcement and notes.

> and help them update within this timescale

For the repositories, execute the commands listed in the previous message.
For scrips and other references to the main branch,
they need to do the change themselves but in most cases is a simple rename
master->main.

> Has this analysis been published anywhere

No. It has been discussed by the Steering Council as you can see in the
February update:
https://github.com/python/steering-council/blob/main/updates/2021-02-steering-council-update.md



On Wed, 10 Mar 2021 at 14:21, Stestagg  wrote:

> Hi
>
> I would be great to read the impact analysis for this change, and
> understand who the identified impacted parties are, and what the plan is to
> notify them and help them update within this timescale.
>
> Has this analysis been published anywhere? I know there are lots of places
> where discussions/documentation happens
>
> Thanks
>
> Steve
>
>
>
> On Wed, Mar 10, 2021 at 2:10 PM Pablo Galindo Salgado 
> wrote:
>
>> Hi,
>>
>> I am writing on behalf of the Python Release Management team. The
>> Steering Council has requested the RM team to schedule
>> and perform the necessary changes to rename the default branch from
>> master to main.
>>
>> # The changes
>>
>> What follows is the technical description of the changes and the
>> timeline. In order to keep this thread focused on this particular
>> aspect, if you wish to discuss anything related to the change itself,
>> please, open a new email thread or reuse an existing one.
>>
>>- The change will be executed ***when beta 1 is released***, as the
>>beta 1 release requires some branching engineering already
>>to create the 3.10 branch and point 3.11 to the new one as well as
>>changing CI, buildbots...etc **
>> *This is scheduled for Monday, 2021-05-03**.*
>>- The CI will be adapted to work with the new "main" branch.
>>- The buildbots will be adapted to work with the new "main" branch.
>>- Branch protection rules will be adapted.
>>- The different bots will be adapted by the respective bot maintainer
>>teams.
>>- All the URLs that point to master in the README and other places
>>will be adapted to point to main instead (notice this is
>>not strictly necessary because GitHub redirects automatically).
>>
>> Notice that the renaming will automatically:
>>
>>
>>- Re-target any open pull requests
>>- Update any draft releases based on the branch
>>- Move any branch protection rules that explicitly reference the old
>>name
>>- Show a notice to repository contributors, maintainers, and admins
>>on the repository homepage with instructions to update local copies of the
>>repository
>>- Show a notice to contributors who git push to the old branch
>>- Redirect web requests for the old branch name to the new branch name
>>- Return a "Moved Permanently" response in API requests for the old
>>branch name
>>
>> Check this <https://github.com/github/renaming> for more information.
>>
>> # What you need to do?
>>
>> You just need to update your local clone after the branch name changes.
>> From the local clone of the repository on a computer,
>> run the following commands to update the name of the default branch.
>>
>> $ git branch -m master main
>> $ git fetch origin
>> $ git branch -u origin/main main
>>
>> Apart from that, you should update any local script or command that uses
>> the name "master" to use the name "main".
>>
>> Regards from windy London,
>> Pablo Galindo Salgado
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing lis

[Python-Dev] Re: [Release management] schedule for renaming the default branch

[Pablo Galindo Salgado]

> I have above 200 feature branches in my local > repository. Will renaming
> the master branch cause any problems?

It should not be any problem at all. If you have some open PRs from some of
those branches, they will be automatically retargeted to the new branch
name in GitHub automatically.

On Thu, 11 Mar 2021, 07:37 Serhiy Storchaka,  wrote:

> 10.03.21 16:06, Pablo Galindo Salgado пише:
> > # What you need to do?
> >
> > You just need to update your local clone after the branch name changes.
> > From the local clone of the repository on a computer,
> > run the following commands to update the name of the default branch.
> >
> > $ git branch -m master main
> > $ git fetch origin
> > $ git branch -u origin/main main
> >
> > Apart from that, you should update any local script or command that uses
> > the name "master" to use the name "main".
>
> I have above 200 feature branches in my local repository. Will renaming
> the master branch cause any problems?
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/G2LLBPJMIBD2DOA7AVTYYA77PGLGYLUE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6IVJZTCQ47F6MDBIF3UNY2DACHRBXHJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Buildbot UI auto-refreshes

[Pablo Galindo Salgado]

Hi,

I think this may be https://bugs.python.org/issue41701.

If that's the case this is not on the buildbot worker itself but the proxy.

-

Pablo


On Sun, 21 Mar 2021, 17:12 Antoine Pitrou,  wrote:

>
> Hello,
>
> It seems the buildbot Web UI has the bad habit of auto-refreshing
> itself (by doing a full page reload) every minute or two.  I also
> couldn't find a setting to disable it (despite being logged in).  Am I
> missing something?
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/FFMLZFROC5DRUGSWENHZAIO4UVFGQPO4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UCIM74R5GTYDUEGFLS3WYRIDT5W2PF4L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: On the migration from master to main

[Pablo Galindo Salgado]

> f I have branch fix_something that is branched off master from a while
ago, do I need to do anything to it?

In general no, branches are basically a file that points to a given commit
so even if they branched from master the actual
commit sha is what matters. The only case if you have a branch that is
tracking master as a remote, in which case you need
to do those steps for said branch. But this is quite uncommon.

> Or will I just be able to do `git merge main` once master gets renamed,
and it'll just work with the name change making no difference?

You should be able to do that without any problem that I can foresee once
you rename it in your local fork.

> PS If I think hard about it, and ponder the git model of blobs and commits,
etc, I can see that no commits change with the renaming, and
> the names "master" and "main" are just aliases for particular commits. So
I think I've convinced myself that nothing else is needed. But that
> thought process is what I mean by "not obvious" ;-)

Exactly, that's the way I reason about this as well

Pablo


On Fri, 26 Mar 2021 at 21:19, Paul Moore  wrote:

> On Fri, 26 Mar 2021 at 19:54, Mariatta  wrote:
> >
> >
> > The branch has not been renamed. It will be renamed after the release of
> beta 1.
> >
> > There is a pending PR on DevGuide on how to change your existing local
> CPython repository to the new branch:
> >
> https://github.com/python/devguide/pull/662/files#diff-611263234e072dc403733b3dd46d0609cfe44b24d29f05841637b41d093adf0cR157
> >
> > The details and timeline on the branch renaming can be read here:
> >
> >
> https://mail.python.org/archives/list/python-committ...@python.org/message/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
>
> Thanks, that is the same set of commands I've seen in a few places.
> What isn't obvious to me (but may be to the people writing the
> documentation) is what happens to branches I have. If I have branch
> fix_something that is branched off master from a while ago, do I need
> to do anything to it? Or will I just be able to do `git merge main`
> once master gets renamed, and it'll just work with the name change
> making no difference?
>
> I suspect it *will* just work, in which case all that is needed is a
> brief note making that clear. But to many of us, it's not "obvious"
> that this is the case.
>
> Paul
>
> PS If I think hard about it, and ponder the git model of blobs and
> commits, etc, I can see that no commits change with the renaming, and
> the names "master" and "main" are just aliases for particular commits.
> So I think I've convinced myself that nothing else is needed. But that
> thought process is what I mean by "not obvious" ;-)
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/EZRUP7UEBAS65QGRSTX7UF5WNGO3DXUB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VTIWXA2KEH2WZ4MVX5OPGPJMCHCYYASK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 644 Accepted -- Require OpenSSL 1.1.1 or newer

[Pablo Galindo Salgado]

Hi Christian,

Thank you for submitting PEP 644 (Require OpenSSL 1.1.1). After evaluating
the situation and discussing the PEP, the Steering Council is happy with
the PEP,
and hereby accepts it. The SC is of the opinion that this change will make
it considerable
easier to maintain the extension modules that depend on OpenSSL while
allowing us
to leverage the new APIs and improvements in the future. As indicated in
the PEP,
OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost all
supported
platforms and distributions almost all known Major CI providers provide
images with
OpenSSL 1.1.1, so we believe this has the right balance to improve the
situation without
causing major impact or breakage.

Nevertheless, the Steering Council would like to see this change reflected
properly in the
documentation, What's New (linking against the new instructions here:
https://docs.python.org/3.10/using/unix.html#custom-openssl) and release
manager notices
so this is properly communicated to our users.

The Steering Council also thanks Christian for his patience explaining
different aspects
of the PEP, including in the PEP some extra requested information and for
considering
the concerts raised.

Congratulations, Christian!

With thanks from the whole Python Steering Council,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/INLCO2EZVQW7R7J2OL6HWVLVU3TQRAZV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: SC feedback: PEP 648 -- Extensible customizations of the interpreter at startup

[Pablo Galindo Salgado]

Hi Nick,

Please don't, since that would force everyone to start using PEP 648 just
> to extend sys.path, which would be just as bad as the status quo.


I think Barry is referring to deprecate the execution capabilities of pth
files (https://bugs.python.org/issue33944), not the files themselves.

Cheers,
Pablo Galindo Salgado

On Wed, 31 Mar 2021 at 00:34, Nick Coghlan  wrote:

>
>
> On Wed, 31 Mar 2021, 3:15 am Barry Warsaw,  wrote:
>
>> .  We would like to eventually go farther, including deprecation of pth
>> files entirely, but that is outside the scope of this PEP.
>>
>
> Please don't, since that would force everyone to start using PEP 648 just
> to extend sys.path, which would be just as bad as the status quo.
>
> Cheers,
> Nick.
>
>
>
>> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/MSOV7NKDCO7L3X46SDDLVTUEN7ER2EDB/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TZW2UT4B7OUVVZ3YCRHRZNU4HZS7W66R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 652 Accepted -- Maintaining the Stable ABI

[Pablo Galindo Salgado]

Hi Petr,

Thank you for submitting PEP 652 (Maintaining the Stable ABI). After
evaluating
the situation and discussing the PEP, the Steering Council is happy with
the PEP
and hereby accepts it. The Steering council thinks that this is a great
step forward
in order to have a clear definition of what goes into the Stable ABI and
what guarantees
the Python core team offers regarding the stable ABI while offering at the
same time a
plan to improve the maintenance and stability of the stable ABI.

We would also like to see some improvements in the official documentation
(not only on the
devguide) regarding this topic and what guarantees do we offer (currently
we only have a small
section about this in https://docs.python.org/3/c-api/stable.html but there
is a lot of information
and clarifications in the PEP that we would like to be also in the
documentation).

Congratulations, Petr!

With thanks from the whole Python Steering Council,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IN4XMFLQJ6D6V67EXU27GV3QWSEHHNNH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] The last Python 3.10 alpha (3.10.0a7) is available - Prepare for beta freeze

[Pablo Galindo Salgado]

Br. do you feel that? That's the chill of *beta freeze* coming
closer. Meanwhile, your friendly CPython release team doesn’t
rest even on holidays and we have prepared a shiny new release for you:
Python 3.10.0a7.


Dear fellow core developer:
This alpha is the last release before feature freeze (2021-05-03), so make
sure that all new features and PEPs are landed in the master branch before
we
release the first beta. Please, be specially mindfully to check the CI and
the buildbots, maybe even using the test-with-buildbots label in GitHub
before
merging so the release team don’t need to fix a bunch of reference leaks or
platform-specific problems on the first beta release.



*Go get the new alpha here:*
https://www.python.org/downloads/release/python-3100a7/


*Python 3.10.0a7*Release Date: April 5, 2021

This is an early developer preview of Python 3.10

*Major new features of the 3.10 series, compared to 3.9*

Python 3.10 is still in development. This release, 3.10.0a7 is the last of
seven planned alpha releases. Alpha releases are intended to make it easier
to test the current state of new features and bug fixes and to test the
release process. During the alpha phase, features may be added up until the
start of the beta phase (2021-05-03) and, if necessary, may be modified or
deleted up until the release candidate phase (2021-10-04). Please keep in
mind that this is a preview release and its use is not recommended for
production environments.

Many new features for Python 3.10 are still being planned and written.
Among the new major new features and changes so far:

PEP 623 – Deprecate and prepare for the removal of the wstr member in
PyUnicodeObject.
PEP 604 – Allow writing union types as X | Y
PEP 612 – Parameter Specification Variables
PEP 626 – Precise line numbers for debugging and other tools.
bpo-38605: from __future__ import annotations (PEP 563) is now the default.
PEP 618 – Add Optional Length-Checking To zip.
bpo-12782: Parenthesized context managers are now officially allowed.
PEP 632 – Deprecate distutils module.
PEP 613 – Explicit Type Aliases
PEP 634 – Structural Pattern Matching: Specification
PEP 635 – Structural Pattern Matching: Motivation and Rationale
PEP 636 – Structural Pattern Matching: Tutorial
PEP 644 – Require OpenSSL 1.1.1 or newer
PEP 624 – Remove Py_UNICODE encoder APIs
PEP 597 – Add optional EncodingWarning
(Hey, fellow core developer, if a feature you find important is missing
from this list, let Pablo know.)
The next pre-release of Python 3.10 will be 3.10.0b1 ( the first beta
release and feature freeze ), currently scheduled for Monday, 2021-05-03.

*And now for something completely different*

In physics, the twin paradox is a thought experiment in special relativity
involving identical twins, one of whom makes a journey into space in a
high-speed rocket and returns home to find that the twin who remained on
Earth has aged more. This result appears puzzling because each twin sees
the other twin as moving, and so, as a consequence of an incorrect and
naive application of time dilation and the principle of relativity, each
should paradoxically find the other to have aged less. However, this
scenario can be resolved by realising that the travelling twin is
undergoing acceleration, which makes him a non-inertial observer. In both
views, there is no symmetry between the spacetime paths of the twins.
Therefore, the twin paradox is not a paradox in the sense of a logical
contradiction.

Your friendly release team,
Pablo Galindo Salgado @pablogsal
Ned Deily @nad
Steve Dower @steve.dower
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G2J3SBJ4EOW6MQZZJQNTXKHRWTAVBMKP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [Release manager team communication] 3.10 feature freeze is 2 weeks away

[Pablo Galindo Salgado]

Hi everyone,

This is a friendly reminder from the release manager team that Python 3.10
feature freeze is two weeks away (Monday, 3 May 2021).

Please, be aware of the following:

* No new features or improvements should be merged after feature freeze.
>From the devguide (https://devguide.python.org/devcycle/#beta) :
> After a first beta release is published, no new features are accepted.
Only bug fixes can now be committed. This is when core developers should
> concentrate on the task of fixing regressions and other new issues filed
by users who have downloaded the alpha and beta release

* If your change involves some platform-specific behaviour or has a
non-trivial amount of C code, make sure you run the buildbots
in your Pull Request by using the "test-with-buildbots" label (
https://discuss.python.org/t/now-you-can-test-a-pr-with-the-buildbots-before-merging/2966
).
Alternatively you could check the buildbots post-merge in the buildbot
server:
https://buildbot.python.org/all/#/builders?tags=%2B3.x&tags=%2Bstable
This is very important because if problems are detected at the time of the
release, the release management team may have to revert
the changes and therefore those will not be included in Python 3.10.

* The master branch will be renamed to main (check our previous
communication about how this will be done and how you should
proceed:
https://mail.python.org/archives/list/python-dev@python.org/thread/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
)

If you have any questions or concerns, please contact me as soon as
possible.

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SIJQE3BZ6ICCGNJWFR4YR65BQBJJZZAZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [Release manager team communication] 3.10 feature freeze is 2 weeks away

[Pablo Galindo Salgado]

Hi everyone,

This is a friendly reminder from the release manager team that Python 3.10
feature freeze is two weeks away (Monday, 3 May 2021).

Please, be aware of the following:

* No new features or improvements should be merged after feature freeze.
>From the devguide (https://devguide.python.org/devcycle/#beta) :
> After a first beta release is published, no new features are accepted.
Only bug fixes can now be committed. This is when core developers should
> concentrate on the task of fixing regressions and other new issues filed
by users who have downloaded the alpha and beta release

* If your change involves some platform-specific behaviour or has a
non-trivial amount of C code,
*make sure you run the buildbotsin your Pull Request* by using the
"test-with-buildbots" label (
https://discuss.python.org/t/now-you-can-test-a-pr-with-the-buildbots-before-merging/2966
).
Alternatively you could check the buildbots post-merge in the buildbot
server:
https://buildbot.python.org/all/#/builders?tags=%2B3.x&tags=%2Bstable
This is *very important* because if problems are detected at the time of
the release,
*the release management team may have to revertthe changes* and therefore
those will not be included in Python 3.10.

* The master branch will be renamed to main (check our previous
communication about how this will be done and how you should
proceed:
https://mail.python.org/archives/list/python-dev@python.org/thread/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
)

If you have any questions or concerns, please contact me as soon as
possible.

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L4FJDHA7JGPX64HX6L7IDAJ5R4F6KR6G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Anyone else gotten bizarre personal replies to mailing list posts?

[Pablo Galindo Salgado]

I had I and still don't know what's going on. Mine was in a response to a
release announcement so it was extra weird. Here is what I received:

I have now formally filed a final lawsuit against the manager of the python
> program company, because all of him is also a criminal act, and GNU has EU
> legal certification, only my key can log in, and the key must be recycled
> after the death of the holder, and gitlab allows to change It’s the most
> basic and important crime to log in by people who support the snatching of
> the key. I have to explain to you that the key is to be registered and
> authenticated. My girlfriend wants to authenticate me with this key, and
> my information is also there. The key is authenticated, so I will not log
> in now, and I have submitted a lawsuit against him and the authority of the
> key holder to the U.S. Supreme Court and the European Union. I will not
> log in until there is a judgment or the U.S. Supreme Court allows me.
> People will be litigated, and the information that has been changed online
> will be found out, and I have library files, I have all the original
> materials, please cooperate with me, my key is called the Boss key, all
> websites of the program, companies, Institutions, banks, third-party
> platforms, and only my keys can have them, including patents and
> copyrights.


On Fri, 23 Apr 2021 at 16:44, Nathaniel Smith  wrote:

> I just got the reply below sent directly to my personal account, and I'm
> confused about what's going on. If it's just a one off I'll chalk it up to
> random internet weirdness, but if other folks are getting these too it
> might be something the list admins should look into? Or... something?
>
> -- Forwarded message -
> From: Hoi lam Poon 
> Date: Fri, Apr 23, 2021, 02:01
> Subject: Re: [Python-Dev] Re: PEP 654: Exception Groups and except*
> [REPOST]
> To: Nathaniel Smith 
>
>
> Stop pretending, I can definitely get the key control file, your working
> group, all past actions and instructions cannot be cleared in front of me
> at all. You have been playing around for a few days, and I won’t stop you.
> Your face? I won’t, you know, you can’t drive me away, and that file is
> all, after I get it, you will be convicted even if you disband, I swear
>
> 在 2021年4月23日 週五 16:23，Nathaniel Smith  寫道：
>
>> On Wed, Apr 21, 2021 at 4:50 PM Guido van Rossum 
>> wrote:
>> > On Wed, Apr 21, 2021 at 3:26 PM Nathaniel Smith  wrote:
>> >> Sure. This was in my list of reasons why the backwards compatibility
>> >> tradeoffs are forcing us into awkward compromises. I only elaborated
>> >> on it b/c in your last email you said you didn't understand why this
>> >> was a problem :-). And except* is definitely useful. But I think there
>> >> are options for 'except' that haven't been considered fully.
>> >
>> > Do you have any suggestions, or are you just telling us to think
>> harder? Because we've already thought as hard as we could and within all
>> the constraints (backwards compatibility and otherwise) we just couldn't
>> think of a better one.
>>
>> The main possibility that I don't think we've examined fully is to
>> make 'except' blocks fire multiple times when there are multiple
>> exceptions. We ruled it out early b/c it's incompatible with nested
>> EGs, but if flat EGs are better anyway, then the balance shifts around
>> and it might land somewhere different. it's a tricky discussion
>> though, b/c both the current proposal and the alternative have very
>> complex implications and downsides. So we probably shouldn't get too
>> distracted by that until after the flat vs nested discussion has
>> settled down more.
>>
>> I'm not trying to filibuster here -- I really want some form of EGs to
>> land. I think python has the potential to be the most elegant and
>> accessible language around for writing concurrent programs, and EGs
>> are a key part of that. I don't want to fight about anything; I just
>> want to work together to make sure we have a full picture of our
>> options, so we can be confident we're making the best choice.
>>
>> > The real cost here is that we would need a new "TracebackGroup"
>> concept, since the internal data structures and APIs keep the traceback
>> chain and the exception object separated until the exception is caught. In
>> our early design stages we actually explored this and the complexity of the
>> data structures was painful. We eventually realized that we didn't need
>> this concept at all, and the result is much clearer, despite what you seem
>> to think.
>>
>> I'm not talking about TracebackGroups (at least, I think I'm not?). I
>> think it can be done with exactly our current data structures, nothing
>> new.
>>
>> - When an EG is raised, build the traceback for just that EG while
>> it's unwinding. This means if any C code peeks at exc_info while it's
>> in flight, it'll only see the current branch of the traceback tree,
>> but that seems fine.
>> - When the exception is caught and we go 

[Python-Dev] [Release management team communication] 3.10 feature freeze is 1 week away

[Pablo Galindo Salgado]

Hi everyone,

This is the last friendly reminder from the release management team that
Python 3.10 feature freeze is a week away (Monday, 3 May 2021).

Some important things to have in mind:

* We have some release blockers that must be resolved before the release is
done:

https://bugs.python.org/issue43908
https://bugs.python.org/issue43933
https://bugs.python.org/issue43916

If you want to help making sure that the release is on time, help us
resolving those blockers. You can also search the issues in
https://bugs.python.org/issue?@template=search&status=1 with the
"release-blocker" priority.

* Please, don't wait until the last moment (Sunday) to merge PRs into
master. This is because if many changes accumulate into a time window and
problems are detected, is much more difficult to locate the problematic
changes among the rest. If there are any problems with your commits, you
also would have the risk of having your changes reverted to unblock the
release without any time window to merge them again before feature freeze.

Please, also be aware of the following:

* No new features or improvements should be merged after the feature
freeze. From the devguide (https://devguide.python.org/devcycle/#beta) :

> After a first beta release is published, no new features are accepted.
Only bug fixes and improvements to documentation and tests can now be
committed.
> This is when core developers should concentrate on the task of fixing
regressions and other new issues filed by users who have downloaded the
alpha and beta releases.
> Being in beta can be viewed much like being in RC but without the extra
overhead of needing commit reviews.

* If your change involves some platform-specific behaviour or has a
non-trivial amount of C code, make sure you run the buildbots
in your Pull Request by using the "test-with-buildbots" label (
https://discuss.python.org/t/now-you-can-test-a-pr-with-the-buildbots-before-merging/2966
).
Alternatively you could check the buildbots post-merge in the buildbot
server:
https://buildbot.python.org/all/#/builders?tags=%2B3.x&tags=%2Bstable
This is very important because if problems are detected at the time of the
release, the release management team may have to revert
the changes and therefore those will not be included in Python 3.10.

* The master branch will be renamed to main (check our previous
communication about how this will be done and how you should
proceed
https://mail.python.org/archives/list/python-dev@python.org/thread/QWW7KGBW5UH2N5FOZOFXQBQPYELWQM3O/
)

If you have any questions or concerns, please contact me as soon as
possible.

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V7V5JHKZCJVE2GTI5NFEP3PNKOLH35VL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [Release manager team communication] master blocked until 3.10 beta 1 is released

[Pablo Galindo Salgado]

Hi everyone,

Unfortunately, the CI for master is currently quite unstable and he had
some problems that we
have been fixing all weekend and today. We are doing our best to stabilize
the test suite and making
sure that everything is ready to go for the release with no issues, but
this will take some time (also
bear in mind that today is a bank holiday in the UK, so I am doing my best
to get the release ready
in a holiday). Given this and the fact that we need to do a branch rename,
**we have decided to block the master branch** until we can stabilize the
test suite.

In the meantime, if you have some urgent fix or PR that you absolutely need
to get merge before
feature freeze, you can add me as a reviewer to said PR so we can evaluate
the merge.

Thanks for your understanding.

I will update once master is unblocked again.

Regards from cloudy London, Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AKSCARRZJKKU427Z7DIMYO7MK5RSXOFR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hi there,

We are preparing a PEP and we would like to start some early discussion
about one of the main aspects of the PEP.

The work we are preparing is to allow the interpreter to produce more
fine-grained error messages, pointing to
the source associated to the instructions that are failing. For example:

Traceback (most recent call last):

  File "test.py", line 14, in 

lel3(x)

^^^

  File "test.py", line 12, in lel3

return lel2(x) / 23

   ^^^

  File "test.py", line 9, in lel2

return 25 + lel(x) + lel(x)

^^

  File "test.py", line 6, in lel

return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)

 ^

TypeError: 'NoneType' object is not subscriptable

The cost of this is having the start column number and end column number
information for every bytecode instruction
and this is what we want to discuss (there is also some stack cost to
re-raise exceptions but that's not a big problem in
any case). Given that column numbers are not very big compared with line
numbers, we plan to store these as unsigned chars
or unsigned shorts. We ran some experiments over the standard library and
we found that the overhead of all pyc files is:

* If we use shorts, the total overhead is ~3% (total size 28MB and the
extra size is 0.88 MB).
* If we use chars. the total overhead is ~1.5% (total size 28 MB and the
extra size is 0.44MB).

One of the disadvantages of using chars is that we can only report columns
from 1 to 255 so if an error happens in a column
bigger than that then we would have to exclude it (and not show the
highlighting) for that frame. Unsigned short will allow
the values to go from 0 to 65535.

Unfortunately these numbers are not easily compressible, as every
instruction would have very different offsets.

There is also the possibility of not doing this based on some build flag on
when using -O to allow users to opt out, but given the fact
that these numbers can be quite useful to other tools like coverage
measuring tools, tracers, profilers and the such adding conditional
logic to many places would complicate the implementation considerably and
will potentially reduce the usability of those tools so we prefer
not to have the conditional logic. We believe this is extra cost is very
much worth the better error reporting but we understand and respect
other points of view.

Does anyone see a better way to encode this information **without
complicating a lot the implementation**? What are people thoughts on the
feature?

Thanks in advance,

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DB3RTYBF2BXTY6ZHP3Z4DXCRWPJIQUFD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Technically the main concern may be the size of the unmarshalled pyc files
in memory, more than the storage size of disk.

On Fri, 7 May 2021, 23:04 Antoine Pitrou,  wrote:

> On Fri, 7 May 2021 22:45:38 +0100
> Pablo Galindo Salgado  wrote:
> >
> > The cost of this is having the start column number and end column number
> > information for every bytecode instruction
> > and this is what we want to discuss (there is also some stack cost to
> > re-raise exceptions but that's not a big problem in
> > any case). Given that column numbers are not very big compared with line
> > numbers, we plan to store these as unsigned chars
> > or unsigned shorts. We ran some experiments over the standard library and
> > we found that the overhead of all pyc files is:
> >
> > * If we use shorts, the total overhead is ~3% (total size 28MB and the
> > extra size is 0.88 MB).
> > * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
> > extra size is 0.44MB).
>
> More generally, if some people in 2021 are still concerned with the size
> of pyc files (why not), how about introducing a new version of the pyc
> format with built-in LZ4 compression?
>
> LZ4 decompression is extremely fast on modern CPUs (several GB/s) and
> vendoring the C library should be simple.
> https://github.com/lz4/lz4
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/PQZ6OTWG6K6W65YXRLKEH7UOD5FM24TN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SNDIHPBDW4Y3KGSOEL7MBJER3IEBIFTN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Thanks a lot Gregory for the comments!

An additional cost to this is things that parse text tracebacks not knowing
> how to handle it and things that log tracebacks generating additional
> output.

We should provide a way for people to disable the feature on a process as
> part of this while they address tooling and logging issues.  (via the usual
> set of command line flag + python env var + runtime API)


Absolutely! We were thinking about that and that's easy enough as that is a
single conditional on the display function + the extra init configuration.

Neither of those is large. While I'd lean towards uint8_t instead of
> uint16_t because not even humans can understand a 255 character line so why
> bother being pretty about such a thing... Just document the caveat and move
> on with the lower value. A future pyc format could change it if a
> compelling argument were ever found.


I very much agree with you here but is worth noting that I have heard the
counter-argument that the longer the line is, the more important may be to
distinguish what part of the line is wrong.

A compromise if you want to handle longer lines: A single uint16_t.
> Represent the start column in the 9 bits and width in the other 7 bits. (or
> any variations thereof)  it's all a matter of what tradeoff you want to
> make for space reasons.  encoding as start + width instead of start + end
> is likely better anyways if you care about compression as the width byte
> will usually be small and thus be friendlier to compression.  I'd
> personally ignore compression entirely.


I would personally prefer not to implement very tricky compression
algorithms because tools may need to parse this and I don't want to
complicate the logic a lot. Handling lnotab is already a bit painful and
when bugs ocur it makes debugging very tricky. Having the possibility to
index something based on the index of the instruction is quite a good API
in my opinion.

Overall doing this is going to be a big win for developer productivity!


Thanks! We think that this has a lot of potential indeed! :)

Pablo
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OHSQ6VLMVSZHCLEUQZ52NXCWEGLG2DQN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

This is actually a very good point. The only disadvantage is that it
complicates the parsing a bit and we loose the possibility of indexing
the table by instruction offset.

On Fri, 7 May 2021 at 23:01, Larry Hastings  wrote:

> On 5/7/21 2:45 PM, Pablo Galindo Salgado wrote:
>
> Given that column numbers are not very big compared with line numbers, we
> plan to store these as unsigned chars
> or unsigned shorts. We ran some experiments over the standard library and
> we found that the overhead of all pyc files is:
>
> * If we use shorts, the total overhead is ~3% (total size 28MB and the
> extra size is 0.88 MB).
> * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
> extra size is 0.44MB).
>
> One of the disadvantages of using chars is that we can only report columns
> from 1 to 255 so if an error happens in a column
> bigger than that then we would have to exclude it (and not show the
> highlighting) for that frame. Unsigned short will allow
> the values to go from 0 to 65535.
>
> Are lnotab entries required to be a fixed size?  If not:
>
> if column < 255:
> lnotab.write_one_byte(column)
> else:
> lnotab.write_one_byte(255)
> lnotab.write_two_bytes(column)
>
>
> I might even write four bytes instead of two in the latter case,
>
>
> */arry*
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/B3SFCZPXIKGO3LM6UJVSJXFIRAZH2R26/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SJP2RIMEFEVWKBWOA2V2X4BMFGHHEZ5J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> haha, true... Does our parser even have a maximum line length? (I'm not
suggesting being unlimited or matching that if huge, 64k is already
ridiculous

We use py_ssize_t in some places but at the end of the day the lines and
columns have a limit of INT_MAX IIRC


On Fri, 7 May 2021 at 23:35, Gregory P. Smith  wrote:

>
> On Fri, May 7, 2021 at 3:24 PM Pablo Galindo Salgado 
> wrote:
>
>> Thanks a lot Gregory for the comments!
>>
>> An additional cost to this is things that parse text tracebacks not
>>> knowing how to handle it and things that log tracebacks
>>> generating additional output.
>>
>> We should provide a way for people to disable the feature on a process as
>>> part of this while they address tooling and logging issues.  (via the usual
>>> set of command line flag + python env var + runtime API)
>>
>>
>> Absolutely! We were thinking about that and that's easy enough as that is
>> a single conditional on the display function + the extra init configuration.
>>
>> Neither of those is large. While I'd lean towards uint8_t instead of
>>> uint16_t because not even humans can understand a 255 character line so why
>>> bother being pretty about such a thing... Just document the caveat and move
>>> on with the lower value. A future pyc format could change it if a
>>> compelling argument were ever found.
>>
>>
>> I very much agree with you here but is worth noting that I have heard the
>> counter-argument that the longer the line is, the more important may be to
>> distinguish what part of the line is wrong.
>>
>
> haha, true... Does our parser even have a maximum line length? (I'm not
> suggesting being unlimited or matching that if huge, 64k is already
> ridiculous)
>
>
>>
>> A compromise if you want to handle longer lines: A single uint16_t.
>>> Represent the start column in the 9 bits and width in the other 7 bits. (or
>>> any variations thereof)  it's all a matter of what tradeoff you want to
>>> make for space reasons.  encoding as start + width instead of start + end
>>> is likely better anyways if you care about compression as the width byte
>>> will usually be small and thus be friendlier to compression.  I'd
>>> personally ignore compression entirely.
>>
>>
>> I would personally prefer not to implement very tricky compression
>> algorithms because tools may need to parse this and I don't want to
>> complicate the logic a lot. Handling lnotab is already a bit painful and
>> when bugs ocur it makes debugging very tricky. Having the possibility to
>> index something based on the index of the instruction is quite a good API
>> in my opinion.
>>
>> Overall doing this is going to be a big win for developer productivity!
>>
>>
>> Thanks! We think that this has a lot of potential indeed! :)
>>
>> Pablo
>>
>>
>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TNTBLEMSQ7JKZ2I75WJZSQHYIB6NSXCS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Thanks, Irit for your comment!

> Is it really every instruction? Or only those that can raise exceptions?

Technically only the ones that can raise exceptions, but the majority can
and optimizing this to only restrict to the set that can raise exceptions
has
the danger than the mapping needs to be maintained for new instructions and
that if some instruction starts raising exceptions while it didn't before
then it can introduce subtle bugs.

On the other hand I think the stronger argument to do this on every
instruction is that there is a lot of tools that can find this information
quite useful
such as coverage tools, profilers, state inspection tools and more. For
example, a coverage tool will be able to tell you what part of

x = f(x) if g(x) else y(x)

actually was executed, while currently, it will highlight the full line.
Although in this case these instructions can raise exceptions and would be
covered,
the distinction is different and both criteria could lead to a different
subset.

In short, that may be an optimization but I think I would prefer to avoid
that complexity taking into account the other problems that can raise and
the extra complication

On Fri, 7 May 2021 at 23:21, Irit Katriel 
wrote:

>
>
> On Fri, May 7, 2021 at 10:52 PM Pablo Galindo Salgado 
> wrote:
>
>>
>> The cost of this is having the start column number and end column number
>> information for every bytecode instruction
>>
>
>
> Is it really every instruction? Or only those that can raise exceptions?
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W72ZOEIRWXSIY5OCGTBRSGHHKDXGZL6Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> One thought: could the stored column position not include the
indentation? Would that help?

The compiler doesn't have access easy access to the source unfortunately so
we don't know how much is the indentation. This can make life
a bit harder for other tools, although it can make it easier for reporting
the exception as the current traceback display removes indentation.


On Fri, 7 May 2021 at 23:37, MRAB  wrote:

> On 2021-05-07 22:45, Pablo Galindo Salgado wrote:
> > Hi there,
> >
> > We are preparing a PEP and we would like to start some early discussion
> > about one of the main aspects of the PEP.
> >
> > The work we are preparing is to allow the interpreter to produce more
> > fine-grained error messages, pointing to
> > the source associated to the instructions that are failing. For example:
> >
> > Traceback (most recent call last):
> >
> >File "test.py", line 14, in 
> >
> >  lel3(x)
> >
> >  ^^^
> >
> >File "test.py", line 12, in lel3
> >
> >  return lel2(x) / 23
> >
> > ^^^
> >
> >File "test.py", line 9, in lel2
> >
> >  return 25 + lel(x) + lel(x)
> >
> >  ^^
> >
> >File "test.py", line 6, in lel
> >
> >  return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
> >
> >   ^
> >
> > TypeError: 'NoneType' object is not subscriptable
> >
> >
> > The cost of this is having the start column number and end column number
> > information for every bytecode instruction
> > and this is what we want to discuss (there is also some stack cost to
> > re-raise exceptions but that's not a big problem in
> > any case). Given that column numbers are not very big compared with line
> > numbers, we plan to store these as unsigned chars
> > or unsigned shorts. We ran some experiments over the standard library
> > and we found that the overhead of all pyc files is:
> >
> > * If we use shorts, the total overhead is ~3% (total size 28MB and the
> > extra size is 0.88 MB).
> > * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
> > extra size is 0.44MB).
> >
> > One of the disadvantages of using chars is that we can only report
> > columns from 1 to 255 so if an error happens in a column
> > bigger than that then we would have to exclude it (and not show the
> > highlighting) for that frame. Unsigned short will allow
> > the values to go from 0 to 65535.
> >
> [snip]How common are lines are longer than 255 characters, anyway?
>
> One thought: could the stored column position not include the
> indentation? Would that help?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/MHF3PMCJOR6VK765OSA7NSO66NY3QU3V/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OKQYNAI2B2BRCFMYJPLYPG2HHHUB5QR6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Some update on the numbers. We have made some draft implementation to
corroborate the
numbers with some more realistic tests and seems that our original
calculations were wrong.
The actual increase in size is quite bigger than previously advertised:

Using bytes object to encode the final object and marshalling that to disk
(so using uint8_t) as the underlying
type:

BEFORE:

❯ ./python -m compileall -r 1000 Lib > /dev/null
❯ du -h Lib -c --max-depth=0
70M Lib
70M total

AFTER:
❯ ./python -m compileall -r 1000 Lib > /dev/null
❯ du -h Lib -c --max-depth=0
76M Lib
76M total

So that's an increase of 8.56 % over the original value. This is storing
the start offset and end offset with no compression
whatsoever.

On Fri, 7 May 2021 at 22:45, Pablo Galindo Salgado 
wrote:

> Hi there,
>
> We are preparing a PEP and we would like to start some early discussion
> about one of the main aspects of the PEP.
>
> The work we are preparing is to allow the interpreter to produce more
> fine-grained error messages, pointing to
> the source associated to the instructions that are failing. For example:
>
> Traceback (most recent call last):
>
>   File "test.py", line 14, in 
>
> lel3(x)
>
> ^^^
>
>   File "test.py", line 12, in lel3
>
> return lel2(x) / 23
>
>^^^
>
>   File "test.py", line 9, in lel2
>
> return 25 + lel(x) + lel(x)
>
> ^^
>
>   File "test.py", line 6, in lel
>
> return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
>
>  ^
>
> TypeError: 'NoneType' object is not subscriptable
>
> The cost of this is having the start column number and end column number
> information for every bytecode instruction
> and this is what we want to discuss (there is also some stack cost to
> re-raise exceptions but that's not a big problem in
> any case). Given that column numbers are not very big compared with line
> numbers, we plan to store these as unsigned chars
> or unsigned shorts. We ran some experiments over the standard library and
> we found that the overhead of all pyc files is:
>
> * If we use shorts, the total overhead is ~3% (total size 28MB and the
> extra size is 0.88 MB).
> * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
> extra size is 0.44MB).
>
> One of the disadvantages of using chars is that we can only report columns
> from 1 to 255 so if an error happens in a column
> bigger than that then we would have to exclude it (and not show the
> highlighting) for that frame. Unsigned short will allow
> the values to go from 0 to 65535.
>
> Unfortunately these numbers are not easily compressible, as every
> instruction would have very different offsets.
>
> There is also the possibility of not doing this based on some build flag
> on when using -O to allow users to opt out, but given the fact
> that these numbers can be quite useful to other tools like coverage
> measuring tools, tracers, profilers and the such adding conditional
> logic to many places would complicate the implementation considerably and
> will potentially reduce the usability of those tools so we prefer
> not to have the conditional logic. We believe this is extra cost is very
> much worth the better error reporting but we understand and respect
> other points of view.
>
> Does anyone see a better way to encode this information **without
> complicating a lot the implementation**? What are people thoughts on the
> feature?
>
> Thanks in advance,
>
> Regards from cloudy London,
> Pablo Galindo Salgado
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QDEKMTZRMPEKPFFBPCGUYWLLR43A6M6U/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

One last note for clarity: that's the increase of size in the stdlib, the
increase of size
for pyc files goes from 28.471296MB to 34.750464MB, which is an increase of
22%.

On Sat, 8 May 2021 at 01:43, Pablo Galindo Salgado 
wrote:

> Some update on the numbers. We have made some draft implementation to
> corroborate the
> numbers with some more realistic tests and seems that our original
> calculations were wrong.
> The actual increase in size is quite bigger than previously advertised:
>
> Using bytes object to encode the final object and marshalling that to disk
> (so using uint8_t) as the underlying
> type:
>
> BEFORE:
>
> ❯ ./python -m compileall -r 1000 Lib > /dev/null
> ❯ du -h Lib -c --max-depth=0
> 70M Lib
> 70M total
>
> AFTER:
> ❯ ./python -m compileall -r 1000 Lib > /dev/null
> ❯ du -h Lib -c --max-depth=0
> 76M Lib
> 76M total
>
> So that's an increase of 8.56 % over the original value. This is storing
> the start offset and end offset with no compression
> whatsoever.
>
> On Fri, 7 May 2021 at 22:45, Pablo Galindo Salgado 
> wrote:
>
>> Hi there,
>>
>> We are preparing a PEP and we would like to start some early discussion
>> about one of the main aspects of the PEP.
>>
>> The work we are preparing is to allow the interpreter to produce more
>> fine-grained error messages, pointing to
>> the source associated to the instructions that are failing. For example:
>>
>> Traceback (most recent call last):
>>
>>   File "test.py", line 14, in 
>>
>> lel3(x)
>>
>> ^^^
>>
>>   File "test.py", line 12, in lel3
>>
>> return lel2(x) / 23
>>
>>^^^
>>
>>   File "test.py", line 9, in lel2
>>
>> return 25 + lel(x) + lel(x)
>>
>> ^^
>>
>>   File "test.py", line 6, in lel
>>
>> return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
>>
>>  ^
>>
>> TypeError: 'NoneType' object is not subscriptable
>>
>> The cost of this is having the start column number and end column number
>> information for every bytecode instruction
>> and this is what we want to discuss (there is also some stack cost to
>> re-raise exceptions but that's not a big problem in
>> any case). Given that column numbers are not very big compared with line
>> numbers, we plan to store these as unsigned chars
>> or unsigned shorts. We ran some experiments over the standard library and
>> we found that the overhead of all pyc files is:
>>
>> * If we use shorts, the total overhead is ~3% (total size 28MB and the
>> extra size is 0.88 MB).
>> * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
>> extra size is 0.44MB).
>>
>> One of the disadvantages of using chars is that we can only report
>> columns from 1 to 255 so if an error happens in a column
>> bigger than that then we would have to exclude it (and not show the
>> highlighting) for that frame. Unsigned short will allow
>> the values to go from 0 to 65535.
>>
>> Unfortunately these numbers are not easily compressible, as every
>> instruction would have very different offsets.
>>
>> There is also the possibility of not doing this based on some build flag
>> on when using -O to allow users to opt out, but given the fact
>> that these numbers can be quite useful to other tools like coverage
>> measuring tools, tracers, profilers and the such adding conditional
>> logic to many places would complicate the implementation considerably and
>> will potentially reduce the usability of those tools so we prefer
>> not to have the conditional logic. We believe this is extra cost is very
>> much worth the better error reporting but we understand and respect
>> other points of view.
>>
>> Does anyone see a better way to encode this information **without
>> complicating a lot the implementation**? What are people thoughts on the
>> feature?
>>
>> Thanks in advance,
>>
>> Regards from cloudy London,
>> Pablo Galindo Salgado
>>
>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RICGTXCABZPK7RLDB7SISR4E64S6FEKR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> I'm wondering if it's possible to compromise with one position that's
not as complete but still gives a good hint:

Even if is possible, it will be quite less useful (a lot of users wanted to
highlight full ranges for syntax errors, and that change was very well
received
when we announce it in 3.10) and most importantly, will render the feature
much less useful for other tools such as profilers, coverage tools, and the
like.

It will also make the feature less useful for people that want to display
even more information such as error reporting tools, IDEsetc


On Sat, 8 May 2021 at 02:41, MRAB  wrote:

> On 2021-05-08 01:43, Pablo Galindo Salgado wrote:
> > Some update on the numbers. We have made some draft implementation to
> > corroborate the
> > numbers with some more realistic tests and seems that our original
> > calculations were wrong.
> > The actual increase in size is quite bigger than previously advertised:
> >
> > Using bytes object to encode the final object and marshalling that to
> > disk (so using uint8_t) as the underlying
> > type:
> >
> > BEFORE:
> >
> > ❯ ./python -m compileall -r 1000 Lib > /dev/null
> > ❯ du -h Lib -c --max-depth=0
> > 70M Lib
> > 70M total
> >
> > AFTER:
> > ❯ ./python -m compileall -r 1000 Lib > /dev/null
> > ❯ du -h Lib -c --max-depth=0
> > 76M Lib
> > 76M total
> >
> > So that's an increase of 8.56 % over the original value. This is storing
> > the start offset and end offset with no compression
> > whatsoever.
> >
> [snip]
>
> I'm wondering if it's possible to compromise with one position that's
> not as complete but still gives a good hint:
>
> For example:
>
>File "test.py", line 6, in lel
>  return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
>^
>
> TypeError: 'NoneType' object is not subscriptable
>
> That at least tells you which subscript raised the exception.
>
>
> Another example:
>
>Traceback (most recent call last):
>  File "test.py", line 4, in 
>print(1 / x + 1 / y)
>^
>ZeroDivisionError: division by zero
>
> as distinct from:
>
>Traceback (most recent call last):
>  File "test.py", line 4, in 
>print(1 / x + 1 / y)
>^
>ZeroDivisionError: division by zero
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/4RGQALI6T6HBNRDUUEYX4FA2YKTZDBNA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7OKDSXZZ7TQFQ3X4RZGNGLX5UDF2B5QW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Although we were originally not sympathetic with it, we may need to offer
an opt-out mechanism for those users that care about the impact of the
overhead of the new data in pyc files
and in in-memory code objectsas was suggested by some folks (Thomas, Yury,
and others). For this, we could propose that the functionality will be
deactivated along with the extra
information when Python is executed in optimized mode (``python -O``) and
therefore pyo files will not have the overhead associated with the extra
required data. Notice that Python
already strips docstrings in this mode so it would be "aligned" with the
current mechanism of optimized mode.

Although this complicates the implementation, it certainly is still much
easier than dealing with compression (and more useful for those that don't
want the feature). Notice that we also
expect pessimistic results from compression as offsets would be quite
random (although predominantly in the range 10 - 120).

On Sat, 8 May 2021 at 01:56, Pablo Galindo Salgado 
wrote:

> One last note for clarity: that's the increase of size in the stdlib, the
> increase of size
> for pyc files goes from 28.471296MB to 34.750464MB, which is an increase
> of 22%.
>
> On Sat, 8 May 2021 at 01:43, Pablo Galindo Salgado 
> wrote:
>
>> Some update on the numbers. We have made some draft implementation to
>> corroborate the
>> numbers with some more realistic tests and seems that our original
>> calculations were wrong.
>> The actual increase in size is quite bigger than previously advertised:
>>
>> Using bytes object to encode the final object and marshalling that to
>> disk (so using uint8_t) as the underlying
>> type:
>>
>> BEFORE:
>>
>> ❯ ./python -m compileall -r 1000 Lib > /dev/null
>> ❯ du -h Lib -c --max-depth=0
>> 70M Lib
>> 70M total
>>
>> AFTER:
>> ❯ ./python -m compileall -r 1000 Lib > /dev/null
>> ❯ du -h Lib -c --max-depth=0
>> 76M Lib
>> 76M total
>>
>> So that's an increase of 8.56 % over the original value. This is storing
>> the start offset and end offset with no compression
>> whatsoever.
>>
>> On Fri, 7 May 2021 at 22:45, Pablo Galindo Salgado 
>> wrote:
>>
>>> Hi there,
>>>
>>> We are preparing a PEP and we would like to start some early discussion
>>> about one of the main aspects of the PEP.
>>>
>>> The work we are preparing is to allow the interpreter to produce more
>>> fine-grained error messages, pointing to
>>> the source associated to the instructions that are failing. For example:
>>>
>>> Traceback (most recent call last):
>>>
>>>   File "test.py", line 14, in 
>>>
>>> lel3(x)
>>>
>>> ^^^
>>>
>>>   File "test.py", line 12, in lel3
>>>
>>> return lel2(x) / 23
>>>
>>>^^^
>>>
>>>   File "test.py", line 9, in lel2
>>>
>>> return 25 + lel(x) + lel(x)
>>>
>>> ^^
>>>
>>>   File "test.py", line 6, in lel
>>>
>>> return 1 + foo(a,b,c=x['z']['x']['y']['z']['y'], d=e)
>>>
>>>  ^
>>>
>>> TypeError: 'NoneType' object is not subscriptable
>>>
>>> The cost of this is having the start column number and end column number
>>> information for every bytecode instruction
>>> and this is what we want to discuss (there is also some stack cost to
>>> re-raise exceptions but that's not a big problem in
>>> any case). Given that column numbers are not very big compared with line
>>> numbers, we plan to store these as unsigned chars
>>> or unsigned shorts. We ran some experiments over the standard library
>>> and we found that the overhead of all pyc files is:
>>>
>>> * If we use shorts, the total overhead is ~3% (total size 28MB and the
>>> extra size is 0.88 MB).
>>> * If we use chars. the total overhead is ~1.5% (total size 28 MB and the
>>> extra size is 0.44MB).
>>>
>>> One of the disadvantages of using chars is that we can only report
>>> columns from 1 to 255 so if an error happens in a column
>>> bigger than that then we would have to exclude it (and not show the
>>> highlighting) for that frame. Unsigned short will allow
>>> the values to go from 0 to 65535.
>>>
>>> Unfortunately these numbers are 

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hi Brett,

Just to be clear, .pyo files have not existed for a while:
> https://www.python.org/dev/peps/pep-0488/.


Whoops, my bad, I wanted to refer to the pyc files that are generated
with -OO, which have the "opt-2" prefix.

This only kicks in at the -OO level.


I will correct the PEP so it reflex this more exactly.

I personally prefer the idea of dropping the data with -OO since if you're
> stripping out docstrings you're already hurting introspection capabilities
> in the name of memory. Or one could go as far as to introduce -Os to do -OO
> plus dropping this extra data.


This is indeed the plan, sorry for the confusion. The opt-out mechanism is
using -OO, precisely as we are already dropping other data.

Thanks for the clarifications!



On Sat, 8 May 2021 at 19:41, Brett Cannon  wrote:

>
>
> On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado 
> wrote:
>
>> Although we were originally not sympathetic with it, we may need to offer
>> an opt-out mechanism for those users that care about the impact of the
>> overhead of the new data in pyc files
>> and in in-memory code objectsas was suggested by some folks (Thomas,
>> Yury, and others). For this, we could propose that the functionality will
>> be deactivated along with the extra
>> information when Python is executed in optimized mode (``python -O``) and
>> therefore pyo files will not have the overhead associated with the extra
>> required data.
>>
>
> Just to be clear, .pyo files have not existed for a while:
> https://www.python.org/dev/peps/pep-0488/.
>
>
>> Notice that Python
>> already strips docstrings in this mode so it would be "aligned" with the
>> current mechanism of optimized mode.
>>
>
> This only kicks in at the -OO level.
>
>
>>
>> Although this complicates the implementation, it certainly is still much
>> easier than dealing with compression (and more useful for those that don't
>> want the feature). Notice that we also
>> expect pessimistic results from compression as offsets would be quite
>> random (although predominantly in the range 10 - 120).
>>
>
> I personally prefer the idea of dropping the data with -OO since if you're
> stripping out docstrings you're already hurting introspection capabilities
> in the name of memory. Or one could go as far as to introduce -Os to do -OO
> plus dropping this extra data.
>
> As for .pyc file size, I personally wouldn't worry about it. If someone is
> that space-constrained they either aren't using .pyc files or are only
> shipping a single set of .pyc files under -OO and skipping source code. And
> .pyc files are an implementation detail of CPython so there  shouldn't be
> too much of a concern for other interpreters.
>
> -Brett
>
>
>>
>> On Sat, 8 May 2021 at 01:56, Pablo Galindo Salgado 
>> wrote:
>>
>>> One last note for clarity: that's the increase of size in the stdlib,
>>> the increase of size
>>> for pyc files goes from 28.471296MB to 34.750464MB, which is an increase
>>> of 22%.
>>>
>>> On Sat, 8 May 2021 at 01:43, Pablo Galindo Salgado 
>>> wrote:
>>>
>>>> Some update on the numbers. We have made some draft implementation to
>>>> corroborate the
>>>> numbers with some more realistic tests and seems that our original
>>>> calculations were wrong.
>>>> The actual increase in size is quite bigger than previously advertised:
>>>>
>>>> Using bytes object to encode the final object and marshalling that to
>>>> disk (so using uint8_t) as the underlying
>>>> type:
>>>>
>>>> BEFORE:
>>>>
>>>> ❯ ./python -m compileall -r 1000 Lib > /dev/null
>>>> ❯ du -h Lib -c --max-depth=0
>>>> 70M Lib
>>>> 70M total
>>>>
>>>> AFTER:
>>>> ❯ ./python -m compileall -r 1000 Lib > /dev/null
>>>> ❯ du -h Lib -c --max-depth=0
>>>> 76M Lib
>>>> 76M total
>>>>
>>>> So that's an increase of 8.56 % over the original value. This is
>>>> storing the start offset and end offset with no compression
>>>> whatsoever.
>>>>
>>>> On Fri, 7 May 2021 at 22:45, Pablo Galindo Salgado 
>>>> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> We are preparing a PEP and we would like to start some early
>>>>> discussion about one of the main aspects of the PEP.
>>>>>
>>>>> The work we are 

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> We can't piggy back on -OO as the only way to disable this, it needs to
have an option of its own.  -OO is unusable as code that relies on
"doc"strings as application data such as http://www.dabeaz.com/ply/ply.html
exists.

-OO is the only sensible way to disable the data. There are two things to
disable:

* The data in pyc files
* Printing the exception highlighting

Printing the exception highlighting can be disabled via combo of
environment variable / -X option but collecting the data can only be
disabled by -OO. The reason is that this will end in pyc files
so when the data is not there, a different kind of pyc files need to be
produced and I really don't want to have another set of pyc file extension
just to deactivate this. Notice that also a configure
time variable won't work because it will cause crashes when reading pyc
files produced by the interpreter compiled without the flag.

On Sat, 8 May 2021 at 21:13, Gregory P. Smith  wrote:

>
>
> On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado 
> wrote:
>
>> Hi Brett,
>>
>> Just to be clear, .pyo files have not existed for a while:
>>> https://www.python.org/dev/peps/pep-0488/.
>>
>>
>> Whoops, my bad, I wanted to refer to the pyc files that are generated
>> with -OO, which have the "opt-2" prefix.
>>
>> This only kicks in at the -OO level.
>>
>>
>> I will correct the PEP so it reflex this more exactly.
>>
>> I personally prefer the idea of dropping the data with -OO since if
>>> you're stripping out docstrings you're already hurting introspection
>>> capabilities in the name of memory. Or one could go as far as to introduce
>>> -Os to do -OO plus dropping this extra data.
>>
>>
>> This is indeed the plan, sorry for the confusion. The opt-out mechanism
>> is using -OO, precisely as we are already dropping other data.
>>
>
> We can't piggy back on -OO as the only way to disable this, it needs to
> have an option of its own.  -OO is unusable as code that relies on
> "doc"strings as application data such as
> http://www.dabeaz.com/ply/ply.html exists.
>
> -gps
>
>
>>
>> Thanks for the clarifications!
>>
>>
>>
>> On Sat, 8 May 2021 at 19:41, Brett Cannon  wrote:
>>
>>>
>>>
>>> On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado <
>>> pablog...@gmail.com> wrote:
>>>
>>>> Although we were originally not sympathetic with it, we may need to
>>>> offer an opt-out mechanism for those users that care about the impact of
>>>> the overhead of the new data in pyc files
>>>> and in in-memory code objectsas was suggested by some folks (Thomas,
>>>> Yury, and others). For this, we could propose that the functionality will
>>>> be deactivated along with the extra
>>>> information when Python is executed in optimized mode (``python -O``)
>>>> and therefore pyo files will not have the overhead associated with the
>>>> extra required data.
>>>>
>>>
>>> Just to be clear, .pyo files have not existed for a while:
>>> https://www.python.org/dev/peps/pep-0488/.
>>>
>>>
>>>> Notice that Python
>>>> already strips docstrings in this mode so it would be "aligned" with
>>>> the current mechanism of optimized mode.
>>>>
>>>
>>> This only kicks in at the -OO level.
>>>
>>>
>>>>
>>>> Although this complicates the implementation, it certainly is still
>>>> much easier than dealing with compression (and more useful for those that
>>>> don't want the feature). Notice that we also
>>>> expect pessimistic results from compression as offsets would be quite
>>>> random (although predominantly in the range 10 - 120).
>>>>
>>>
>>> I personally prefer the idea of dropping the data with -OO since if
>>> you're stripping out docstrings you're already hurting introspection
>>> capabilities in the name of memory. Or one could go as far as to introduce
>>> -Os to do -OO plus dropping this extra data.
>>>
>>> As for .pyc file size, I personally wouldn't worry about it. If someone
>>> is that space-constrained they either aren't using .pyc files or are only
>>> shipping a single set of .pyc files under -OO and skipping source code. And
>>> .pyc files are an implementation detail of CPython so there  shouldn't be
>>> too much of a concern for other interpreters.
>>>
>&g

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> I don't think the optional existence of column number information needs a
different kind of pyc file.  Just a flag in a pyc file's header at most.
It isn't a new type of file.

That could work, but in my personal opinion, I would prefer not to do that
as it complicates things and I think is overkill.

On Sat, 8 May 2021 at 21:45, Gregory P. Smith  wrote:

>
> On Sat, May 8, 2021 at 1:32 PM Pablo Galindo Salgado 
> wrote:
>
>> > We can't piggy back on -OO as the only way to disable this, it needs
>> to have an option of its own.  -OO is unusable as code that relies on
>> "doc"strings as application data such as
>> http://www.dabeaz.com/ply/ply.html exists.
>>
>> -OO is the only sensible way to disable the data. There are two things to
>> disable:
>>
>
> nit: I wouldn't choose the word "sensible" given that -OO is already
> fundamentally unusable without knowing if any code in your entire
> transitive dependencies might depend on the presence of docstrings...
>
>
>>
>> * The data in pyc files
>> * Printing the exception highlighting
>>
>> Printing the exception highlighting can be disabled via combo of
>> environment variable / -X option but collecting the data can only be
>> disabled by -OO. The reason is that this will end in pyc files
>> so when the data is not there, a different kind of pyc files need to be
>> produced and I really don't want to have another set of pyc file extension
>> just to deactivate this. Notice that also a configure
>> time variable won't work because it will cause crashes when reading pyc
>> files produced by the interpreter compiled without the flag.
>>
>
> I don't think the optional existence of column number information needs a
> different kind of pyc file.  Just a flag in a pyc file's header at most.
> It isn't a new type of file.
>
>
>> On Sat, 8 May 2021 at 21:13, Gregory P. Smith  wrote:
>>
>>>
>>>
>>> On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado <
>>> pablog...@gmail.com> wrote:
>>>
>>>> Hi Brett,
>>>>
>>>> Just to be clear, .pyo files have not existed for a while:
>>>>> https://www.python.org/dev/peps/pep-0488/.
>>>>
>>>>
>>>> Whoops, my bad, I wanted to refer to the pyc files that are generated
>>>> with -OO, which have the "opt-2" prefix.
>>>>
>>>> This only kicks in at the -OO level.
>>>>
>>>>
>>>> I will correct the PEP so it reflex this more exactly.
>>>>
>>>> I personally prefer the idea of dropping the data with -OO since if
>>>>> you're stripping out docstrings you're already hurting introspection
>>>>> capabilities in the name of memory. Or one could go as far as to introduce
>>>>> -Os to do -OO plus dropping this extra data.
>>>>
>>>>
>>>> This is indeed the plan, sorry for the confusion. The opt-out mechanism
>>>> is using -OO, precisely as we are already dropping other data.
>>>>
>>>
>>> We can't piggy back on -OO as the only way to disable this, it needs to
>>> have an option of its own.  -OO is unusable as code that relies on
>>> "doc"strings as application data such as
>>> http://www.dabeaz.com/ply/ply.html exists.
>>>
>>> -gps
>>>
>>>
>>>>
>>>> Thanks for the clarifications!
>>>>
>>>>
>>>>
>>>> On Sat, 8 May 2021 at 19:41, Brett Cannon  wrote:
>>>>
>>>>>
>>>>>
>>>>> On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado <
>>>>> pablog...@gmail.com> wrote:
>>>>>
>>>>>> Although we were originally not sympathetic with it, we may need to
>>>>>> offer an opt-out mechanism for those users that care about the impact of
>>>>>> the overhead of the new data in pyc files
>>>>>> and in in-memory code objectsas was suggested by some folks (Thomas,
>>>>>> Yury, and others). For this, we could propose that the functionality will
>>>>>> be deactivated along with the extra
>>>>>> information when Python is executed in optimized mode (``python -O``)
>>>>>> and therefore pyo files will not have the overhead associated with the
>>>>>> extra required data.
>>>>>>
>>>>>
>>>>> Just to be cle

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> That could work, but in my personal opinion, I would prefer not to do
that as it complicates things and I think is overkill.

Let me expand on this:

I recognize the problem that -OO can be quite unusable if some of your
dependencies depend on docstrings and that It would be good to separate
this from that option, but I am afraid of the following:

- New APIs in the marshal module and other places to pass down the extra
information to read/write or not the extra information.
- Complication of the pyc format with more entries in the header.
- Complication of the implementation.

Given that the reasons to deactivate this option exist, but I expect them
to be very rare, I would prefer to maximize simplicity and maintainability.

On Sat, 8 May 2021 at 21:50, Pablo Galindo Salgado 
wrote:

> > I don't think the optional existence of column number information needs
> a different kind of pyc file.  Just a flag in a pyc file's header at most.
> It isn't a new type of file.
>
> That could work, but in my personal opinion, I would prefer not to do that
> as it complicates things and I think is overkill.
>
> On Sat, 8 May 2021 at 21:45, Gregory P. Smith  wrote:
>
>>
>> On Sat, May 8, 2021 at 1:32 PM Pablo Galindo Salgado 
>> wrote:
>>
>>> > We can't piggy back on -OO as the only way to disable this, it needs
>>> to have an option of its own.  -OO is unusable as code that relies on
>>> "doc"strings as application data such as
>>> http://www.dabeaz.com/ply/ply.html exists.
>>>
>>> -OO is the only sensible way to disable the data. There are two things
>>> to disable:
>>>
>>
>> nit: I wouldn't choose the word "sensible" given that -OO is already
>> fundamentally unusable without knowing if any code in your entire
>> transitive dependencies might depend on the presence of docstrings...
>>
>>
>>>
>>> * The data in pyc files
>>> * Printing the exception highlighting
>>>
>>> Printing the exception highlighting can be disabled via combo of
>>> environment variable / -X option but collecting the data can only be
>>> disabled by -OO. The reason is that this will end in pyc files
>>> so when the data is not there, a different kind of pyc files need to be
>>> produced and I really don't want to have another set of pyc file extension
>>> just to deactivate this. Notice that also a configure
>>> time variable won't work because it will cause crashes when reading pyc
>>> files produced by the interpreter compiled without the flag.
>>>
>>
>> I don't think the optional existence of column number information needs a
>> different kind of pyc file.  Just a flag in a pyc file's header at most.
>> It isn't a new type of file.
>>
>>
>>> On Sat, 8 May 2021 at 21:13, Gregory P. Smith  wrote:
>>>
>>>>
>>>>
>>>> On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado <
>>>> pablog...@gmail.com> wrote:
>>>>
>>>>> Hi Brett,
>>>>>
>>>>> Just to be clear, .pyo files have not existed for a while:
>>>>>> https://www.python.org/dev/peps/pep-0488/.
>>>>>
>>>>>
>>>>> Whoops, my bad, I wanted to refer to the pyc files that are generated
>>>>> with -OO, which have the "opt-2" prefix.
>>>>>
>>>>> This only kicks in at the -OO level.
>>>>>
>>>>>
>>>>> I will correct the PEP so it reflex this more exactly.
>>>>>
>>>>> I personally prefer the idea of dropping the data with -OO since if
>>>>>> you're stripping out docstrings you're already hurting introspection
>>>>>> capabilities in the name of memory. Or one could go as far as to 
>>>>>> introduce
>>>>>> -Os to do -OO plus dropping this extra data.
>>>>>
>>>>>
>>>>> This is indeed the plan, sorry for the confusion. The opt-out
>>>>> mechanism is using -OO, precisely as we are already dropping other data.
>>>>>
>>>>
>>>> We can't piggy back on -OO as the only way to disable this, it needs to
>>>> have an option of its own.  -OO is unusable as code that relies on
>>>> "doc"strings as application data such as
>>>> http://www.dabeaz.com/ply/ply.html exists.
>>>>
>>>> -gps
>>>>
>>>>
>>>>>
>>>>> Thank

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> Why not put in it -O instead?  Then -O means lose asserts and lose
fine-grained tracebacks, while -OO continues to also
strip out doc strings.

What if someone wants to keep asserts but do not want the extra data?

On Sat, 8 May 2021 at 22:05, Ethan Furman  wrote:

> On 5/8/21 1:31 PM, Pablo Galindo Salgado wrote:
>  >> We can't piggy back on -OO as the only way to disable this, it needs to
>  >> have an option of its own.  -OO is unusable as code that relies on
> "doc"
>  >> strings as application data such as http://www.dabeaz.com/ply/ply.html
>  >> exists.
>  >
>  > -OO is the only sensible way to disable the data. There are two things
> to disable:
>  >
>  > * The data in pyc files
>  > * Printing the exception highlighting
>
> Why not put in it -O instead?  Then -O means lose asserts and lose
> fine-grained tracebacks, while -OO continues to also
> strip out doc strings.
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BEE4BGUZHXBTVDPOW5R4DC3S463XC3EJ/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WK4KXZPOSWYMI3C5AILQCEYVZRCDFL7N/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> I don't think the optional existence of column number information needs a
different kind of pyc file.  Just a flag in a pyc file's header at most.
It isn't a new type of file.

Greg, what do you think if instead of not writing it to the pyc file with
-OO or adding a header entry to decide to read/write, we place None in the
field? That way we can
leverage the option that we intend to add to deactivate displaying the
traceback new information to reduce the data in the pyc files. The only
problem
is that there will be still a tiny bit of overhead: an extra object per
code object (None), but that's much much better than something that scales
with the
number of instructions :)

What's your opinion on this?


On Sat, 8 May 2021 at 21:45, Gregory P. Smith  wrote:

>
> On Sat, May 8, 2021 at 1:32 PM Pablo Galindo Salgado 
> wrote:
>
>> > We can't piggy back on -OO as the only way to disable this, it needs
>> to have an option of its own.  -OO is unusable as code that relies on
>> "doc"strings as application data such as
>> http://www.dabeaz.com/ply/ply.html exists.
>>
>> -OO is the only sensible way to disable the data. There are two things to
>> disable:
>>
>
> nit: I wouldn't choose the word "sensible" given that -OO is already
> fundamentally unusable without knowing if any code in your entire
> transitive dependencies might depend on the presence of docstrings...
>
>
>>
>> * The data in pyc files
>> * Printing the exception highlighting
>>
>> Printing the exception highlighting can be disabled via combo of
>> environment variable / -X option but collecting the data can only be
>> disabled by -OO. The reason is that this will end in pyc files
>> so when the data is not there, a different kind of pyc files need to be
>> produced and I really don't want to have another set of pyc file extension
>> just to deactivate this. Notice that also a configure
>> time variable won't work because it will cause crashes when reading pyc
>> files produced by the interpreter compiled without the flag.
>>
>
> I don't think the optional existence of column number information needs a
> different kind of pyc file.  Just a flag in a pyc file's header at most.
> It isn't a new type of file.
>
>
>> On Sat, 8 May 2021 at 21:13, Gregory P. Smith  wrote:
>>
>>>
>>>
>>> On Sat, May 8, 2021 at 11:58 AM Pablo Galindo Salgado <
>>> pablog...@gmail.com> wrote:
>>>
>>>> Hi Brett,
>>>>
>>>> Just to be clear, .pyo files have not existed for a while:
>>>>> https://www.python.org/dev/peps/pep-0488/.
>>>>
>>>>
>>>> Whoops, my bad, I wanted to refer to the pyc files that are generated
>>>> with -OO, which have the "opt-2" prefix.
>>>>
>>>> This only kicks in at the -OO level.
>>>>
>>>>
>>>> I will correct the PEP so it reflex this more exactly.
>>>>
>>>> I personally prefer the idea of dropping the data with -OO since if
>>>>> you're stripping out docstrings you're already hurting introspection
>>>>> capabilities in the name of memory. Or one could go as far as to introduce
>>>>> -Os to do -OO plus dropping this extra data.
>>>>
>>>>
>>>> This is indeed the plan, sorry for the confusion. The opt-out mechanism
>>>> is using -OO, precisely as we are already dropping other data.
>>>>
>>>
>>> We can't piggy back on -OO as the only way to disable this, it needs to
>>> have an option of its own.  -OO is unusable as code that relies on
>>> "doc"strings as application data such as
>>> http://www.dabeaz.com/ply/ply.html exists.
>>>
>>> -gps
>>>
>>>
>>>>
>>>> Thanks for the clarifications!
>>>>
>>>>
>>>>
>>>> On Sat, 8 May 2021 at 19:41, Brett Cannon  wrote:
>>>>
>>>>>
>>>>>
>>>>> On Fri, May 7, 2021 at 7:31 PM Pablo Galindo Salgado <
>>>>> pablog...@gmail.com> wrote:
>>>>>
>>>>>> Although we were originally not sympathetic with it, we may need to
>>>>>> offer an opt-out mechanism for those users that care about the impact of
>>>>>> the overhead of the new data in pyc files
>>>>>> and in in-memory code objectsas was suggested by some folks (Thomas,
>>>>>> Yury, and others). For thi

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Thanks Greg for the great, detailed response

I think I understand now better your proposal and I think is a good idea
and I would like to explore that. I have some questions:

* One problem I see is that that will make the constructor of the code
object depend on global options in the interpreter. Someone using the C-API
and passing down that attribute will be surprised to find that it was
modified by a global. I am not saying is bad but I can see some problems
with that.

* The alternative is to modify all calls to the code object constructor.
This is easy to do in the compiler because code objects are constructed
very close where the meta data is crated but is going to be a pain in other
places, because the code objects are constructed in places where we would
either need new APIs or to hide global state as the previous point.

* Another alternative is to walk the graph and strip the fields but that's
going to have a performance impact.

I think that if we decide to offer an opt out, this is actually one of the
best options but I am still slightly concerned about the extra complexity,
potential new APIs and maintainability.



On Sat, 8 May 2021, 22:55 Gregory P. Smith,  wrote:

>
>
> On Sat, May 8, 2021 at 2:09 PM Pablo Galindo Salgado 
> wrote:
>
>> > Why not put in it -O instead?  Then -O means lose asserts and lose
>> fine-grained tracebacks, while -OO continues to also
>> strip out doc strings.
>>
>> What if someone wants to keep asserts but do not want the extra data?
>>
>
> exactly my theme.  our existing -O and -OO already don't serve all user
> needs.  (I've witnessed people who need asserts but don't want docstrings
> wasting ram jump through hacky hoops to do that).  Complicating these
> options more by combining additional actions on them them doesn't help.
>
> The reason we have -O and -OO generate their own special opt-1 and opt-2
> pyc files is because both of those change the generated bytecode and
> overall flow of the program by omitting instructions and data.  code using
> those will run slightly faster as there are fewer instructions.
>
> The change we're talking about here doesn't do that.  It just adds
> additional metadata to whatever instructions are generated.  So it doesn't
> feel -O related.
>
> While some people aren't going to like the overhead, I'm happy not
> offering the choice.
>
> > Greg, what do you think if instead of not writing it to the pyc file
> with -OO or adding a header entry to decide to read/write, we place None in
> the field? That way we can
> > leverage the option that we intend to add to deactivate displaying the
> traceback new information to reduce the data in the pyc files. The only
> problem
> > is that there will be still a tiny bit of overhead: an extra object per
> code object (None), but that's much much better than something that scales
> with the
> > number of instructions :)
> >
> > What's your opinion on this?
>
> I don't understand the pyc structure enough to comment on how that works,
> but that sounds fine from a way to store less data if these are stored as a
> side table rather than intermingled with each instruction itself.  *If
> anyone even cares about storing less data.*  I would not activate
> generation of that in py_compile and compileall based on the -X flag to
> disable display of tracebacks though.  A flag changing a setting of the
> current runtime regarding traceback printing detail level should not change
> the metadata in pyc files it emits.  I realize -O and -OO behave this way,
> but I don't view those as a great example. We're not writing new uniquely
> named pyc files, I suggest making this an explicit option for py_compile
> and compileall if we're going to support generation of pyc files without
> column data at all.
>
> I'm unclear on what the specific goals are with all of these option
> possibilities.
>
> Who non-hypothetically cares about a 22% pyc file size increase?  I don't
> think we should be concerned.  I'm in favor of always writing them and the
> 20% size increase that results in.  If pyc size is an issue that should be
> its own separate enhancement PEP.  When it comes to pyc files there is more
> data we may want to store in the future for performance reasons - I don't
> see them shrinking without an independent effort.
>
> Caring about additional data retained in memory at runtime makes more
> sense to me as ram cost is much greater than storage cost and is paid
> repeatedly per process.  Storing an additional reference to None on code
> objects where a column information table is perfectly fine.  That can be a
> -X style interpreter startup option.  It isn&#

[Python-Dev] PEP 657 – Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hi,

We have prepared a PEP with our proposal for fine-grained error locations
in tracebacks. You can find the PEP here:

https://www.python.org/dev/peps/pep-0657/

The discussion is happening in the discourse server:

https://discuss.python.org/t/pep-657-include-fine-grained-error-locations-in-tracebacks/8629

To avoid splitting the discussion, *please redirect your comments there*
instead of replying to this thread.

Thanks!

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ALTWCBWXKBSW5TMQ33GYFXO4SB4TB4QY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Future PEP: Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

That is going to be very hard to read, unfortunately. Especially when the
line is
not simple. Highlighting the range is quite a fundamental part of the
proposal and
is driven by the great welcoming of highlighting ranges for syntax errors,
which many
users have reached to say that they find it extremely useful as a visual
feature.

Also, people with vision problems have mentioned how important having a
highlighting
section under the code to quickly understand the problem.

On Mon, 10 May 2021 at 11:46, Irit Katriel via Python-Dev <
python-dev@python.org> wrote:

>
> Another alternative is instead of
>
> File blah.py line 3:
> return x/0
>   ^^^
>
> to have
>
> File blah.py line 3 cols 12-14:
>   x/0
>
>
> On Mon, May 10, 2021 at 11:12 AM Steven D'Aprano 
> wrote:
>
>> On Mon, May 10, 2021 at 05:34:12AM -0400, Terry Reedy wrote:
>> > On 5/10/2021 3:28 AM, M.-A. Lemburg wrote:
>> >
>> > >I'm mostly thinking of tracebacks which go >10 levels deep, which is
>> > >rather common in larger applications. For those tracebacks, the top
>> > >entries are mostly noise you never look at when debugging. The proposal
>> > >now adds another 10 extra lines to jump over :-)
>> >
>> > If the slice were instead marked with color tagging, as I hope will be
>> > possible in IDLE and other IDEs, then no extra lines well be needed
>>
>> That's great for people using IDLE, but for those using the vanilla
>> Python interpreter, M-A.L makes a good point about increasing the
>> vertical size of the traceback which will almost always be ignored.
>>
>> Its especially the case for beginners. Its hard enough to get newbies to
>> read *any* of the traceback. Anything which increases the visual noise
>> of that is going to make it harder.
>>
>>
>> --
>> Steve
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/3ADVDPF4Z5DMXKG2CMJ3JTIN2SC76AUC/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YWNDAKU7RYBFWCRMXK3UZEYFE7NLUSNY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WTWY2SZOKY7Y3MKVCETB4ILIBK7BQNYK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Important: Python3.10 what's new

[Pablo Galindo Salgado]

Hi everyone,

Now that we are in the beta period for 3.10 is very important that we make
sure that all
improvements, new APIs and deprecations are reflected in the 3.10 what's
New document.

IMPORTANT!!

If you have worked on:

* Make a PEP that affects Python 3.10.
* A new feature/class/function/argument/option...
* Deprecating something.
* Remove something.
* Improve something important.

Please, make sure is reflected on the What's new document for Python 3.10.
Users normally don't
read the changelog directly and learn about what's available from the
What's New document so
you *absolutely* want your work to be reflected there. *Small* tutorials
and small code samples
and descriptions are a fantastic thing to add as well.

Specially for deprecations and removals, this is our public window to the
world so that learn what's
coming or how to port code to Python 3.10.

Thanks for helping to make Python3.10 a great release.

Regards from sunny London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DC37BKDIPECMRHZSXI3RSBTWE3TOKLDZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hi Mark,

Thanks for your useful feedback. Some comments:

> 1.  Errors spanning multiple lines.

That is addressed. Check the latest version of the PEP: we are propising
storing the end lines as well:

https://www.python.org/dev/peps/pep-0657/#specification


> 2. Repeated binary operations on the same line.

It has been suggested to add this on top of the previous information, but
we think this is problematic since these locations need to be
manually calculated in the compiler, while the ranges are derived from the
AST ranges. Given how error prone the old manual calculations of the AST
were, we really want to avoid manual calculations here. Additionally, many
users have mentioned that a single caret is difficult to read and that
motivates the ranges in SyntaxErrors that we introduced.
> 3. Tracking locations in the compiler. PEP 657 proposes that no
compression be used

No, PEP 657 doesn't specify the compression, which is different (check the
latest version). We say:

"The internal storage, compression and encoding of the information is left
as an implementation detail and can be changed at any point as long as the
public API remains unchanged."

> I think it would be better to provide an API like PEP 626 and not
restrict the internal format used.

Indeed, that is what we are doing. Check
https://www.python.org/dev/peps/pep-0657/#id10

Cheers,
Pablo Galindo Salgado


On Mon, 17 May 2021 at 14:17, Mark Shannon  wrote:

> Hi everyone,
>
> I fully agree with the rationale for the PEP, better error messages
> always help.
> However, I think the proposed implementation could be misleading in some
> cases and wastes memory.
>
>
> Use one position, not one-and-a-half positions.
> ---
>
> The main problem with PEP 657, IMO, is that it wants to present the
> location of an error as a pair of positions, but without the line number
> of the second position.
> Consequently it ends up with one and a half positions, not two.
> This does not work well for a number of reasons.
>
> 1.  Errors spanning multiple lines.
>
> Consider:
>
>  (i1 + i2 +
>   s1
>   )
>
> Where i1, i2 are integers and s1 is a string.
>
> With a single location, the error points to the second `+`. PEP 657
> would highlight the whole line, `i1 + i2 +`, making it unclear where the
> error occurred.
>
>
> 2. Repeated binary operations on the same line.
>
> A single location can also be clearer when all the code is on one line.
>
> i1 + i2 + s1
>
> PEP 657:
>
> i1 + i2 + s1
> 
>
> Using a single location:
>
> i1 + i2 + s1
>  ^
>
> 3. Tracking locations in the compiler.
>
> While nicer locations for errors is great, it won't be popular if it has
> a negative impact on performance.
> Locations need to tracked through the compiler. The simpler the
> location, the easier this is to do correctly without a negative
> performance impact.
> It is already tricky to do this correctly with just line numbers because
> we have both source that compiles to no bytecodes (try, pass) and
> bytecodes that have no source (implicit return None and except cleanups).
>
>
> A single location can still be presented as a whole token, as tokenizing
> a single line of code is easy and fast. So when presenting an error, the
> whole token can be highlighted.
>
> E.g:
>
> NameError
>  name
>  
>
> Compression
> ---
>
> PEP 657 proposes that no compression be used, but also mandates the use
> of a lossy compression scheme (truncating all offsets over 255).
> I think it would be better to provide an API like PEP 626 and not
> restrict the internal format used. In fact, extending or replacing the
> API of PEP 626 seems the best approach to me.
>
> I wouldn't worry about memory consumption.
> The code object layout and unmarshalling process needs to be
> re-implemented for reduced memory use and faster startup:
> https://github.com/markshannon/faster-cpython/blob/master/tiers.md#tier-0
> A few bytes more or less in the location table(s) is inconsequential.
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TKY4YMPAQZDKCK7NV4AQ3IFAN5MF76DU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

P.S. We will add "using a single caret" to the "rejected ideas section"
with some rationale.

On Mon, 17 May 2021, 14:28 Pablo Galindo Salgado, 
wrote:

> Hi Mark,
>
> Thanks for your useful feedback. Some comments:
>
> > 1.  Errors spanning multiple lines.
>
> That is addressed. Check the latest version of the PEP: we are propising
> storing the end lines as well:
>
> https://www.python.org/dev/peps/pep-0657/#specification
>
>
> > 2. Repeated binary operations on the same line.
>
> It has been suggested to add this on top of the previous information, but
> we think this is problematic since these locations need to be
> manually calculated in the compiler, while the ranges are derived from the
> AST ranges. Given how error prone the old manual calculations of the AST
> were, we really want to avoid manual calculations here. Additionally, many
> users have mentioned that a single caret is difficult to read and that
> motivates the ranges in SyntaxErrors that we introduced.
> > 3. Tracking locations in the compiler. PEP 657 proposes that no
> compression be used
>
> No, PEP 657 doesn't specify the compression, which is different (check the
> latest version). We say:
>
> "The internal storage, compression and encoding of the information is
> left as an implementation detail and can be changed at any point as long as
> the public API remains unchanged."
>
> > I think it would be better to provide an API like PEP 626 and not
> restrict the internal format used.
>
> Indeed, that is what we are doing. Check
> https://www.python.org/dev/peps/pep-0657/#id10
>
> Cheers,
> Pablo Galindo Salgado
>
>
> On Mon, 17 May 2021 at 14:17, Mark Shannon  wrote:
>
>> Hi everyone,
>>
>> I fully agree with the rationale for the PEP, better error messages
>> always help.
>> However, I think the proposed implementation could be misleading in some
>> cases and wastes memory.
>>
>>
>> Use one position, not one-and-a-half positions.
>> ---
>>
>> The main problem with PEP 657, IMO, is that it wants to present the
>> location of an error as a pair of positions, but without the line number
>> of the second position.
>> Consequently it ends up with one and a half positions, not two.
>> This does not work well for a number of reasons.
>>
>> 1.  Errors spanning multiple lines.
>>
>> Consider:
>>
>>  (i1 + i2 +
>>   s1
>>   )
>>
>> Where i1, i2 are integers and s1 is a string.
>>
>> With a single location, the error points to the second `+`. PEP 657
>> would highlight the whole line, `i1 + i2 +`, making it unclear where the
>> error occurred.
>>
>>
>> 2. Repeated binary operations on the same line.
>>
>> A single location can also be clearer when all the code is on one line.
>>
>> i1 + i2 + s1
>>
>> PEP 657:
>>
>> i1 + i2 + s1
>> 
>>
>> Using a single location:
>>
>> i1 + i2 + s1
>>  ^
>>
>> 3. Tracking locations in the compiler.
>>
>> While nicer locations for errors is great, it won't be popular if it has
>> a negative impact on performance.
>> Locations need to tracked through the compiler. The simpler the
>> location, the easier this is to do correctly without a negative
>> performance impact.
>> It is already tricky to do this correctly with just line numbers because
>> we have both source that compiles to no bytecodes (try, pass) and
>> bytecodes that have no source (implicit return None and except cleanups).
>>
>>
>> A single location can still be presented as a whole token, as tokenizing
>> a single line of code is easy and fast. So when presenting an error, the
>> whole token can be highlighted.
>>
>> E.g:
>>
>> NameError
>>  name
>>  
>>
>> Compression
>> ---
>>
>> PEP 657 proposes that no compression be used, but also mandates the use
>> of a lossy compression scheme (truncating all offsets over 255).
>> I think it would be better to provide an API like PEP 626 and not
>> restrict the internal format used. In fact, extending or replacing the
>> API of PEP 626 seems the best approach to me.
>>
>> I wouldn't worry about memory consumption.
>> The code object layout and unmarshalling process needs to be
>> re-implemented for reduced memory use and faster startup:
>> https://github.com/markshannon/faster-cpython/blob/master/tiers.md#tier-0
>> A few bytes more or

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hi Nathaniel,

Thanks a lot for your suggestion! I like the idea although I still think is
more complex than our current proposal, but on the other hand it allows for
a much richer results so I'm quite excited to try it out. We are going to
give it a go to explore it with a prototype and if we are convinced we will
update the PEP.

One thing I'm not a fan of is that tools that want to leverage this
information (notice our pep also offers this info for tools as an important
aspect of it) will need to reparse the file AND search the AST node, which
also involves transforming the full tree to Python objects, which is even
slower. This is unfortunately a worse API than just get the full location
given an instruction offset. Also, while displaying tracebacks may be a
good scenario for the speed tradeoff, it may not be for other tools.
Finally, if there is no file available all this information is lost,
although one could argue that then is not extremely useful...

Regards from sunny London,
Pablo Galindo Salgado

On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:

> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
> > 2. Repeated binary operations on the same line.
> >
> > A single location can also be clearer when all the code is on one line.
> >
> > i1 + i2 + s1
> >
> > PEP 657:
> >
> > i1 + i2 + s1
> > 
> >
> > Using a single location:
> >
> > i1 + i2 + s1
> >  ^
>
> It's true this case is a bit confusing with the whole operation span
> highlighted, but I'm not sure the single location version is much better. I
> feel like a Really Good UI would like, highlight the two operands in
> different colors or something, or at least underline the two separate items
> whose type is incompatible separately:
>
> TypeError: unsupported operand type(s) for +: 'int' + 'str':
> i1 + i2 + s1
> ^^^   ~~
>
> More generally, these error messages are the kind of thing where the UI
> can always be tweaked to improve further, and those tweaks can make good
> use of any rich source information that's available.
>
> So, here's another option to consider:
>
> - When parsing, assign each AST node a unique, deterministic id (e.g.
> sequentially across the AST tree from top-to-bottom, left-to-right).
> - For each bytecode offset, store the corresponding AST node id in an
> lnotab-like table
> - When displaying a traceback, we already need to go find and read the
> original .py file to print source code at all. Re-parse it, and use the ids
> to find the original AST node, in context with full structure. Let the
> traceback formatter do whatever clever stuff it wants with this info.
>
> Of course if the .py and .pyc files don't match, this might produce
> gibberish. We already have that problem with showing source lines, but it
> might be even more confusing if we get some random unrelated AST node. This
> could be avoided by storing some kind of hash in the code object, so that
> we can validate the .py file we find hasn't changed (sha512 if we're
> feeling fancy, crc32 if we want to save space, either way is probably fine).
>
> This would make traceback printing more expensive, but only if you want
> the fancy features, and traceback printing is already expensive (it does
> file I/O!). Usually by the time you're rendering a traceback it's more
> important to optimize for human time than CPU time. It would take less
> memory than PEP 657, and the same as Mark's proposal (both only track one
> extra integer per bytecode offset). And it would allow for arbitrarily rich
> traceback display.
>
> (I guess in theory you could make this even cheaper by using it to replace
> lnotab, instead of extending it. But I think keeping lnotab around is a
> good idea, as a fallback for cases where you can't find the original source
> but still want some hint at location information.)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BUXFOSAEBXLIHH432PKBCXOGXUAHQIVP/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SZPCHKVI32YI6EMRLLHKGTAU6M6VAKJZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Hu, actually another problem of this approach:

Nodes are created and modified after the optimization pass, so the AST
produced by the parser is not enough to reconstruct the actual
information, we need to also run the optimization passes, but
unfortunately, this is (by design) not don't in the Python API (to preserve
all possible information about the original code), so this complicates
quite a lot the API to get this, as `ast.parse` is not enough to get the
original tree.

On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
wrote:

> Hi Nathaniel,
>
> Thanks a lot for your suggestion! I like the idea although I still think
> is more complex than our current proposal, but on the other hand it allows
> for a much richer results so I'm quite excited to try it out. We are going
> to give it a go to explore it with a prototype and if we are convinced we
> will update the PEP.
>
> One thing I'm not a fan of is that tools that want to leverage this
> information (notice our pep also offers this info for tools as an important
> aspect of it) will need to reparse the file AND search the AST node, which
> also involves transforming the full tree to Python objects, which is even
> slower. This is unfortunately a worse API than just get the full location
> given an instruction offset. Also, while displaying tracebacks may be a
> good scenario for the speed tradeoff, it may not be for other tools.
> Finally, if there is no file available all this information is lost,
> although one could argue that then is not extremely useful...
>
> Regards from sunny London,
> Pablo Galindo Salgado
>
> On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:
>
>> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
>> > 2. Repeated binary operations on the same line.
>> >
>> > A single location can also be clearer when all the code is on one line.
>> >
>> > i1 + i2 + s1
>> >
>> > PEP 657:
>> >
>> > i1 + i2 + s1
>> > 
>> >
>> > Using a single location:
>> >
>> > i1 + i2 + s1
>> >  ^
>>
>> It's true this case is a bit confusing with the whole operation span
>> highlighted, but I'm not sure the single location version is much better. I
>> feel like a Really Good UI would like, highlight the two operands in
>> different colors or something, or at least underline the two separate items
>> whose type is incompatible separately:
>>
>> TypeError: unsupported operand type(s) for +: 'int' + 'str':
>> i1 + i2 + s1
>> ^^^   ~~
>>
>> More generally, these error messages are the kind of thing where the UI
>> can always be tweaked to improve further, and those tweaks can make good
>> use of any rich source information that's available.
>>
>> So, here's another option to consider:
>>
>> - When parsing, assign each AST node a unique, deterministic id (e.g.
>> sequentially across the AST tree from top-to-bottom, left-to-right).
>> - For each bytecode offset, store the corresponding AST node id in an
>> lnotab-like table
>> - When displaying a traceback, we already need to go find and read the
>> original .py file to print source code at all. Re-parse it, and use the ids
>> to find the original AST node, in context with full structure. Let the
>> traceback formatter do whatever clever stuff it wants with this info.
>>
>> Of course if the .py and .pyc files don't match, this might produce
>> gibberish. We already have that problem with showing source lines, but it
>> might be even more confusing if we get some random unrelated AST node. This
>> could be avoided by storing some kind of hash in the code object, so that
>> we can validate the .py file we find hasn't changed (sha512 if we're
>> feeling fancy, crc32 if we want to save space, either way is probably fine).
>>
>> This would make traceback printing more expensive, but only if you want
>> the fancy features, and traceback printing is already expensive (it does
>> file I/O!). Usually by the time you're rendering a traceback it's more
>> important to optimize for human time than CPU time. It would take less
>> memory than PEP 657, and the same as Mark's proposal (both only track one
>> extra integer per bytecode offset). And it would allow for arbitrarily rich
>> traceback display.
>>
>> (I guess in theory you could make this even cheaper by using it to
>> replace lnotab, instead of extending it. But I think keeping lnotab around
>> is a good idea, as a fallback for cases where you ca

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Yet another problem that I found:

One integer is actually not enough to assign IDs. One unsigned integer can
cover 4,294,967,295 AST nodes, but is technically possible
to have more than that in a single file. While in PEP 657 we are tracking
offsets that are normally very low < 100 typically or end lines
that are easily compressible (as end lines are normally equal to the start
of the line), node IDs can be arbitrarily big. Which is worse:
the parser creates some AST nodes that throw away if the parser fails
(that's how PEG parsers work), so there will be a lot of IDs that
don't get assigned (and the parser doesn't have an easy way to know how
many IDs has thrown away). This forces us to either use something
bigger than an integer (probably a size_t) or to deal with overflow.

On Tue, 18 May 2021 at 10:23, Pablo Galindo Salgado 
wrote:

> Hu, actually another problem of this approach:
>
> Nodes are created and modified after the optimization pass, so the AST
> produced by the parser is not enough to reconstruct the actual
> information, we need to also run the optimization passes, but
> unfortunately, this is (by design) not don't in the Python API (to preserve
> all possible information about the original code), so this complicates
> quite a lot the API to get this, as `ast.parse` is not enough to get the
> original tree.
>
> On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
> wrote:
>
>> Hi Nathaniel,
>>
>> Thanks a lot for your suggestion! I like the idea although I still think
>> is more complex than our current proposal, but on the other hand it allows
>> for a much richer results so I'm quite excited to try it out. We are going
>> to give it a go to explore it with a prototype and if we are convinced we
>> will update the PEP.
>>
>> One thing I'm not a fan of is that tools that want to leverage this
>> information (notice our pep also offers this info for tools as an important
>> aspect of it) will need to reparse the file AND search the AST node, which
>> also involves transforming the full tree to Python objects, which is even
>> slower. This is unfortunately a worse API than just get the full location
>> given an instruction offset. Also, while displaying tracebacks may be a
>> good scenario for the speed tradeoff, it may not be for other tools.
>> Finally, if there is no file available all this information is lost,
>> although one could argue that then is not extremely useful...
>>
>> Regards from sunny London,
>> Pablo Galindo Salgado
>>
>> On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:
>>
>>> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
>>> > 2. Repeated binary operations on the same line.
>>> >
>>> > A single location can also be clearer when all the code is on one line.
>>> >
>>> > i1 + i2 + s1
>>> >
>>> > PEP 657:
>>> >
>>> > i1 + i2 + s1
>>> > 
>>> >
>>> > Using a single location:
>>> >
>>> > i1 + i2 + s1
>>> >  ^
>>>
>>> It's true this case is a bit confusing with the whole operation span
>>> highlighted, but I'm not sure the single location version is much better. I
>>> feel like a Really Good UI would like, highlight the two operands in
>>> different colors or something, or at least underline the two separate items
>>> whose type is incompatible separately:
>>>
>>> TypeError: unsupported operand type(s) for +: 'int' + 'str':
>>> i1 + i2 + s1
>>> ^^^   ~~
>>>
>>> More generally, these error messages are the kind of thing where the UI
>>> can always be tweaked to improve further, and those tweaks can make good
>>> use of any rich source information that's available.
>>>
>>> So, here's another option to consider:
>>>
>>> - When parsing, assign each AST node a unique, deterministic id (e.g.
>>> sequentially across the AST tree from top-to-bottom, left-to-right).
>>> - For each bytecode offset, store the corresponding AST node id in an
>>> lnotab-like table
>>> - When displaying a traceback, we already need to go find and read the
>>> original .py file to print source code at all. Re-parse it, and use the ids
>>> to find the original AST node, in context with full structure. Let the
>>> traceback formatter do whatever clever stuff it wants with this info.
>>>
>>> Of course if the .py and .pyc files don't match, this might produce
>>> gibberish. We already have th

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> One integer is actually not enough to assign IDs.

Actually, disregard this particular problem. I think that we could
perfectly stop assigning IDs if we reach the overflow limit and call it a
day
since you need to have a truly horrendous file to reach 4,294,967,295 AST nodes
(I did some tests to check this).

On Tue, 18 May 2021 at 13:25, Pablo Galindo Salgado 
wrote:

> Yet another problem that I found:
>
> One integer is actually not enough to assign IDs. One unsigned integer can
> cover 4,294,967,295 AST nodes, but is technically possible
> to have more than that in a single file. While in PEP 657 we are tracking
> offsets that are normally very low < 100 typically or end lines
> that are easily compressible (as end lines are normally equal to the start
> of the line), node IDs can be arbitrarily big. Which is worse:
> the parser creates some AST nodes that throw away if the parser fails
> (that's how PEG parsers work), so there will be a lot of IDs that
> don't get assigned (and the parser doesn't have an easy way to know how
> many IDs has thrown away). This forces us to either use something
> bigger than an integer (probably a size_t) or to deal with overflow.
>
> On Tue, 18 May 2021 at 10:23, Pablo Galindo Salgado 
> wrote:
>
>> Hu, actually another problem of this approach:
>>
>> Nodes are created and modified after the optimization pass, so the AST
>> produced by the parser is not enough to reconstruct the actual
>> information, we need to also run the optimization passes, but
>> unfortunately, this is (by design) not don't in the Python API (to preserve
>> all possible information about the original code), so this complicates
>> quite a lot the API to get this, as `ast.parse` is not enough to get the
>> original tree.
>>
>> On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
>> wrote:
>>
>>> Hi Nathaniel,
>>>
>>> Thanks a lot for your suggestion! I like the idea although I still think
>>> is more complex than our current proposal, but on the other hand it allows
>>> for a much richer results so I'm quite excited to try it out. We are going
>>> to give it a go to explore it with a prototype and if we are convinced we
>>> will update the PEP.
>>>
>>> One thing I'm not a fan of is that tools that want to leverage this
>>> information (notice our pep also offers this info for tools as an important
>>> aspect of it) will need to reparse the file AND search the AST node, which
>>> also involves transforming the full tree to Python objects, which is even
>>> slower. This is unfortunately a worse API than just get the full location
>>> given an instruction offset. Also, while displaying tracebacks may be a
>>> good scenario for the speed tradeoff, it may not be for other tools.
>>> Finally, if there is no file available all this information is lost,
>>> although one could argue that then is not extremely useful...
>>>
>>> Regards from sunny London,
>>> Pablo Galindo Salgado
>>>
>>> On Tue, 18 May 2021, 01:43 Nathaniel Smith,  wrote:
>>>
>>>> On Mon, May 17, 2021 at 6:18 AM Mark Shannon  wrote:
>>>> > 2. Repeated binary operations on the same line.
>>>> >
>>>> > A single location can also be clearer when all the code is on one
>>>> line.
>>>> >
>>>> > i1 + i2 + s1
>>>> >
>>>> > PEP 657:
>>>> >
>>>> > i1 + i2 + s1
>>>> > 
>>>> >
>>>> > Using a single location:
>>>> >
>>>> > i1 + i2 + s1
>>>> >  ^
>>>>
>>>> It's true this case is a bit confusing with the whole operation span
>>>> highlighted, but I'm not sure the single location version is much better. I
>>>> feel like a Really Good UI would like, highlight the two operands in
>>>> different colors or something, or at least underline the two separate items
>>>> whose type is incompatible separately:
>>>>
>>>> TypeError: unsupported operand type(s) for +: 'int' + 'str':
>>>> i1 + i2 + s1
>>>> ^^^   ~~
>>>>
>>>> More generally, these error messages are the kind of thing where the UI
>>>> can always be tweaked to improve further, and those tweaks can make good
>>>> use of any rich source information that's available.
>>>>
>>>> So, here's another option to consider:
>>>>
>&g

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

Ok, we have implemented a rough prototype and we have decided not to go
with this for the following reasons:

* Is almost the same storage cost as the solution we already have. Since
storing the node id cost 4 bytes (an integer) per instruction
and our solution needs 2+ bytes per instruction for the offsets (normally
just 2 as offsets are generally < 256) + the compressed end line,
which is very small since is almost always the same as the start line.
* It actually doesn't have more advantages. The current solution in the PEP
can do exactly the same as this solution if you allow reparsing when
displaying tracebacks. This is because with the start line, end line, start
offset and end offset and the original file, you can extract the source that
is associated with the instruction, parse it (and this
is much faster because you just need to parse the tiny fragment) and then
you get an AST node that you can use for whatever you want.
* The proposed solution forces to reparse several times entire files just
to extract a single AST node. Even worse: for frames in the same file it
forces
to reparse those again and again unless you complicate the implementation
by adding AST caching. But even that it won't be free as it will incur in
quite a lot of extra memory and this is problematic, especially when
displaying exceptions as memory can be low, which the current design takes
into account.
* Nodes are created and modified after the optimization pass, so the AST
produced by the parser is not enough to reconstruct the actual
information, we need to also run the optimization passes, but
unfortunately, this is (by design) not don't in the Python API (to preserve
all possible information about the original code), so this complicates
quite a lot the API to get this, as `ast.parse` is not enough to get the
original tree.
* The implementation is quite more complex then the one the PEP has since
to do it right implies having to hash the source files, implement node id
propagation in the parser and a node visitor to find the correct node +
reparsing in the traceback.
* It makes the AST (both the C one and the Python one) bigger, which
increases the memory costs of parsing and very slightly affects performance
(we measured a 4% decrease in perf to add the new field).
* It makes the API for 3rd party tools very non-straightforward, forcing
reparsing and finding the AST nodes. Even if we provide
some new functionality in the ast module or similar to make this easier, it
quite a lot of overhead just to get the position information.

Even if is possible to solve many of these problems, we think the
complexity is not worth it, especially since it actually doesn't give more
functionality.

On Tue, 18 May 2021 at 13:47, Pablo Galindo Salgado 
wrote:

> > One integer is actually not enough to assign IDs.
>
> Actually, disregard this particular problem. I think that we could
> perfectly stop assigning IDs if we reach the overflow limit and call it a
> day
> since you need to have a truly horrendous file to reach 4,294,967,295 AST 
> nodes
> (I did some tests to check this).
>
> On Tue, 18 May 2021 at 13:25, Pablo Galindo Salgado 
> wrote:
>
>> Yet another problem that I found:
>>
>> One integer is actually not enough to assign IDs. One unsigned integer
>> can cover 4,294,967,295 AST nodes, but is technically possible
>> to have more than that in a single file. While in PEP 657 we are tracking
>> offsets that are normally very low < 100 typically or end lines
>> that are easily compressible (as end lines are normally equal to the
>> start of the line), node IDs can be arbitrarily big. Which is worse:
>> the parser creates some AST nodes that throw away if the parser fails
>> (that's how PEG parsers work), so there will be a lot of IDs that
>> don't get assigned (and the parser doesn't have an easy way to know how
>> many IDs has thrown away). This forces us to either use something
>> bigger than an integer (probably a size_t) or to deal with overflow.
>>
>> On Tue, 18 May 2021 at 10:23, Pablo Galindo Salgado 
>> wrote:
>>
>>> Hu, actually another problem of this approach:
>>>
>>> Nodes are created and modified after the optimization pass, so the AST
>>> produced by the parser is not enough to reconstruct the actual
>>> information, we need to also run the optimization passes, but
>>> unfortunately, this is (by design) not don't in the Python API (to preserve
>>> all possible information about the original code), so this complicates
>>> quite a lot the API to get this, as `ast.parse` is not enough to get the
>>> original tree.
>>>
>>> On Tue, 18 May 2021 at 08:53, Pablo Galindo Salgado 
>>> wrote:
>>>
>>>> Hi Nat

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

>
> Excellent point! Do you know how reliable this is in practice, i.e.
> what proportion of bytecode source spans are something you can
> successfully pass to ast.parse? If it works it's obviously nicer, but
> I can't tell how often it works. E.g. anything including
> return/break/continue/yield/await will fail, since those require an
> enclosing context to be legal. I doubt return/break/continue will
> raise exceptions often, but yield/await do all the time.


All those limitations are compiler-time limitations because they imply
scoping. A valid AST is any piece of a converted parse tree, or a piece
of the PEG sub grammar:

>>> ast.dump(ast.parse("yield"))
'Module(body=[Expr(value=Yield())], type_ignores=[])'
>>> ast.dump(ast.parse("return"))
'Module(body=[Return()], type_ignores=[])'
>>> ast.dump(ast.parse("continue"))
'Module(body=[Continue()], type_ignores=[])'
>>> ast.dump(ast.parse("await x"))
"Module(body=[Expr(value=Await(value=Name(id='x', ctx=Load(],
type_ignores=[])"

On Thu, 20 May 2021 at 03:22, Nathaniel Smith  wrote:

> On Tue, May 18, 2021 at 2:49 PM Pablo Galindo Salgado
>  wrote:
> > * It actually doesn't have more advantages. The current solution in the
> PEP can do exactly the same as this solution if you allow reparsing when
> > displaying tracebacks. This is because with the start line, end line,
> start offset and end offset and the original file, you can extract the
> source that
> > is associated with the instruction, parse it (and this
> > is much faster because you just need to parse the tiny fragment) and
> then you get an AST node that you can use for whatever you want.
>
> Excellent point! Do you know how reliable this is in practice, i.e.
> what proportion of bytecode source spans are something you can
> successfully pass to ast.parse? If it works it's obviously nicer, but
> I can't tell how often it works. E.g. anything including
> return/break/continue/yield/await will fail, since those require an
> enclosing context to be legal. I doubt return/break/continue will
> raise exceptions often, but yield/await do all the time.
>
> You could kluge it by wrapping the source span in a dummy 'async def'
> before parsing, since that makes yield/await legal, but OTOH it makes
> 'yield from' and 'from X import *' illegal.
>
> I guess you could have a helper that attempts passing the string to
> ast.parse, and if that fails tries wrapping it in a loop/sync
> def/async def/etc. until one of them succeeds. Maybe that would be a
> useful utility to add to the traceback module?
>
> Or add a PyCF_YOLO flag that tries to make sense of an arbitrary
> out-of-context string.
>
> (Are there any other bits of syntax that require specific contexts
> that I'm not thinking of? If __enter__/__exit__ raise an exception,
> then what's the corresponding span? The entire 'with' block, or just
> the 'with' line itself?)
>
> -n
>
> PS: this is completely orthogonal to PEP 657, but if you're excited
> about making tracebacks more readable, another piece of low-hanging
> fruit would be to print method __qualname__s instead of __name__s in
> the traceback output. The reason we don't do that now is that
> __qualname__ lives on the function object, but in a traceback, we
> can't get the function object. The traceback only has access to the
> code object, and the code object doesn't have __qualname__, just
> __name__. Probably the cleanest way to do this would be to make the
> traceback or code object have a pointer back to the function object.
> See also https://bugs.python.org/issue12857.
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VLTCIVKY3PBZC6LAMQ7EFZBO53KSGWYD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657 -- Include Fine Grained Error Locations in Tracebacks

[Pablo Galindo Salgado]

> Hopefully if we add more pseudokeywords in the future it won't break
the ability to parse traceback spans.

It won't because soft keywords are now handled natively with the peg
parser (as "match" and "case" now) instead of hacked into the tokenizer :)


On Fri, 21 May 2021 at 01:55, Nathaniel Smith  wrote:

> On Wed, May 19, 2021 at 7:28 PM Pablo Galindo Salgado
>  wrote:
> >>
> >> Excellent point! Do you know how reliable this is in practice, i.e.
> >> what proportion of bytecode source spans are something you can
> >> successfully pass to ast.parse? If it works it's obviously nicer, but
> >> I can't tell how often it works. E.g. anything including
> >> return/break/continue/yield/await will fail, since those require an
> >> enclosing context to be legal. I doubt return/break/continue will
> >> raise exceptions often, but yield/await do all the time.
> >
> >
> > All those limitations are compiler-time limitations because they imply
> > scoping. A valid AST is any piece of a converted parse tree, or a piece
> > of the PEG sub grammar:
> >
> > >>> ast.dump(ast.parse("yield"))
> > 'Module(body=[Expr(value=Yield())], type_ignores=[])'
> > >>> ast.dump(ast.parse("return"))
> > 'Module(body=[Return()], type_ignores=[])'
> > >>> ast.dump(ast.parse("continue"))
> > 'Module(body=[Continue()], type_ignores=[])'
> > >>> ast.dump(ast.parse("await x"))
> > "Module(body=[Expr(value=Await(value=Name(id='x', ctx=Load(],
> type_ignores=[])"
>
> Ah, nice! I guess I was confused by memories of the behavior in 3.6
> and earlier, where 'await' was a pseudokeyword:
>
> ❯ docker run -it --rm python:3.6-alpine
> >>> import ast
> >>> ast.parse("await f()")
> SyntaxError: invalid syntax
>
> Hopefully if we add more pseudokeywords in the future it won't break
> the ability to parse traceback spans.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RQ65HRD47SR73B256EUXPUCSVY65USDV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] IMPORTANT: Python 3.10b2 release blockers

[Pablo Galindo Salgado]

Hi,

Tomorrow is the scheduled release of Python 3.10 beta 2 but unfortunately
we have several release blockers:

https://bugs.python.org/issue41282: Deprecate and remove distutils
https://bugs.python.org/issue40222: "Zero cost" exception handling
https://bugs.python.org/issue42972: [C API] Heap types (PyType_FromSpec)
must fully implement the GC protocol
https://bugs.python.org/issue44043: 3.10 b1 armhf Bus Error in hashlib
test: test_gil

We also have the address sanitizer buildbot failing :

https://buildbot.python.org/all/#/builders/582/builds/165/steps/5/logs/stdio

and some segmentation faults on the fedora stable buildbot:

https://buildbot.python.org/all/#/builders/543/builds/190

Some of these issues have PRs but some of them have not. Please, if you are
involved or you maintain one of the
areas involved in these issues, take a look at them and act with one of the
following:

* Fix the issue making a PR
* Review an existing PR and / or merge it
* If you intend to mark it as a deferred blocker, please provide a
rationale and contact me first by pinging me in the issue.

Until these issues are fixed or deferred, the release team will not be able
to make a new beta release.

Thanks for your help,

Regards from stormy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FHFI7QKWNHAVWVFTCHJGTYD3ZFVEUXDD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: IMPORTANT: Python 3.10b2 release blockers

[Pablo Galindo Salgado]

Small correction:

https://bugs.python.org/issue40222: "Zero cost" exception handling

and the segfaults on these buildbots:

https://buildbot.python.org/all/#/builders/582/builds/165/steps/5/logs/stdio
https://buildbot.python.org/all/#/builders/543/builds/190

are 3-11 (main branch) only, but they are also quite important to get fixed
as soon as possible,
so the buildbot failures don't pile up.

On the other hand, seems that there is a nasty race condition on
test_asyncio and many of the refleak
builders for 3.10 hang, rendering them not usefull:

https://buildbot.python.org/all/#/builders/693/builds/21
https://buildbot.python.org/all/#/builders/677/builds/22
https://buildbot.python.org/all/#/builders/669/builds/22
...

You can access the release dashboard for the buildbots here:

https://buildbot.python.org/all/#/release_status

On Mon, 24 May 2021 at 23:45, Pablo Galindo Salgado 
wrote:

> Hi,
>
> Tomorrow is the scheduled release of Python 3.10 beta 2 but unfortunately
> we have several release blockers:
>
> https://bugs.python.org/issue41282: Deprecate and remove distutils
> https://bugs.python.org/issue40222: "Zero cost" exception handling
> https://bugs.python.org/issue42972: [C API] Heap types (PyType_FromSpec)
> must fully implement the GC protocol
> https://bugs.python.org/issue44043: 3.10 b1 armhf Bus Error in hashlib
> test: test_gil
>
> We also have the address sanitizer buildbot failing :
>
>
> https://buildbot.python.org/all/#/builders/582/builds/165/steps/5/logs/stdio
>
> and some segmentation faults on the fedora stable buildbot:
>
> https://buildbot.python.org/all/#/builders/543/builds/190
>
> Some of these issues have PRs but some of them have not. Please, if you
> are involved or you maintain one of the
> areas involved in these issues, take a look at them and act with one of
> the following:
>
> * Fix the issue making a PR
> * Review an existing PR and / or merge it
> * If you intend to mark it as a deferred blocker, please provide a
> rationale and contact me first by pinging me in the issue.
>
> Until these issues are fixed or deferred, the release team will not be
> able to make a new beta release.
>
> Thanks for your help,
>
> Regards from stormy London,
> Pablo Galindo Salgado
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GBD57CUUU4K5NMQDTEZXNCX76YISEIGQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Difficulty of testing beta releases now available

[Pablo Galindo Salgado]

> - Cython doesn't work because of _PyGen_Send change [1]

I think this is fixed on the latest cython (0.29.23)


On Wed, 26 May 2021 at 01:05, Neil Schemenauer 
wrote:

> On 2021-05-04, Łukasz Langa wrote:
> > We strongly encourage maintainers of third-party Python projects
> > to test with 3.10 during the beta phase and report issues found to
> > the Python bug tracker  as soon as
> > possible.
>
> Testing with Python 3.10b1 is not easy, at least for me.  Here is a
> list of initial problems I ran into, from memory:
>
> - Cython doesn't work because of _PyGen_Send change [1]
>
> - scipy cannot be installed because it has requires_python =
>   ">=3.7,<3.10".  If you manually install from source, it seems to
>   work.
>
> - numpy cannot be installed because of _Py_HashDouble() change [2]
>
> - trio cannot be used because of TracebackException.__init__ changes [3]
>
> For the above problems, I would suggest the 3rd party package has
> the issue and it's not a problem with the Python release.  However,
> I guess that few people are using Python without 3rd party packages.
> So, it seems unsurprising that beta and RC releases are not well
> tested.  It has taken me quite a few hours to get a working version
> of Python 3.10 with all required dependancies such that I can run
> unit tests for some application code.
>
> Can we do any things to improve the situation?  Perhaps using the
> pre-release functionality of PyPI would help.  We would have to
> somehow encourage 3rd party packages to upload pre-releases that are
> compatible with our beta/RC releases.
>
>
> 1. https://github.com/cython/cython/issues/3876
> 2. https://github.com/numpy/numpy/issues/19033
> 3. https://github.com/python-trio/trio/issues/1899
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2Z23GRYT5VDHR4YJKS6YIQG7G46SC27T/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZJPJRK4OYCA7EOIMYRW4U5N6YPJVKVXF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: IMPORTANT: Python 3.10b2 release blockers

[Pablo Galindo Salgado]

Hi,

Friendly reminder that the Python3.10 beta 2 is still blocked on:

https://bugs.python.org/issue42972

Thanks for your help,

Regards from stormy London,
Pablo Galindo Salgado

On Mon, 24 May 2021 at 23:54, Pablo Galindo Salgado 
wrote:

> Small correction:
>
> https://bugs.python.org/issue40222: "Zero cost" exception handling
>
> and the segfaults on these buildbots:
>
>
> https://buildbot.python.org/all/#/builders/582/builds/165/steps/5/logs/stdio
> https://buildbot.python.org/all/#/builders/543/builds/190
>
> are 3-11 (main branch) only, but they are also quite important to get
> fixed as soon as possible,
> so the buildbot failures don't pile up.
>
> On the other hand, seems that there is a nasty race condition on
> test_asyncio and many of the refleak
> builders for 3.10 hang, rendering them not usefull:
>
> https://buildbot.python.org/all/#/builders/693/builds/21
> https://buildbot.python.org/all/#/builders/677/builds/22
> https://buildbot.python.org/all/#/builders/669/builds/22
> ...
>
> You can access the release dashboard for the buildbots here:
>
> https://buildbot.python.org/all/#/release_status
>
> On Mon, 24 May 2021 at 23:45, Pablo Galindo Salgado 
> wrote:
>
>> Hi,
>>
>> Tomorrow is the scheduled release of Python 3.10 beta 2 but unfortunately
>> we have several release blockers:
>>
>> https://bugs.python.org/issue41282: Deprecate and remove distutils
>> https://bugs.python.org/issue40222: "Zero cost" exception handling
>> https://bugs.python.org/issue42972: [C API] Heap types (PyType_FromSpec)
>> must fully implement the GC protocol
>> https://bugs.python.org/issue44043: 3.10 b1 armhf Bus Error in hashlib
>> test: test_gil
>>
>> We also have the address sanitizer buildbot failing :
>>
>>
>> https://buildbot.python.org/all/#/builders/582/builds/165/steps/5/logs/stdio
>>
>> and some segmentation faults on the fedora stable buildbot:
>>
>> https://buildbot.python.org/all/#/builders/543/builds/190
>>
>> Some of these issues have PRs but some of them have not. Please, if you
>> are involved or you maintain one of the
>> areas involved in these issues, take a look at them and act with one of
>> the following:
>>
>> * Fix the issue making a PR
>> * Review an existing PR and / or merge it
>> * If you intend to mark it as a deferred blocker, please provide a
>> rationale and contact me first by pinging me in the issue.
>>
>> Until these issues are fixed or deferred, the release team will not be
>> able to make a new beta release.
>>
>> Thanks for your help,
>>
>> Regards from stormy London,
>> Pablo Galindo Salgado
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/I43AL4Q2PIDNISC7OOJBAGZTPGRB67KR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.10.0b2 is available

[Pablo Galindo Salgado]

After fighting with some release blockers, implementing a bunch of GC
traversal functions, and fixing some pending reference leaks, we finally
have Python 3.10.0 beta 2 ready for you! Thanks to everyone that helped to
unblock the release!

https://www.python.org/downloads/release/python-3100b2/

# This is a beta preview of Python 3.10

Python 3.10 is still in development. 3.10.0b2 is the second of four planned
beta release previews. Beta release previews are intended to give the wider
community the opportunity to test new features and bug fixes and to prepare
their projects to support the new feature release.

We **strongly encourage** maintainers of third-party Python projects to
**test with 3.10** during the beta phase and report issues found to [the
Python bug tracker](https://bugs.python.org/) as soon as possible. While
the release is planned to be feature complete entering the beta phase, it
is possible that features may be modified or, in rare cases, deleted up
until the start of the release candidate phase (Monday, 2021-08-02). Our
goal is to have no ABI changes after beta 4 and as few code changes as
possible after 3.10.0rc1, the first release candidate. To achieve that, it
will be **extremely important** to get as much exposure for 3.10 as
possible during the beta phase.

Please keep in mind that this is a preview release and its use is **not**
recommended for production environments.

The next pre-release of Python 3.10 will be 3.10.0b3, currently scheduled
for Thursday, 2021-06-17.

# And now for something completely different

The Ehrenfest paradox concerns the rotation of a "rigid" disc in the theory
of relativity. In its original 1909 formulation as presented by Paul
Ehrenfest in relation to the concept of Born rigidity within special
relativity, it discusses an ideally rigid cylinder that is made to rotate
about its axis of symmetry. The radius R as seen in the laboratory frame is
always perpendicular to its motion and should therefore be equal to its
value R0 when stationary. However, the circumference (2πR) should appear
Lorentz-contracted to a smaller value than at rest. This leads to the
apparent contradiction that R = R0 and R < R0.

# We hope you enjoy those new releases!

Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

Regards from very sunny London,

Your friendly release team,
Pablo Galindo @pablogsal
Ned Deily @nad
Steve Dower @steve.dower
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/K24LNBMWJH6E6YKZK5PMAVEAQFPAPTYN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.10.0b3 is available

[Pablo Galindo Salgado]

Summer is almost here (at least in half of the planet) and Python 3.10 is
finishing baking in the oven. For those of you that want to taste it before
is finally ready (and if you are a library developer, you certainly do!)
you can have the second-to-last beta now, but be careful as is still very
hot ;)
https://www.python.org/downloads/release/python-3100b3/

#This is a beta preview of Python 3.10

Python 3.10 is still in development. 3.10.0b3 is the third of four planned
beta release previews. Beta release previews are intended to give the wider
community the opportunity to test new features and bug fixes and to prepare
their projects to support the new feature release.

We strongly encourage maintainers of third-party Python projects to test
with 3.10 during the beta phase and report issues found to the Python bug
tracker as soon as possible. While the release is planned to be feature
complete entering the beta phase, it is possible that features may be
modified or, in rare cases, deleted up until the start of the release
candidate phase (Monday, 2021-08-02). Our goal is to have no ABI changes
after beta 4 and as few code changes as possible after 3.10.0rc1, the first
release candidate. To achieve that, it will be extremely important to get
as much exposure for 3.10 as possible during the beta phase.

Please keep in mind that this is a preview release and its use is not
recommended for production environments.

The next pre-release of Python 3.10 will be 3.10.0b4, currently scheduled
for Saturday, 2021-07-10.

#And now for something completely different

There are no green stars. Why? In general, objects don’t emit a single
wavelength of light when they shine. Instead, they emit photons in a range
of wavelengths. If you were to use some sort of detector that is sensitive
to the wavelengths of light emitted by an object, and then plotted the
number of them versus wavelength, you get a lopsided plot called a
blackbody curve. For an object as hot as the Sun, that curve peaks at
blue-green, so it emits most of its photons there. But it still emits some
that are bluer, and some that are redder. When we look at the Sun, we see
all these colors blended together. Our eyes mix them up to produce one
color: white. A warmer star will put out more blue, and a cooler one
redder, but no matter what, our eyes just won’t see that as green. Due to
how we perceive color, the only way to see a star as being green is for it
to be only emitting green light. But as starts always emit radiation
following the blackbody curve, that’s pretty much impossible.

# We hope you enjoy those new releases!

Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

Regards from very cloudy London,

Your friendly release team,
Pablo Galindo @pablogsal
Ned Deily @nad
Steve Dower @steve.dower
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VNUESN72NYKV5DKM6GJ3WUFUS6DWVUSN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] April Steering Council update

[Pablo Galindo Salgado]

 devs use the file, and the consensus was that setting expectations for
   both reviewers and PR authors could be beneficial and could help to avoid
   misunderstandings. It was decided that some devguide changes could be
   proposed.


Regards from rainy London.
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4MJU47K6GZHCSXWEWQCJHX5D2KUNN6MH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is it too late to critique PEP 657? (Include Fine Grained Error Locations in Tracebacks)

[Pablo Galindo Salgado]

> I was expected the announcement of a BDFL delegate for PEP 657, as the
author is a steering council member.

Just to clarify this: as I was the author I didn't get to vote on the
approval or rejection of the PEP. Also, there is nowhere that I know where
this situation requires a BDFL delegate like you claim.

> It seems that a PEP submitted by a SC member has been accepted by the SC
with seemingly no external review.

That is very not true. We have added ton of beedback for at least 2
python-dev threads and a discourse thread:

*
https://discuss.python.org/t/pep-657-include-fine-grained-error-locations-in-tracebacks/8629
*
https://mail.python.org/archives/list/python-dev@python.org/thread/DB3RTYBF2BXTY6ZHP3Z4DXCRWPJIQUFD/
*
https://mail.python.org/archives/list/python-dev@python.org/thread/KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD/#KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD

The design has been iterated many times and we have draft implementations
from some of the alternative ideas as the ones proposed by Nathaniel.

> interact poorly with debugging and profiling.

"interact poorly with debugging and profiling." is making unfounded claims,
which is especially bad for a feature that is supposed to aid debugging.
You mentioned your worries on
https://mail.python.org/archives/list/python-dev@python.org/thread/KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD/#KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD
and we answered them.

Kind regards,
Pablo




On Tue, 29 Jun 2021 at 16:30, Mark Shannon  wrote:

> Hi,
>
> I was expected the announcement of a BDFL delegate for PEP 657, as the
> author is a steering council member.
>
> It seems that a PEP submitted by a SC member has been accepted by the SC
> with seemingly no external review.
>
> PEP 657 is likely to cause significant worsening of start up time and
> interact poorly with debugging and profiling.
>
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OPTIO7SWLTEZCWMAHVDKF7WUX2LSQ46X/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2VCQBSF67EZDRAFZOZBIG7IJHN2IEBW5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is it too late to critique PEP 657? (Include Fine Grained Error Locations in Tracebacks)

[Pablo Galindo Salgado]

Also, regarding startup times. Here are our benchmarks, measured with Linux
perf (https://perf.wiki.kernel.org/index.php/Main_Page)

* Importing all the standard library without PEP 657:

 Performance counter stats for './python import_all_stdlib.py' (500 runs):

303,78 msec task-clock#0,999 CPUs utilized
   ( +-  0,05% )
 4  context-switches  #0,014 K/sec
   ( +-  6,33% )
 0  cpu-migrations#0,000 K/sec

 7.602  page-faults   #0,025 M/sec
   ( +-  0,05% )
 1.146.643.923  cycles#3,775 GHz
   ( +-  0,03% )
 1.613.882.354  instructions  #1,41  insn per cycle
  ( +-  0,01% )
   391.399.735  branches  # 1288,440 M/sec
   ( +-  0,01% )
 6.501.180  branch-misses #1,66% of all
branches  ( +-  0,02% )

  0,304092 +- 0,000147 seconds time elapsed  ( +-  0,05% )

* Importing all the standard library with PEP 657:

 Performance counter stats for './python import_all_stdlib.py' (500 runs):

310,56 msec task-clock#0,999 CPUs utilized
   ( +-  0,06% )
 5  context-switches  #0,017 K/sec
   ( +-  9,39% )
 0  cpu-migrations#0,000 K/sec

 8.359  page-faults   #0,027 M/sec
   ( +-  0,05% )
 1.170.505.335  cycles#3,769 GHz
   ( +-  0,03% )
 1.637.448.123  instructions  #1,40  insn per cycle
  ( +-  0,01% )
   396.921.300  branches  # 1278,097 M/sec
   ( +-  0,01% )
 6.568.450  branch-misses #1,65% of all
branches  ( +-  0,02% )

  0,310904 +- 0,000179 seconds time elapsed  ( +-  0,06% )

The overhead is a 2% increase in startup time, which is 0.006812004
seconds of difference.

On Tue, 29 Jun 2021 at 16:39, Pablo Galindo Salgado 
wrote:

> > I was expected the announcement of a BDFL delegate for PEP 657, as the
> author is a steering council member.
>
> Just to clarify this: as I was the author I didn't get to vote on the
> approval or rejection of the PEP. Also, there is nowhere that I know where
> this situation requires a BDFL delegate like you claim.
>
> > It seems that a PEP submitted by a SC member has been accepted by the SC
> with seemingly no external review.
>
> That is very not true. We have added ton of beedback for at least 2
> python-dev threads and a discourse thread:
>
> *
> https://discuss.python.org/t/pep-657-include-fine-grained-error-locations-in-tracebacks/8629
> *
> https://mail.python.org/archives/list/python-dev@python.org/thread/DB3RTYBF2BXTY6ZHP3Z4DXCRWPJIQUFD/
> *
> https://mail.python.org/archives/list/python-dev@python.org/thread/KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD/#KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD
>
> The design has been iterated many times and we have draft implementations
> from some of the alternative ideas as the ones proposed by Nathaniel.
>
> > interact poorly with debugging and profiling.
>
> "interact poorly with debugging and profiling." is making unfounded
> claims, which is especially bad for a feature that is supposed to aid
> debugging. You mentioned your worries on
> https://mail.python.org/archives/list/python-dev@python.org/thread/KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD/#KR7ACFCUNMHT4M7R4XNHGRFV27HZBDFD
> and we answered them.
>
> Kind regards,
> Pablo
>
>
>
>
> On Tue, 29 Jun 2021 at 16:30, Mark Shannon  wrote:
>
>> Hi,
>>
>> I was expected the announcement of a BDFL delegate for PEP 657, as the
>> author is a steering council member.
>>
>> It seems that a PEP submitted by a SC member has been accepted by the SC
>> with seemingly no external review.
>>
>> PEP 657 is likely to cause significant worsening of start up time and
>> interact poorly with debugging and profiling.
>>
>>
>> Cheers,
>> Mark.
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/OPTIO7SWLTEZCWMAHVDKF7WUX2LSQ46X/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TBDLJ5ZKN4Q7MUKOLTPNPHBQV5GUTVJO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Critique of PEP 657

[Pablo Galindo Salgado]

s more
> to the implementation than the PEP)
> Using four separate objects to hold the location info adds a lot of
> overhead for most functions.


The internal representation of the data is an implementation detail. Our
plan is to go with the most straightforward implementation first
using separate tables, but if you see an opportunity for improvement and
you have some good ideas to merge them into a single table,
then you are more than welcome to make a Pull Request improving upon the
initial version.

Regards from cloudy London,
Pablo Galindo Salgado
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PREWIYYVY4RSERCX7OQ27MU2EF4PIQVM/
Code of Conduct: http://python.org/psf/codeofconduct/