[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

If creating a fake frame is a common use case, we can maybe write a
public C API for that. For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.

Victor

On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel  wrote:
>
> Guido van Rossum schrieb am 13.08.21 um 19:24:
> > In 3.11 we're changing a lot of details about code objects. Part of this is
> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine
> > Grained Error Locations in Tracebacks).
> >
> > As a result, the set of fields of the code object is changing. This is
> > fine, the structure is part of the internal API anyway.
> >
> > But there's a problem with two public API functions, PyCode_New() and
> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
> > signatures are incompatible with previous versions, and they have to be
> > since the set of values needed to create a code object is different. (The
> > types.CodeType constructor signature is also changed, and so is its
> > replace() method, but these aren't part of any stable API).
> >
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP
> > 387 stable ABI. What should we do?
> >
> > A. We could deprecate them, keep (restore) their old signatures, and create
> > crippled code objects (no exception table, no endline/column tables,
> > qualname defaults to name).
> >
> > B. We could deprecate them, restore the old signatures, and always raise an
> > error when they are called.
> >
> > C. We could just delete them.
> >
> > D. We could keep them, with modified signatures, and to heck with ABI
> > compatibility for these two.
> >
> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the
> > posonlyargcount (which is the only difference between the two), and d*mn
> > the torpedoes.
> >
> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New()
> > (and deprecate it).
> >
> > If these weren't part of the stable ABI, I'd choose (E). [...]
>
> I also vote for (E). The creation of a code object is tied to interpreter
> internals and thus shouldn't be (or have been) declared stable.
>
> I think the only problem with that argument is that code objects are
> required for frames. You could argue the same way about frames, but then it
> becomes really tricky to, you know, create frames for non-Python code.
>
> Since we're discussing this in the context of PEP 657, I wonder if there's
> a better way to create tracebacks from C code, other than creating fake
> frames with fake code objects.
>
> Cython uses code objects and frames for the following use cases:
>
> - tracing generated C code at the Python syntax level
> - profiling C-implemented functions
> - tracebacks for C code
>
> Having a way to do these three efficiently (i.e. with close to zero runtime
> overhead) without having to reach into internals of the interpreter state,
> code objects and frames, would be nice.
>
> Failing that, I'm ok with declaring the relevant structs and C-API
> functions non-stable and letting Cython use them as such, as we always did.
>
> Stefan
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/XYNNMH57O7CYWHYKTD3ELZTM3B4M53HL/
> Code of Conduct: http://python.org/psf/codeofconduct/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QYDIQ5SS3FQCTDBBZ3JTI643KV4F6TNY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Pablo Galindo Salgado]

> If creating a fake frame is a common use case, we can maybe write a
public C API for that.

I don't think we should think on those terms. We certainly don't want to be
on a case
where yet again we cannot change the internals because we have an official
C-API exposed.


> For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.

You saw "parser" doing that? What is "parser"? Certainly is not the CPython
parser because I don't recall that we do any of that.


On Wed, 1 Sept 2021 at 17:48, Victor Stinner  wrote:

> If creating a fake frame is a common use case, we can maybe write a
> public C API for that. For example, I saw parser injecting frames to
> show the file name and line number of the parsed file in the
> traceback.
>
> Victor
>
> On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel  wrote:
> >
> > Guido van Rossum schrieb am 13.08.21 um 19:24:
> > > In 3.11 we're changing a lot of details about code objects. Part of
> this is
> > > the "Faster CPython" work, part of it is other things (e.g. PEP 657 --
> Fine
> > > Grained Error Locations in Tracebacks).
> > >
> > > As a result, the set of fields of the code object is changing. This is
> > > fine, the structure is part of the internal API anyway.
> > >
> > > But there's a problem with two public API functions, PyCode_New() and
> > > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch,
> their
> > > signatures are incompatible with previous versions, and they have to be
> > > since the set of values needed to create a code object is different.
> (The
> > > types.CodeType constructor signature is also changed, and so is its
> > > replace() method, but these aren't part of any stable API).
> > >
> > > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of
> the PEP
> > > 387 stable ABI. What should we do?
> > >
> > > A. We could deprecate them, keep (restore) their old signatures, and
> create
> > > crippled code objects (no exception table, no endline/column tables,
> > > qualname defaults to name).
> > >
> > > B. We could deprecate them, restore the old signatures, and always
> raise an
> > > error when they are called.
> > >
> > > C. We could just delete them.
> > >
> > > D. We could keep them, with modified signatures, and to heck with ABI
> > > compatibility for these two.
> > >
> > > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add
> the
> > > posonlyargcount (which is the only difference between the two), and
> d*mn
> > > the torpedoes.
> > >
> > > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
> PyCode_New()
> > > (and deprecate it).
> > >
> > > If these weren't part of the stable ABI, I'd choose (E). [...]
> >
> > I also vote for (E). The creation of a code object is tied to interpreter
> > internals and thus shouldn't be (or have been) declared stable.
> >
> > I think the only problem with that argument is that code objects are
> > required for frames. You could argue the same way about frames, but then
> it
> > becomes really tricky to, you know, create frames for non-Python code.
> >
> > Since we're discussing this in the context of PEP 657, I wonder if
> there's
> > a better way to create tracebacks from C code, other than creating fake
> > frames with fake code objects.
> >
> > Cython uses code objects and frames for the following use cases:
> >
> > - tracing generated C code at the Python syntax level
> > - profiling C-implemented functions
> > - tracebacks for C code
> >
> > Having a way to do these three efficiently (i.e. with close to zero
> runtime
> > overhead) without having to reach into internals of the interpreter
> state,
> > code objects and frames, would be nice.
> >
> > Failing that, I'm ok with declaring the relevant structs and C-API
> > functions non-stable and letting Cython use them as such, as we always
> did.
> >
> > Stefan
> >
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/XYNNMH57O7CYWHYKTD3ELZTM3B4M53HL/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/QYDIQ5SS3FQCTDBBZ3JTI643KV4F6TNY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.pytho

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

(context)

> Guido van Rossum schrieb am 13.08.21 um 19:24:
> > In 3.11 we're changing a lot of details about code objects. Part of this
> is
> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 --
> Fine
> > Grained Error Locations in Tracebacks).
> >
> > As a result, the set of fields of the code object is changing. This is
> > fine, the structure is part of the internal API anyway.
> >
> > But there's a problem with two public API functions, PyCode_New() and
> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
> > signatures are incompatible with previous versions, and they have to be
> > since the set of values needed to create a code object is different. (The
> > types.CodeType constructor signature is also changed, and so is its
> > replace() method, but these aren't part of any stable API).
> >
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the
> PEP
> > 387 stable ABI. What should we do?
> >
> > A. We could deprecate them, keep (restore) their old signatures, and
> create
> > crippled code objects (no exception table, no endline/column tables,
> > qualname defaults to name).
> >
> > B. We could deprecate them, restore the old signatures, and always raise
> an
> > error when they are called.
> >
> > C. We could just delete them.
> >
> > D. We could keep them, with modified signatures, and to heck with ABI
> > compatibility for these two.
> >
> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add
> the
> > posonlyargcount (which is the only difference between the two), and d*mn
> > the torpedoes.
> >
> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
> PyCode_New()
> > (and deprecate it).
> >
> > If these weren't part of the stable ABI, I'd choose (E). [...]
>

On Tue, Aug 31, 2021 at 7:07 PM Stefan Behnel  wrote:

> I also vote for (E). The creation of a code object is tied to interpreter
> internals and thus shouldn't be (or have been) declared stable.
>

I think you're one of the few people who call those functions, and if even
you think it's okay to break backward compatibility here, I think we should
just talk to the SC to be absolved of having these two in the stable ABI.
(Petr, do you agree? Without your backing I don't feel comfortable even
asking for this.)


> I think the only problem with that argument is that code objects are
> required for frames. You could argue the same way about frames, but then
> it
> becomes really tricky to, you know, create frames for non-Python code.
>

Note there's nothing in the stable ABI to create frames. There are only
functions to *get* an existing frame, to inspect a frame, and to eval it.
In any case even if there was a stable ABI function to create a frame from
a code object, one could argue that it's sufficient to be able to get an
existing code object from e.g. a function object.


> Since we're discussing this in the context of PEP 657, I wonder if there's
> a better way to create tracebacks from C code, other than creating fake
> frames with fake code objects.
>
> Cython uses code objects and frames for the following use cases:
>
> - tracing generated C code at the Python syntax level
> - profiling C-implemented functions
> - tracebacks for C code
>
> Having a way to do these three efficiently (i.e. with close to zero
> runtime
> overhead) without having to reach into internals of the interpreter state,
> code objects and frames, would be nice.
>
> Failing that, I'm ok with declaring the relevant structs and C-API
> functions non-stable and letting Cython use them as such, as we always did.
>

I think others have answered this already -- in any case it's not the
immediate subject of this thread, and I don't have a strong opinion on it.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Z6VSLFMAHYBETSYEYGG4K7EA7DNW3L42/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

I apologize, I keep making the same mistake.

The PyCode_New[WithPosArgs] functions are *not* in the stable ABI or in the
limited API, so there's no need to petition the SC, nor do I need Petr's
approval.

We may be bound by backwards compatibility for the *cpython* API, but I
think that if Cython is okay if we just break this we should be fine. Users
of the CPython API are expected to recompile for each new version, and if
someone were to be using these functions with the old set of parameters the
compiler would give them an error.

So let's just choose (E) and d*mn backwards compatibility for these two
functions.

That means:
- Get rid of PyCode_NewWithPosArgs altogether
- PyCode_New becomes unstable (and gets a new posinlyargcount argument)

On Wed, Sep 1, 2021 at 11:52 AM Guido van Rossum  wrote:

> (context)
>
>> Guido van Rossum schrieb am 13.08.21 um 19:24:
>> > In 3.11 we're changing a lot of details about code objects. Part of
>> this is
>> > the "Faster CPython" work, part of it is other things (e.g. PEP 657 --
>> Fine
>> > Grained Error Locations in Tracebacks).
>> >
>> > As a result, the set of fields of the code object is changing. This is
>> > fine, the structure is part of the internal API anyway.
>> >
>> > But there's a problem with two public API functions, PyCode_New() and
>> > PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch,
>> their
>> > signatures are incompatible with previous versions, and they have to be
>> > since the set of values needed to create a code object is different.
>> (The
>> > types.CodeType constructor signature is also changed, and so is its
>> > replace() method, but these aren't part of any stable API).
>> >
>> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the
>> PEP
>> > 387 stable ABI. What should we do?
>> >
>> > A. We could deprecate them, keep (restore) their old signatures, and
>> create
>> > crippled code objects (no exception table, no endline/column tables,
>> > qualname defaults to name).
>> >
>> > B. We could deprecate them, restore the old signatures, and always
>> raise an
>> > error when they are called.
>> >
>> > C. We could just delete them.
>> >
>> > D. We could keep them, with modified signatures, and to heck with ABI
>> > compatibility for these two.
>> >
>> > E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add
>> the
>> > posonlyargcount (which is the only difference between the two), and d*mn
>> > the torpedoes.
>> >
>> > F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for
>> PyCode_New()
>> > (and deprecate it).
>> >
>> > If these weren't part of the stable ABI, I'd choose (E). [...]
>>
>
> On Tue, Aug 31, 2021 at 7:07 PM Stefan Behnel 
> wrote:
>
>> I also vote for (E). The creation of a code object is tied to interpreter
>> internals and thus shouldn't be (or have been) declared stable.
>>
>
> I think you're one of the few people who call those functions, and if even
> you think it's okay to break backward compatibility here, I think we should
> just talk to the SC to be absolved of having these two in the stable ABI.
> (Petr, do you agree? Without your backing I don't feel comfortable even
> asking for this.)
>
>
>> I think the only problem with that argument is that code objects are
>> required for frames. You could argue the same way about frames, but then
>> it
>> becomes really tricky to, you know, create frames for non-Python code.
>>
>
> Note there's nothing in the stable ABI to create frames. There are only
> functions to *get* an existing frame, to inspect a frame, and to eval it.
> In any case even if there was a stable ABI function to create a frame from
> a code object, one could argue that it's sufficient to be able to get an
> existing code object from e.g. a function object.
>
>
>> Since we're discussing this in the context of PEP 657, I wonder if
>> there's
>> a better way to create tracebacks from C code, other than creating fake
>> frames with fake code objects.
>>
>> Cython uses code objects and frames for the following use cases:
>>
>> - tracing generated C code at the Python syntax level
>> - profiling C-implemented functions
>> - tracebacks for C code
>>
>> Having a way to do these three efficiently (i.e. with close to zero
>> runtime
>> overhead) without having to reach into internals of the interpreter
>> state,
>> code objects and frames, would be nice.
>>
>> Failing that, I'm ok with declaring the relevant structs and C-API
>> functions non-stable and letting Cython use them as such, as we always
>> did.
>>
>
> I think others have answered this already -- in any case it's not the
> immediate subject of this thread, and I don't have a strong opinion on it.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

[Python-Dev] Re: Making code object APIs unstable

[Victor Stinner]

I saw Python projects injecting fake frames for XML and JSON parsers,
maybe also configuration file (.ini?) parsers. So text files which
have line numbers ;-)

On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado
 wrote:
> I don't think we should think on those terms. We certainly don't want to be 
> on a case
> where yet again we cannot change the internals because we have an official 
> C-API exposed.

PyCode_New() is annoying since it requires to provide *all* arguments.
I'm thinking of an Frame API which only allows to set 2 values:
filename and line number. Nothing else.

Something like: frame = PyFrame_New(filename, lineno).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AII6O6ZXEENM7FJH3JRJIPTG4BRKJ3G5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Discrepancy between what aiter() and `async for` requires on purpose?

[Nick Coghlan]

On Tue, 31 Aug 2021, 2:52 am Brett Cannon,  wrote:

>
> On Sun, Aug 29, 2021 at 2:01 PM Serhiy Storchaka 
> wrote:
>
>>
>> > So my question is whether the discrepancy between what `async for`
>> > expects and what `aiter()` expects on purpose?
>> > https://bugs.python.org/issue31861 
>> > was the issue for creating aiter() and I didn't notice a discussion of
>> > this difference. The key reason I'm asking is this does cause a
>> > deviation compared to the relationship between `for` and `iter()` (which
>> > does not require `__iter__` to be defined on the iterator, although
>> > collections.abc.Iterator does). It also makes the glossary definition
>> > being linked from
>> >
>> https://docs.python.org/3.10/reference/compound_stmts.html#the-async-for-statement
>> > <
>> https://docs.python.org/3.10/reference/compound_stmts.html#the-async-for-statement
>> >
>> > incorrect.
>>
>> PyIter_Check() only checks existence of __next__, not __iter__ (perhaps
>> for performance reasons).
>>
>
> Or maybe no one thought to require __iter__ for iterators?
>

I don't think PyIter_Check is testing the formal definition of an iterator,
I think it's just testing if calling __iter__ can be skipped (as you say,
for performance reasons).

I'm surprised iter() would skip calling __iter__ just because an object
defines __next__, though. Even though "__iter__ is defined and returns
self" is part of the iterator definition, it still feels like a leap from
there to "if __next__ is defined, skip calling __iter__ in iter()".

The optimisation that bypasses the __[a]iter__ method call feels more
legitimate in the actual for loop syntax, it just feels odd to me if the
builtin isn't forcing the call.

Cheers,
Nick.

>



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5UMLDQ4CANKY4WM6RNG67AEJMQI44X42/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Nick Coghlan]

On Thu, 2 Sep 2021, 7:08 am Victor Stinner,  wrote:

> I saw Python projects injecting fake frames for XML and JSON parsers,
> maybe also configuration file (.ini?) parsers. So text files which
> have line numbers ;-)
>
> On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado
>  wrote:
> > I don't think we should think on those terms. We certainly don't want to
> be on a case
> > where yet again we cannot change the internals because we have an
> official C-API exposed.
>
> PyCode_New() is annoying since it requires to provide *all* arguments.
> I'm thinking of an Frame API which only allows to set 2 values:
> filename and line number. Nothing else.
>
> Something like: frame = PyFrame_New(filename, lineno).
>


Perhaps "PyCode_FromLocation(file name, lineno)" for the new "good enough
to satisfy trace hooks and exception tracebacks" API? These are needed when
creating function-like (et al) objects from extension module code, not just
for runtime frames.

And then explicitly define which code object fields are expected to always
be populated and which are expected to be None on emulated code objects
that don't contain Python byte code? (e.g. some fields should be zero or
the empty tuple, rather than being set to None)

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AXGJAQ5MMRAKAKHHQMYVF5XS7PHALH4W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP-535 (Rich comparison chaining) Discussion?

[Nick Coghlan]

On Wed, 1 Sep 2021, 7:18 am Senthil Kumaran,  wrote:

> On Tue, Aug 31, 2021 at 8:57 AM Angus Hollands  wrote:
>
>
>> Should look more like
>> ```python
>> x = np.array(...)
>> y = x[2 < x < 8]
>> ```
>> Is there any interest in pushing this PEP along?
>>
>
> My personal opinion is, it is worth discussing again. I could see the
> benefits this brings to Numpy Arrays.
> The PEP itself was deferred based on boolean overload dislike than the
> short-circuit or chaining operator preference.
>

While I don't plan to reactivate these PEPs myself, I'd be happy to welcome
a co-author that was keen to champion them (535 doesn't stand alone - it
depends on the "binary if" and "binary else" circuit breaking protocol
proposal in 532).

While I believe 535 is mostly OK, 532 needs some updates to account for:

* the new language grammar (the PEP was written before the switch to pegen)
* the walrus operator (some of the possible future extensions mentioned are
irrelevant now)
* fixing the diagram rendering and ReST mark up

Cheers,
Nick.



> --
> Senthil
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ENJYFXADZXDU7RZBD5RJTPS6ZE7YFNMN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QY2FL7PMW6OBU24UFDQVPEJLOHZVQUPO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Greg Ewing]

On 2/09/21 4:46 am, Victor Stinner wrote:

If creating a fake frame is a common use case, we can maybe write a
public C API for that. For example, I saw parser injecting frames to
show the file name and line number of the parsed file in the
traceback.


The way I would like to see this addressed is to make it possible
to attach a filename and line number directly to a traceback object,
without needing a frame or code object at all.

Creating a fake frame and code object just to do this is IMO an
ugly hack that should not be necessary.

--
Greg

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XWX7EXJQ3JJRAL26YFR3GSSMLGSCEH2V/
Code of Conduct: http://python.org/psf/codeofconduct/