[Python-Dev] Making code object APIs unstable

[Guido van Rossum]

In 3.11 we're changing a lot of details about code objects. Part of this is
the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine
Grained Error Locations in Tracebacks).

As a result, the set of fields of the code object is changing. This is
fine, the structure is part of the internal API anyway.

But there's a problem with two public API functions, PyCode_New() and
PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their
signatures are incompatible with previous versions, and they have to be
since the set of values needed to create a code object is different. (The
types.CodeType constructor signature is also changed, and so is its
replace() method, but these aren't part of any stable API).

Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP
387 stable ABI. What should we do?

A. We could deprecate them, keep (restore) their old signatures, and create
crippled code objects (no exception table, no endline/column tables,
qualname defaults to name).

B. We could deprecate them, restore the old signatures, and always raise an
error when they are called.

C. We could just delete them.

D. We could keep them, with modified signatures, and to heck with ABI
compatibility for these two.

E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the
posonlyargcount (which is the only difference between the two), and d*mn
the torpedoes.

F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New()
(and deprecate it).

If these weren't part of the stable ABI, I'd choose (E). But because they
are, I think only (A) or (B) are our options. The problem with (C) is that
if there's code that links to them but doesn't call them (except in some
corner case that the user can avoid), the code won't link even though it
would work fine. The problem with (D) is that if it *is* called by code
expecting the old signature it will segfault. I'm not keen on (A) since it
can cause broken code objects when used to copy a code object with some
modified metadata (e.g. a different filename), since there's no way to pass
the exception table (and several other fields, but the exception table is
an integral part of the code now).

Code wanting to make slight modifications to code objects such as changing
co_name or co_filename should switch to the .replace() API, which is much
better at this (though calling it from C is a pain, it's possible). Code
wanting to synthesize code should be updated for each release; we should
probably require such code to be built with the internal API and use
_PyCode_New(), which takes a struct argument containing all the necessary
fields.

Thoughts? I'm especially interested in Petr's opinion given that this is a
case where we'd like to deprecate something in the stable ABI.

See also discussion in https://bugs.python.org/issue40222 (esp. near the
end).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2021-08-06 - 2021-08-13)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7393 ( -4)
  closed 49249 (+60)
  total  56642 (+56)

Open issues with patches: 2924 


Issues opened (39)
==

#44855: [doc] Complete the sqlite3 exception documentation
https://bugs.python.org/issue44855  opened by erlendaasland

#44859: Improve some sqlite3 errors
https://bugs.python.org/issue44859  opened by serhiy.storchaka

#44860: sysconfig's posix_user scheme has different platlib value to d
https://bugs.python.org/issue44860  opened by uranusjr

#44861: csv.writer stopped to quote values with escapechar with csv.QU
https://bugs.python.org/issue44861  opened by xflr6

#44862: [docs] make "Deprecated since version {deprecated}, will be re
https://bugs.python.org/issue44862  opened by vini.g.fer

#44863: Allow TypedDict to inherit from Generics
https://bugs.python.org/issue44863  opened by sransara

#44864: [argparse] Do not translate user-provided strings in `Argument
https://bugs.python.org/issue44864  opened by jdetrey

#44865: [argparse] Missing translations
https://bugs.python.org/issue44865  opened by jdetrey

#44868: misleading error about fd / follow_symlinks from os.stat()
https://bugs.python.org/issue44868  opened by andrei.avk

#44869: MacOS Monterrey malloc issue
https://bugs.python.org/issue44869  opened by edumorlom

#44874: Deprecate Py_TRASHCAN_SAFE_BEGIN/END
https://bugs.python.org/issue44874  opened by iritkatriel

#44875: Update dis.findlinestarts documentaiton to reflect new usage o
https://bugs.python.org/issue44875  opened by gwk

#44876: .replace functions in datetime do not call __new__
https://bugs.python.org/issue44876  opened by joelgibson

#44877: Python > 3.7 build fails with IBM XL compiler
https://bugs.python.org/issue44877  opened by quellyn

#44881: Consider integration of PyObject_GC_UnTrack() with the trashca
https://bugs.python.org/issue44881  opened by nascheme

#44882: add FileInput.rollback() for in-place filters
https://bugs.python.org/issue44882  opened by samwyse

#44883: configure --with-openssl-rpath=DIR too eager about existence o
https://bugs.python.org/issue44883  opened by Elvis.Pranskevichus

#44884: logging Formatter behavior when using msecs and braces : '{'
https://bugs.python.org/issue44884  opened by fxcallewaert

#44886: asyncio: create_datagram_endpoint() does not return a Datagram
https://bugs.python.org/issue44886  opened by Thomas Trummer

#44887: test_input_tty hangs when run multiple times in the same proce
https://bugs.python.org/issue44887  opened by lukasz.langa

#44888: ssl.OP_LEGACY_SERVER_CONNECT missing
https://bugs.python.org/issue44888  opened by xtkoba

#44889: Specialize LOAD_METHOD with PEP 659 adaptive interpreter
https://bugs.python.org/issue44889  opened by kj

#44892: configparser fails when the file contains a "%" inside a comme
https://bugs.python.org/issue44892  opened by DiddiLeija

#44893: importlib.metadata Entrypoint has a broken _asdict
https://bugs.python.org/issue44893  opened by Ronny.Pfannschmidt

#44894: HTTP request handler should check sys.stderr for None before u
https://bugs.python.org/issue44894  opened by xiaolingbao

#44895: refleak test failure in test_exceptions
https://bugs.python.org/issue44895  opened by iritkatriel

#44896: Issue with unparse in ast module
https://bugs.python.org/issue44896  opened by xiaket

#44897: Integrate trashcan mechanism into _Py_Dealloc
https://bugs.python.org/issue44897  opened by nascheme

#44898: Path.read_bytes() failed when path contains chinese character
https://bugs.python.org/issue44898  opened by russiavk

#44899: tarfile: add support for creating an archive of potentially ch
https://bugs.python.org/issue44899  opened by marko-tuononen

#44900: Implement superinstructions
https://bugs.python.org/issue44900  opened by Mark.Shannon

#44901: Info about used pickle protocol used by multiprocessing.Queue
https://bugs.python.org/issue44901  opened by buhtz

#44903: [Doc] How does one to about getting onto the "Other Graphical 
https://bugs.python.org/issue44903  opened by PySimpleGUI

#44904: Erroneous behaviour for abstract class properties
https://bugs.python.org/issue44904  opened by rzepecki.t

#44905: Abstract instance and class attributes for abstract base class
https://bugs.python.org/issue44905  opened by rzepecki.t

#44906: Crash on deep call stack under Windows
https://bugs.python.org/issue44906  opened by aleReimondo

#44907: examples code output do not match the current version 3.9
https://bugs.python.org/issue44907  opened by yangqing

#44908: recommend httpx as well as requests in http.client/urllib.requ
https://bugs.python.org/issue44908  opened by graingert

#44909: configure should pick /usr/bin/g++ automatically if present
https://bugs.python.org/issue44909  opened by pmenzel



Most recent 15 issues with no re

[Python-Dev] Re: Making code object APIs unstable

[Terry Reedy]

On 8/13/2021 1:24 PM, Guido van Rossum wrote:
In 3.11 we're changing a lot of details about code objects. Part of this 
is the "Faster CPython" work, part of it is other things (e.g. PEP 657 
-- Fine Grained Error Locations in Tracebacks).


As a result, the set of fields of the code object is changing. This is 
fine, the structure is part of the internal API anyway.


But there's a problem with two public API functions, PyCode_New() and 
PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, 
their signatures are incompatible with previous versions, and they have 
to be since the set of values needed to create a code object is 
different. (The types.CodeType constructor signature is also changed, 
and so is its replace() method, but these aren't part of any stable API).


Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the 
PEP 387 stable ABI. What should we do?


PEP 387 is Backwards Compatibility Policy
https://www.python.org/dev/peps/pep-0387/
Did you mean PEP 384 -- Defining a Stable ABI
https://www.python.org/dev/peps/pep-0384/

How are PyCode_xxx included?  384 defines code objects as 'internal 
objects'.  "In some cases, the incompatibilities only affect internal 
objects of the interpreter, such as frame or code objects."  And Firefox 
does not find 'pycode'.


A. We could deprecate them, keep (restore) their old signatures, and 
create crippled code objects (no exception table, no endline/column 
tables, qualname defaults to name).


B. We could deprecate them, restore the old signatures, and always raise 
an error when they are called.


These seem pretty useless.


C. We could just delete them.

D. We could keep them, with modified signatures, and to heck with ABI 
compatibility for these two.


E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add 
the posonlyargcount (which is the only difference between the two), and 
d*mn the torpedoes.


F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for 
PyCode_New() (and deprecate it).



--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3OXTGNT5DTBJBWF7ZXLQB22NYNZWHJI7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Patrick Reader]

I'm a major consumer of these APIs as part of some commercial projects 
(unfortunately I can't discuss too much further) but we find it worth the 
effort of keeping up with CPython internal changes to continue using them. 
Option D seems like the best option from my point of view; any user would need 
to be able to keep up with bytecode changes already so I think backwards 
compatibility isn't really a concern.

Unless you feel it's extremely important to follow PEP 387 (slippery slope?), 
I'd just say "to hell with it". Also, at least if it segfaults, people who 
haven't updated their code will know very quickly, whereas some other options 
might cause subtler bugs.

On 13/08/2021 18:24, Guido van Rossum wrote:
> In 3.11 we're changing a lot of details about code objects. Part of this is 
> the "Faster CPython" work, part of it is other things (e.g. PEP 657 -- Fine 
> Grained Error Locations in Tracebacks).
>
> As a result, the set of fields of the code object is changing. This is fine, 
> the structure is part of the internal API anyway.
>
> But there's a problem with two public API functions, PyCode_New() and 
> PyCode_NewWithPosArgs(). As we have them in the main (3.11) branch, their 
> signatures are incompatible with previous versions, and they have to be since 
> the set of values needed to create a code object is different. (The 
> types.CodeType constructor signature is also changed, and so is its replace() 
> method, but these aren't part of any stable API).
>
> Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the PEP 
> 387 stable ABI. What should we do?
>
> A. We could deprecate them, keep (restore) their old signatures, and create 
> crippled code objects (no exception table, no endline/column tables, qualname 
> defaults to name).
>
> B. We could deprecate them, restore the old signatures, and always raise an 
> error when they are called.
>
> C. We could just delete them.
>
> D. We could keep them, with modified signatures, and to heck with ABI 
> compatibility for these two.
>
> E. We could get rid of PyCode_NewWithPosArgs(), update PyCode() to add the 
> posonlyargcount (which is the only difference between the two), and d*mn the 
> torpedoes.
>
> F. Like (E), but keep PyCode_NewWithPosArgs() as an alias for PyCode_New() 
> (and deprecate it).
>
> If these weren't part of the stable ABI, I'd choose (E). But because they 
> are, I think only (A) or (B) are our options. The problem with (C) is that if 
> there's code that links to them but doesn't call them (except in some corner 
> case that the user can avoid), the code won't link even though it would work 
> fine. The problem with (D) is that if it *is* called by code expecting the 
> old signature it will segfault. I'm not keen on (A) since it can cause broken 
> code objects when used to copy a code object with some modified metadata 
> (e.g. a different filename), since there's no way to pass the exception table 
> (and several other fields, but the exception table is an integral part of the 
> code now).
>
> Code wanting to make slight modifications to code objects such as changing 
> co_name or co_filename should switch to the .replace() API, which is much 
> better at this (though calling it from C is a pain, it's possible). Code 
> wanting to synthesize code should be updated for each release; we should 
> probably require such code to be built with the internal API and use 
> _PyCode_New(), which takes a struct argument containing all the necessary 
> fields.
>
> Thoughts? I'm especially interested in Petr's opinion given that this is a 
> case where we'd like to deprecate something in the stable ABI.
>
> See also discussion in https://bugs.python.org/issue40222 
>  (esp. near the end).
>
> -- 
> --Guido van Rossum (python.org/~guido )
> /Pronouns: he/him //(why is my pronoun here?)/ 
> 
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ZWTBR5ESYR26BUIVMXOKPFRLGGYDJSFC/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NLGBBH6ZPW6O5K2MAUIXEGKLYCAU6NHK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Eric Snow]

On Fri, Aug 13, 2021 at 11:29 AM Guido van Rossum  wrote:
> If these weren't part of the stable ABI, I'd choose (E).

They aren't in the stable ABI (or limited API).  Instead, they are
part of the broader public API (declared in Include/cpython/code.h,
along with "struct PyCodeObject" and others).

FWIW, there is actually very little API related to PyCodeObject that
is in the limited API:

* Include/code.h:typedef struct PyCodeObject PyCodeObject;
* Include/genobject.h:PyCodeObject *prefix##_code;
* Include/pyframe.h:PyAPI_FUNC(PyCodeObject *)
PyFrame_GetCode(PyFrameObject *frame);

All that said, the issue of compatibility remains.  I mostly agree
with Guido's analysis and his choice of (E), as long as it's
appropriately documented as unstable.

However, I'd probably pick (C) with a caveat.  We already have a
classification for this sort of unstable API: "internal".  Given how
code objects are so coupled to the CPython internals, I suggest that
most API related to PyCodeObject belongs in the internal API (in
Include/internal/pycore_code.h) and thus moved out of the public API.
Folks that are creating code objects manually via the C-API are
probably already doing low-level stuff that requires other "internal"
API (via Py_BUILD_CORE, etc.).  Otherwise they should use
types.CodeType instead.

Making that change would naturally include dropping PyCode_New() and
PyCode_NewWithPosArgs(), as described in (C).  However, we already
have _PyCode_New() in the internal API.  (It is slightly different but
effectively equivalent.)  We could either drop the underscore on
_PyCode_New() or move the existing PyCode_NewWithPosArgs() (renamed to
PyCode_New) to live beside it.

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KWWRLL56EI2S5BVADKMDCG4UED76GXXG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Jim J. Jewett]

How badly would the code objects be crippled?

>(no exception table, no endline/column tables,
> qualname defaults to name)

That sounds like it would be a pain for debugging, but might still work for 
source/code that hadn't actually changed and was just being re-compiled and run 
(possibly with the caveat that the data needs to be clean enough to avoid 
exceptions).

Since that is a common use case, and one where there is a good reason not to 
make any source-level changes, it would be good to keep compatibility for that 
minimal level.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/URGJJWVEDKROLYOUWQ5QQ5CROZHLTH7G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Guido van Rossum]

On Fri, Aug 13, 2021 at 11:17 AM Terry Reedy  wrote:

> On 8/13/2021 1:24 PM, Guido van Rossum wrote:
> [...]
> > Unfortunately, PyCode_New() and PyCode_NewWithPosArgs() are part of the
> > PEP 387 stable ABI. What should we do?
>
> PEP 387 is Backwards Compatibility Policy
> https://www.python.org/dev/peps/pep-0387/
> Did you mean PEP 384 -- Defining a Stable ABI
> https://www.python.org/dev/peps/pep-0384/
>

Aargh! You're right! I misremembered the PEP number (and didn't double
check) and when Pablo said we were "constrained by PEP 387" I assumed he
meant the stable ABI. Sorry for the goof-up.

Now, backwards compatibility is still nothing to sneeze at, but at least we
don't have to hem and haw about ABI compatibility.

The question then is, can we break (source) backwards compatibility for
these two functions, because in practice code objects have been unstable?
Or do we need to go the deprecation route here using (A) or (B) for two
releases. Presumably we could combine these two so that it reports a
warning when called, and if the warning has been configured to be replaced
with an error it will raise that.

Thanks also to Eric for pointing out the same thing. (I should probably
have waited to discuss this with him before posting. Sorry again.)


On Fri, Aug 13, 2021 at 3:39 PM Jim J. Jewett  wrote:

> How badly would the code objects be crippled?
>
> >(no exception table, no endline/column tables,
> > qualname defaults to name)
>
> That sounds like it would be a pain for debugging, but might still work
> for source/code that hadn't actually changed and was just being re-compiled
> and run (possibly with the caveat that the data needs to be clean enough to
> avoid exceptions).
>

Yeah, I would be okay with faking it for endline/column tables and
qualname, but I balk at the exception table -- this is used for the new
3.11 concept "zero cost exception handling" where a try block doesn't
require any setup (the opcode for this has gone away). Too much valid code
catches exceptions (e.g. KeyError) to trust that this will mostly do the
right thing.


> Since that is a common use case, and one where there is a good reason not
> to make any source-level changes, it would be good to keep compatibility
> for that minimal level.


I'm actually not sure what use case you're talking about.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OVXUGWFGDLMKKW7VZFFUYEU7ZNHAROVQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Making code object APIs unstable

[Terry Reedy]

On 8/13/2021 8:45 PM, Guido van Rossum wrote:

Now, backwards compatibility is still nothing to sneeze at, but at least 
we don't have to hem and haw about ABI compatibility.


If back compatibility were our sacred prime directive, then we would not 
(or should not) have changed code objects in way that threatens it.


The question then is, can we break (source) backwards compatibility for 
these two functions, because in practice code objects have been 
unstable?


I presume that the existence of those functions is based on an 
expectation of stability that may have been true for years, but not 
recently.  Or an expectation that any change would be part of a one-time 
overhaul where compatibility was suspended as much as for 3.0.  I think 
that whatever we do now should allow for continued version-by-version 
changes in the future.


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JKHISUP624VI7MVUG7XPYO5QTBLFIUT2/
Code of Conduct: http://python.org/psf/codeofconduct/