[Python-Dev] Re: Proposal: declare "unstable APIs"

[Serhiy Storchaka]

04.06.21 12:08, Petr Viktorin пише:
> It is used, and I started the work :)
> See e.g.
> https://docs.python.org/3.10/c-api/sequence.html#c.PySequence_Concat

Great news! Several core developers (including me) tried to solve this
problem, but a large amount of work stopped us at an early stage. Glad
there is a progress.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/44XXQIFBIUNTYTDH7XSN3XKBZ6SLYM5K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

[Robin Becker]

On 04/06/2021 19:29, Steve Dower wrote:
...


While we were investigating the reliably broken behaviour, we figured that related behaviour was 
*unreliably* broken on all platforms.


Specifically, if you imported a module through a relative path (resolved to CWD), changed the CWD, cleared the module 
cache, and reimported, you would still get the original module, contrary to the intent of getting the newly resolved 
module. ("Correct" code would have also cleared the import path caches, not just sys.modules.) This was because the 
module info was cached using the relative path, and so would be used later even though its absolute path had changed.


Someone reported this change in 3.8 and we decided to revert it in existing released, because the bugfix was for an 
obscure enough situation that it really wasn't worth breaking any existing versions. For unreleased versions, it's 
better to fix it so that imports will be more reliable in the future.



Unfortunately we are using these relative imports to implement plugins for an xml language processor so this is likely 
not just an accident of ordering the failing tests. I agree we have to get a plugin module a bit more directly using the 
import internals. I still think we might need to control sys.path as presumably the plugin code could have imports of 
parallel modules.


I'm beginning to understand that messing about with sys.path/modules/meta_path 
etc etc is a lot harder than it used to be.

Incidentally I find that I am able to import a namespace module directly; this 
code doesn't crash (at least in python >= 3.6

###
#timpA.py don't run with any meaningful directory A
import importlib
def main():
module = importlib.import_module('A')
print('module=%r' % module)

if __name__=='__main__':
main()
###

in python 3.6-3.9 this prints out "module=" in 3.10.0b1 
it prints
"module=)>".
I thought I had to use pkgutils or pkg_resources to mess with these new-fangled 
packages.

Likewise I find that the python statement 'import A' also doesn't fail and A is 
 a namespace. I feel quite stupid now :(
--
Robin Becker
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGSXINVBLQOMREURUFN4LVCEIVX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: change of behaviour for '.' in sys.path between 3.10.0a7 and 3.10.0b1

[Brett Cannon]

On Sat., Jun. 5, 2021, 07:38 Robin Becker,  wrote:

> On 04/06/2021 19:29, Steve Dower wrote:
> ...
> >
> > While we were investigating the reliably broken behaviour, we figured
> that related behaviour was
> > *unreliably* broken on all platforms.
> >
> > Specifically, if you imported a module through a relative path (resolved
> to CWD), changed the CWD, cleared the module
> > cache, and reimported, you would still get the original module, contrary
> to the intent of getting the newly resolved
> > module. ("Correct" code would have also cleared the import path caches,
> not just sys.modules.) This was because the
> > module info was cached using the relative path, and so would be used
> later even though its absolute path had changed.
> >
> > Someone reported this change in 3.8 and we decided to revert it in
> existing released, because the bugfix was for an
> > obscure enough situation that it really wasn't worth breaking any
> existing versions. For unreleased versions, it's
> > better to fix it so that imports will be more reliable in the future.
>
>
> Unfortunately we are using these relative imports to implement plugins for
> an xml language processor so this is likely
> not just an accident of ordering the failing tests. I agree we have to get
> a plugin module a bit more directly using the
> import internals. I still think we might need to control sys.path as
> presumably the plugin code could have imports of
> parallel modules.
>
> I'm beginning to understand that messing about with
> sys.path/modules/meta_path etc etc is a lot harder than it used to be.
>

How so? The key details have not changed since Python 3.4 when module specs
were introduced. And meta path, etc. are 2.7 concepts.


> Incidentally I find that I am able to import a namespace module directly;
> this code doesn't crash (at least in python >= 3.6
>
> ###
> #timpA.py don't run with any meaningful directory A
> import importlib
> def main():
>  module = importlib.import_module('A')
>  print('module=%r' % module)
>
> if __name__=='__main__':
>  main()
> ###
>
> in python 3.6-3.9 this prints out "module=" in
> 3.10.0b1 it prints
> "module= at 0x7f1c8a22fe20>)>".
> I thought I had to use pkgutils or pkg_resources to mess with these
> new-fangled packages.
>

Nope, that has never been required since the concept of namespace packages
were introduced to import.


> Likewise I find that the python statement 'import A' also doesn't fail and
> A is  a namespace.


Yep, that's expected.

I feel quite stupid now :(
>

As soon as you step outside the norm there are a lot of details to
understand; import is very much following the "Complex is better than
complicated" zen. So no need to feel stupid, there's a lot to keep track of
when you start to dive into the details.

-Brett

--
> Robin Becker
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGSXINVBLQOMREURUFN4LVCEIVX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KQFXVNNCYYOEBZJTNRXTT7RQBDRTR5LC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proposal: declare "unstable APIs"

[Guido van Rossum]

On Fri, Jun 4, 2021 at 6:15 AM Victor Stinner  wrote:

> On Fri, Jun 4, 2021 at 12:29 AM Guido van Rossum  wrote:
> >> In the C API, there is the internal C API which fits with your
> >> description. To access it, you have to declare the
> >> Py_BUILD_CORE_MODULE macro. It's not usable directly on purpose. It's
> >> an user agreement: I know what I am doing, and I know that this API is
> >> not supported nor stable.
> >
> > Hm, but aren't for example all the fields of code objects (co_name,
> co_argcount, etc.) in the "non-internal" API? Those (and the functions that
> manipulate code objects) are a prime example of what I'd consider
> "unstable".
> >
> > On https://docs.python.org/3/c-api/code.html it already says about the
> fields "The fields of this type are subject to change at any time." But I
> think everything else on that page should be considered unstable as well.
> (And why do we even have PyCode_GetNumFree()?)
>
> Hum, the C API is somehow off-topic, but let me reply anyway ;-)
>

I see the smiley but I still don't understand why you're saying this.

As I explained in my PEP 620, the Python C API never had any design.
>

That's not entirely true. *Parts* of it were definitely designed, e.g. the
existing "abstract" API (PyObject_GetItem, _GetAttr, and many more; IIRC
Zope's Jim Fulton pushed for these). Other parts are indeed not designed
much, all the PyList, PyDict, PyTuple etc. APIs (which came before the
"abstract" ones), and the many APIs that followed their example (to this
day).


> Things were only exposed because it was easy and technically possible,
> and it was a convenient way to define a function in one file and uses
> it from another file.


Historically, it was even worse -- people started embedding Python before
we even had "Py" prefixes. Try to find out about the "great renaming" (I
think it was the late '90s).


> But 30 years later, we identified that exposing
> some things are causing troubles and we are trying to make the C API
> more "abstract".
>

This time the abstraction isn't just about supporting different object
types, it's about allowing us to change other aspects of the interpreter,
in particular object layout ("object model" as it's somehow known) and
memory management.

Who knows what we'll want to change 20 years from now...


> Exposing directly all structures is causing a lot of headaches at
> every new Python 3.x release. Getter and setter functions can handle
> structure changes, retrieve information from another structure, return
> an error, etc. This abstractation is needed for users to not have to
> update their code, and to CPython developers to be able to change
> things.
>
> If possible, I would prefer to make PyThreadState, PyCodeObject and
> other structures opaque, and only go through getter and setter
> functions ;-) PyCode_New() is another problem :-/ The PEP 570 first
> changed it to add a new parameter. It broke Cython and other projects.
> The change was reverted, and PyCode_NewWithPosOnlyArgs() was added.
> The lesson is that it's possible to change PyCodeObject without
> breaking PyCode_New() (which handles the "backward compatibility" for
> you).
>

I'm afraid that won't always be possible though. At some point there just
may not be a valid meaning for the original PyCode_New() call. It was easy
in the case of positional arguments (by default don't have any) but it may
not always be that simple, and we shouldn't make guarantees here.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4EYQNUDHXGSCCVUSEBUHA6S7IE7QXYMS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Proposal: declare "unstable APIs"

[Guido van Rossum]

On Sat, Jun 5, 2021 at 2:12 AM Serhiy Storchaka  wrote:

> 04.06.21 12:08, Petr Viktorin пише:
> > It is used, and I started the work :)
> > See e.g.
> > https://docs.python.org/3.10/c-api/sequence.html#c.PySequence_Concat
>
> Great news! Several core developers (including me) tried to solve this
> problem, but a large amount of work stopped us at an early stage. Glad
> there is a progress.
>

Yes, this is indeed great news. It also brings the existence of these
distinctions (between stable/public/cpython APIs) to the attention of their
users, which helps building mindshare for limiting API use to the stable
API. Thanks Petr for pushing this!

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XWDDWLDPQ3NLPQVOZLVIBQCLPTLESSAB/
Code of Conduct: http://python.org/psf/codeofconduct/