Re: [Python-Dev] Making PyInterpreterState an opaque type

[Jeroen Demeyer]

On 2019-02-19 04:04, Steve Dower wrote:

Otherwise, the
internal memory layout becomes part of the public ABI


Of course, the ABI (not API) depends on the internal memory layout. Why 
is this considered a problem? If you want a fixed ABI, use API level (1) 
from my last post. If you want a fixed API but not ABI, use level (2). 
If you really want stuff to be broken at any time, use (3) or (4). This 
is why I don't see the need to make a difference between (3) and (4): 
neither of them makes any guarantees about stability.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Jeroen Demeyer]

On 2019-02-19 04:04, Steve Dower wrote:

On 18Feb.2019 1324, Jeroen Demeyer wrote:

For a very concrete example, was it really necessary to put
_PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially
given that the very similar PySequence_Fast_ITEMS is in (2), that seems
like a strange and arbitrary limiting choice.


The reason to do this is that we can "guarantee" that we've fixed all
users when we change the internal representation.


I think that CPython should then at least "eat its own dog food" and 
don't use any of the internal functions/macros when implementing the 
stdlib. As I said before: if a function/macro is useful for implementing 
stdlib functionality like "functools" or "json", it's probably useful 
for external modules too.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Antoine Pitrou]

On Mon, 18 Feb 2019 19:04:31 -0800
Steve Dower  wrote:
> 
> > I'm afraid of hiding actually useful private macros under Py_BUILD_CORE.
> > For example, Modules/_functoolsmodule.c and Modules/_json.c use API
> > functions from (4). But if an API function is useful for implementing
> > functools or json, then it's probably also useful for external extension
> > modules: what if I want to implement something similar to functools or
> > json, why shouldn't I be allowed to use those same API functions?
> > 
> > For a very concrete example, was it really necessary to put
> > _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially
> > given that the very similar PySequence_Fast_ITEMS is in (2), that seems
> > like a strange and arbitrary limiting choice.  
> 
> The reason to do this is that we can "guarantee" that we've fixed all
> users when we change the internal representation. Otherwise, the
> internal memory layout becomes part of the public ABI, which is what we
> want to fix. (PyTuple_GET_ITEM is just as problematic, FWIW.)

But PyTuple_GET_ITEM and PyList_GET_ITEM are important for performance,
as are other performance-oriented macros.

> If you always rebuild your extension for every micro version (3.x.y) of
> CPython, then sure, go ahead and use this.

Usually we would guarantee that API details don't change in bugfix
versions (i.e. the 3.x.y -> 3.x.(y + 1) transition).  Has that changed?
That may turn out a big problem for several third-party extensions...

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[INADA Naoki]

On Tue, Feb 19, 2019 at 7:32 PM Jeroen Demeyer  wrote:
>
> On 2019-02-19 04:04, Steve Dower wrote:
> > On 18Feb.2019 1324, Jeroen Demeyer wrote:
> >> For a very concrete example, was it really necessary to put
> >> _PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially
> >> given that the very similar PySequence_Fast_ITEMS is in (2), that seems
> >> like a strange and arbitrary limiting choice.
> >
> > The reason to do this is that we can "guarantee" that we've fixed all
> > users when we change the internal representation.
>
> I think that CPython should then at least "eat its own dog food" and
> don't use any of the internal functions/macros when implementing the
> stdlib. As I said before: if a function/macro is useful for implementing
> stdlib functionality like "functools" or "json", it's probably useful
> for external modules too.

If we are perfect and we can design perfect APIs from start, I agree with you.
But we should fix design mistake of new APIs.

stdlibs are updated with Python itself.  So changing internal APIs with micro
version is OK.
If Cython start using such internal APIs, external modules from PyPI
will be broken
when Python is upgraded.  It feels nightmare to me.
So having experimental APIs only for stdlibs makes sense to me.

On the other hand, it makes sense to move _PyTuple_ITEMS to (3) or even (2).
PyTuple_ITEMS(t) seems better than &PyTuple_GET_ITEM(t, 0).

Regards,

-- 
INADA Naoki  
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Victor Stinner]

Le lun. 18 févr. 2019 à 22:34, Jeroen Demeyer  a écrit :
> First of all, if everybody can actually #define Py_BUILD_CORE and get
> access to the complete API, I don't mind so much. But then it's
> important that this actually keeps working (i.e. that those headers will
> always be installed).
>
> Still, do we really need so many levels of API:
> (1) stable API (with #define Py_LIMITED_API)
> (2) public documented API
> (3) private undocumented API (the default exposed API)
> (4) internal API (with #define Py_BUILD_CORE)

It's not a matter of documentation. It's a matter of warranty provided to users.

I would like to move towards (1) by default: only provide a stable API
by default. IMHO most users will be just fine with this subset of the
API.

The borders between (2), (3) and (4) are unclear. I created
Include/cpython/ which is not really a "private API" but more "CPython
implementation details". A better definition of (1) would be "portable
stable API" whereas (2)+(3) would be "CPython stable API".

And (4) would be the unstable API.

Summary:

(1) Portable stable API
(2) Portable CPython API
(3) Unstable API

... The border between (2) and (3) is a "work-in-progress".

I modified "make install" to install (3) as well: they are users of
this API. Performance can be a good motivation: Cython for example.
Debuggers and profiles really need to access to the lowest level of
API usually because they can only *inspect* (structure fileds) but no
*execute* code (call functions).

> I'm afraid of hiding actually useful private macros under Py_BUILD_CORE.

Again, it's not a matter of usefulness. It's a matter of backward
compatibility announced to users.

I would like to make it more explicit that if you *opt in* for an
unstable API, you are on your own.

My intent is that in 5 years or 10 years, slowly, most C extensions
will use (1) which will allow Python to experiment new optimizations,
and should help PyPy (cpyext module) to become even more efficient.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Victor Stinner]

Le mar. 19 févr. 2019 à 11:57, INADA Naoki  a écrit :
> On the other hand, it makes sense to move _PyTuple_ITEMS to (3) or even (2).
> PyTuple_ITEMS(t) seems better than &PyTuple_GET_ITEM(t, 0).

Please don't use &PyTuple_GET_ITEM() or _PyTuple_ITEMS(). It prevents
to use a more efficient storage for tuple. Something like:
https://pythoncapi.readthedocs.io/optimization_ideas.html#specialized-list-for-small-integers

PyPy already has the issue right now.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbottest on Android emulator with docker

[Victor Stinner]

This is cool! Sadly, I don't have the bandwidth right now to play with
it. I may have a look later.

Victor

Le ven. 15 févr. 2019 à 18:27, Xavier de Gaye  a écrit :
>
> The following command runs the buildbottest on an Android emulator with 
> docker (it will use a little bit more than 11 GB):
>
>  $ docker run -it --privileged xdegaye/abifa:r14b-24-x86_64-master
>
> This command does:
> * pull an image from the Docker hub (only the first time that the command is 
> run, note that this a 2 GB download !) and start a container
> * pull the latest changes from the GitHub cpython repository and 
> cross-compile python
> * start an Android emulator and install python on it
> * run the buildbottest target of cpython Makefile
>
> The image is built from a Dockerfile [2].
>
> This same image can also be used with the 'bash' command line argument to 
> enter bash in the container and run python interactively on the emulator [1]. 
>  If the 'docker run' command also sets a bind mount to a local cpython 
> repository, then it is possible to develop/debug/fix python on the emulator 
> running in this container using one's own clone of cpython.
>
> [1] documentation at https://xdegaye.gitlab.io/abifa/docker.html
> [2] Dockerfile at 
> https://gitlab.com/xdegaye/abifa/blob/master/docker/Dockerfile.r14b-24-x86_64-master
>
> Xavier
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbottest on Android emulator with docker

[Stephane Wirtel]

Nice, we could start to fix the Android issues with this docker image,
maybe modified with the local repository but really useful.

Thank you so much for your job.

--
Stéphane Wirtel - https://wirtel.be - @matrixise
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Adding test.support.safe_rmpath()

[eryk sun]

On 2/16/19, Richard Levasseur  wrote:
>
> First: The tempfile module is a poor fit for testing (don't get me wrong,
> it works, but its not *nice for use in tests*)*.* This is because:
> 1. Using it as a context manager is distracting. The indentation signifies
> a conceptual scope the reader needs to be aware of, but in a test context,
> its usually not useful. At worst, it covers most of the test. At best, its
> constrained to a block at the start.
> 2. tempfile defaults to binary mode instead of text; just another thing to
> bite you.
> 3. On windows, you can't reopen the file, so for cross-platform stuff, you
> can't even use it for this case.

Python opens files with at least read and write sharing in Windows, so
typically there's no problem with opening a file multiple times. The
problem is with deleting and renaming open files. Typically delete
access is not shared, and, even if it is, a normal delete just sets a
disposition. A deleted file is unlinked only after all handles have
been closed. Similarly, replacing an open file via os.replace will
fail because it can't be unlinked.

In Windows 10 we can delete and rename files with POSIX-like
semantics. To do this, open a handle with delete access and call
SetFileInformationByHandle to set the FileDispositionInfoEx or
FileRenameInfoEx information. Thus far this is supported by NTFS, and
I think it's only NTFS. It's still not completely like POSIX, since it
requires delete-access sharing. But it does provide immediate
unlinking, which avoids the race condition when trying to remove a
directory that has watched files. Programs that have open files that
have been unlinked can continue to access them normally.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] int() and math.trunc don't accept objects that only define __index__

[Nick Coghlan]

On Tue, 19 Feb 2019 at 03:31, Rémi Lapeyre  wrote:
> Nick Coghlan proposes to make __int__ defaults to __index__ when only the 
> second
> is defined and asked to open a discussion on python-dev before making any 
> change
> "as the closest equivalent we have to this right now is the "negative" 
> derivation,
> where overriding __eq__ without overriding __hash__ implicitly marks the 
> derived
> class as unhashable (look for "type->tp_hash = 
> PyObject_HashNotImplemented;").".

Reading this again now, it occurs to me that there's another developer
experience improvement we already made along these lines in Python 3:
"By default, __ne__() delegates to __eq__() and inverts the result
unless it is NotImplemented. " [1]

By contrast, the corresponding (and annoying) Python 2 behaviour was:
"The truth of x==y does not imply that x!=y is false. Accordingly,
when defining __eq__(), one should also define __ne__() so that the
operators will behave as expected." [2]

The only difference is that whereas the new `__ne__` delegation
behaviour could just be defined directly in `object.__ne__()`,
`object` doesn't implement `__int__` by default, so the delegating
function would need to be injected into the type when it is defined
(and that's the part that's similar to the `__hash__ = None` negative
derivation).

So +1 from me.

Cheers,
Nick.

[1] https://docs.python.org/3/reference/datamodel.html#object.__ne__
[2] https://docs.python.org/2/reference/datamodel.html#object.__ne__

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Nick Coghlan]

On Tue, 19 Feb 2019 at 05:33, Eric Snow  wrote:
>
> On Sat, Feb 16, 2019 at 3:47 PM Antoine Pitrou  wrote:
> > On Sat, 16 Feb 2019 14:34:46 -0800
> > Steve Dower  wrote:
> > > Which seems to suggest that the answer to "which members are important
> > > to expose?" is "probably none".
> >
> > That sounds intuitive. But we don't know what kind of hacks some
> > extension authors might do, for legitimate reasons...
> >
> > (perhaps some gevent-like framework needs access to the interpreter
> > state?)
>
> In those cases either we will expose accessor functions in the C-API
> or they can define Py_BUILD_CORE.

I really don't want us to ever get into a situation where we're
actively encouraging third party projects to define Py_BUILD_CORE.

If we decide we do want to go down a path like that, I'd instead
prefer to see us define something more like "Py_FRAGILE_API" to make
it clear that folks using those extra interfaces are tightly coupling
themselves to a specific version of CPython, and are likely going to
need to make changes when new versions are released.

Even though we would probably *implement* that by having this snippet
in one of our header files:

#ifdef Py_FRAGILE_API
#define Py_BUILD_CORE
#endif

I still think it would convey the concerns we have more clearly than
simply telling people to define Py_BUILD_CORE would.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] int() and math.trunc don't accept objects that only define __index__

[Rémi Lapeyre]

Another point in favor of the change I just noticed is that int()
accept objects defining __index__ as its `base` argument:


    Python 3.7.2 (default, Jan 13 2019, 12:50:01)
    [Clang 10.0.0 (clang-1000.11.45.5)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> class MyInt:
    ...     def __index__(self):
    ...             return 4
    ...
    >>> int("3", base=MyInt())
    3
    >>> int(MyInt())
    Traceback (most recent call last):
    File "", line 1, in 
    TypeError: int() argument must be a string, a bytes-like object or
a number, not 'MyInt'
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Nick Coghlan]

On Tue, 19 Feb 2019 at 20:41, Antoine Pitrou  wrote:
> On Mon, 18 Feb 2019 19:04:31 -0800
> Steve Dower  wrote:
> > If you always rebuild your extension for every micro version (3.x.y) of
> > CPython, then sure, go ahead and use this.
>
> Usually we would guarantee that API details don't change in bugfix
> versions (i.e. the 3.x.y -> 3.x.(y + 1) transition).  Has that changed?
> That may turn out a big problem for several third-party extensions...

This is the genuine technical difference between the three levels:

* Py_BUILD_CORE -> no ABI stability guarantees at all
* standard -> stable within a maintenance branch
* Py_LIMITED_API -> stable across feature releases

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbottest on Android emulator with docker

[Stephane Wirtel]

Hi Xavier,

I get this exception

Timeout (360 seconds) reached; failed to start emulator
---> Device ready.
---> Install Python on the emulator.
/home/pydev/build/python-native/python -B 
/home/pydev/abifa/Android/tools/install.py
error: device 'emulator-5556' not found
unzip -q /home/pydev/dist/python3.8-android-24-x86_64-stdlib.zip -d 
/tmp/tmpt_v_gdbo
/home/pydev/android/android-sdk/platform-tools/adb -s emulator-5556 shell mkdir 
-p /data/local/tmp/python
Command "('/home/pydev/android/android-sdk/platform-tools/adb', '-s', 
'emulator-5556', 'shell', 'mkdir -p /data/local/tmp/python')" returned non-zero exit 
status 1
/home/pydev/abifa/Android/emulator.mk:57: recipe for target '_install' failed
make: *** [_install] Error 1


Is there a repository with a bug tracker because I would not want to
pullute this ML with that.

Thank you,

Stéphane


--
Stéphane Wirtel - https://wirtel.be - @matrixise
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] new binary wheels PEP idea

[Alexander Revin]

Hi all,

I have an idea regarding Python binary wheels on non-glibc platforms,
and it seems that initially I've posted it to the wrong list ([1])

Long story short, the proposal is to use platform tuples (like
compiler ones) for wheel names, which will allow much broader platform
support, for example:

package-1.0-cp36-cp36m-amd64_linux_gnu.whl
package-1.0-cp36-cp36m-amd64_linux_musl.whl

So eventually only {platform tag} part will be modified. Glibc/musl
detection is quite trivial and eventually will be based on existing
one in PEP 513 [2].

Let me know what you think.

Best regards,
Alex

[1] https://mail.python.org/pipermail/python-list/2019-February/739524.html
[2] https://www.python.org/dev/peps/pep-0513/#id49
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Steve Dower]

On 19Feb2019 0229, Jeroen Demeyer wrote:

On 2019-02-19 04:04, Steve Dower wrote:

On 18Feb.2019 1324, Jeroen Demeyer wrote:

For a very concrete example, was it really necessary to put
_PyTuple_ITEMS in (4)? That's used in _functoolsmodule.c. Especially
given that the very similar PySequence_Fast_ITEMS is in (2), that seems
like a strange and arbitrary limiting choice.


The reason to do this is that we can "guarantee" that we've fixed all
users when we change the internal representation.


I think that CPython should then at least "eat its own dog food" and 
don't use any of the internal functions/macros when implementing the 
stdlib. As I said before: if a function/macro is useful for implementing 
stdlib functionality like "functools" or "json", it's probably useful 
for external modules too.


I'm inclined to agree, but then I'm also one of the advocates for 
breaking out as much as possible of the stdlib into pip-installable 
modules, which would necessitate this :)


There are certainly parts of the stdlib that are there to _enable_ the 
use of these features without exposing them - asyncio has some good 
examples of this. But the rest probably don't. If they really do, then 
we would have to define stable ways to get the same functionality (one 
example - numpy currently relies on being able to check the refcount to 
see if it equals 1, but we could easily provide a 
"Py_HasNoOtherReferences" function that would do the same thing and also 
allow us to one day move or remove reference counts without breaking numpy).


That said, one of the criteria for "are you eligible to use the internal 
API" is "will users always have matched builds of this module" - for the 
standard library, the answer is yes, and so they can use that API.


For third-party extension modules, the answer _might_ be yes, which 
means they _might_ be able to use the API. But both of those "mights" 
are outside of the control of the core development team, so we can't 
take that responsibility for you.


The best we can do is make it easy to use the stable APIs, and make 
using the unstable APIs a deliberate choice so that those who do so are 
aware that they are now more responsible for their user's success than 
they thought they were.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Steve Dower]

On 19Feb2019 0555, Nick Coghlan wrote:

I really don't want us to ever get into a situation where we're
actively encouraging third party projects to define Py_BUILD_CORE.

If we decide we do want to go down a path like that, I'd instead
prefer to see us define something more like "Py_FRAGILE_API" to make
it clear that folks using those extra interfaces are tightly coupling
themselves to a specific version of CPython, and are likely going to
need to make changes when new versions are released.


I mean, my suggestion of "Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY" was only a 
little bit tongue-in-cheek :)


Maybe there's a good Monty Python reference we can use here? 
"Py_ITS_JUST_A_FLESH_WOUND" or "Py_THEN_WE_JUMP_OUT_OF_THE_RABBIT"


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] buildbottest on Android emulator with docker

[Xavier de Gaye]

https://gitlab.com/xdegaye/abifa

The table of content of the first url in my initial post gives also a
link to that repository.under the name 'Repository'.

Xavier

On Tue, Feb 19, 2019 at 4:54 PM Stephane Wirtel  wrote:
>
> Hi Xavier,
>
> I get this exception
>
> Timeout (360 seconds) reached; failed to start emulator
> ---> Device ready.
> ---> Install Python on the emulator.
> /home/pydev/build/python-native/python -B 
> /home/pydev/abifa/Android/tools/install.py
> error: device 'emulator-5556' not found
> unzip -q /home/pydev/dist/python3.8-android-24-x86_64-stdlib.zip -d 
> /tmp/tmpt_v_gdbo
> /home/pydev/android/android-sdk/platform-tools/adb -s emulator-5556 shell 
> mkdir -p /data/local/tmp/python
> Command "('/home/pydev/android/android-sdk/platform-tools/adb', '-s', 
> 'emulator-5556', 'shell', 'mkdir -p /data/local/tmp/python')" returned 
> non-zero exit status 1
> /home/pydev/abifa/Android/emulator.mk:57: recipe for target '_install' failed
> make: *** [_install] Error 1
>
>
> Is there a repository with a bug tracker because I would not want to
> pullute this ML with that.
>
> Thank you,
>
> Stéphane
>
>
> --
> Stéphane Wirtel - https://wirtel.be - @matrixise
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/xdegaye%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Barry Warsaw]

Steve Dower wrote on 2/16/19 14:34:>

This is mostly about being able to assign blame when things break, so
I'm totally okay with extension modules that want to play with internals
declaring Py_BUILD_CORE to get access to them (though I suspect that
won't work out of the box - maybe we should have a
Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?).


Let's call it Py_POINTED_STICK of course!

http://www.montypython.net/scripts/fruit.php

-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is distutils.util.get_platform() the "current" or the "target" platform

[Xavier de Gaye]

[Any reason for dropping python-dev?]


Sorry. just clicked the wrong button.


And the answer is a resounding "yes, it returns the target platform"? It seems 
you're saying this, but the wording of your email sounds just enough of a question that 
I'm not sure whether you are definitively answering it or not.


The answer is yes indeed, it returns the target platform.
This is a listing of the non-obvious steps that lead to this conclusion.

Xavier


On 2/19/19 8:45 PM, Steve Dower wrote:

[Any reason for dropping python-dev?]

On 19Feb2019 1139, Xavier de Gaye wrote:

Is distutils.util.get_platform() the "current" or the "target" platform ?


I *think* you're answering this below, yes?


* When cross-compiling on posix platforms using autoconf, configure is
   run with the '--host=host-type' [1] command line option to specify
   the target platform.

* The AC_CANONICAL_HOST macro is used by configure.ac to get the
   canonical variables `host' and `host_cpu' [2].
   Those variables are used to compute _PYTHON_HOST_PLATFORM.

* The Makefile generated by configure runs setup.py,
   generate-posix-vars, etc... on the build platform using
   PYTHON_FOR_BUILD, a native python interpreter that is set
   to run with the _PYTHON_HOST_PLATFORM environment variable.

* get_platform() in setup.py and in Lib/distutils/util.py returns the
   value of _PYTHON_HOST_PLATFORM when cross-compiling.

So the process of cross-compilation on posix platforms has
get_platform() return the target ('host' in autoconf terminology) platform.

[1] 
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Specifying-Target-Triplets.html
[2] 
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Canonicalizing.html


And the answer is a resounding "yes, it returns the target platform"? It seems 
you're saying this, but the wording of your email sounds just enough of a question that 
I'm not sure whether you are definitively answering it or not.

Cheers,
Steve


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Stefan Behnel]

Nick Coghlan schrieb am 19.02.19 um 15:00:
> On Tue, 19 Feb 2019 at 20:41, Antoine Pitrou wrote:
>> On Mon, 18 Feb 2019 19:04:31 -0800 Steve Dower wrote:
>>> If you always rebuild your extension for every micro version (3.x.y) of
>>> CPython, then sure, go ahead and use this.
>>
>> Usually we would guarantee that API details don't change in bugfix
>> versions (i.e. the 3.x.y -> 3.x.(y + 1) transition).  Has that changed?
>> That may turn out a big problem for several third-party extensions...
> 
> This is the genuine technical difference between the three levels:
> 
> * Py_BUILD_CORE -> no ABI stability guarantees at all
> * standard -> stable within a maintenance branch
> * Py_LIMITED_API -> stable across feature releases

I'm happy with this split, and i think this is how it should be. There is
no reason (not withstanding critical bugs) to break the C-API within a
maintenance (3.x) release series. Apart from the 3.5.[12] slip, CPython has
proven very reliable in these guarantees.

We can (or at least could) easily take care in Cython to enable version
specific features and optimisations only from CPython alpha/beta releases
on, and not when they should become available in later point releases, so
that users can compile their code in, say, CPython 3.7.5 and it will work
correctly in 3.7.1.

We never cared about Py_BUILD_CORE (because that's obviously internal), and
it's also not very likely that we will have a Py_LIMITED_API backend
anywhere in the near future (although we would consider PRs for it that
implement the support as an optional C compile time feature).

What I would ask, though, and I think that's also Jeroen's request, is to
be careful what you lock up behind Py_BUILD_CORE. Any new functionality
should be available to extension modules by default, unless there is a good
reason why it should remain internal. Usually, there is a reason why this
functionality was added, and I doubt that there are many cases where these
reasons are entirely internal to CPython.

One thing that is not mentioned above are underscore private C-API
functions. I imagine that they are a bit annoying for CPython itself
because promoting them to public means renaming them, which is already a
breaking change. But they are a clear marker for potential future breakage,
which is good. Still, my experience so far suggests that they also fall
under the "keep stable in maintenance branch" rule, which is even better.

So, yeah, I'm happy with the status quo, and a bit worried about all the
moving around of declarations and that scent of a sword of Damocles hanging
over their potential confinement. IMHO, things should just be public and
potentially marked as "unstable" to advertise a risk of breakage in a
future CPython X.Y feature releases. Then it's up to the users to decide
how much work they want to invest into keeping up with C-API changes vs.
potentially sub-optimal but stable C-API usage.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Steve Dower]

On 19Feb2019 1212, Stefan Behnel wrote:

So, yeah, I'm happy with the status quo, and a bit worried about all the
moving around of declarations and that scent of a sword of Damocles hanging
over their potential confinement. IMHO, things should just be public and
potentially marked as "unstable" to advertise a risk of breakage in a
future CPython X.Y feature releases. Then it's up to the users to decide
how much work they want to invest into keeping up with C-API changes vs.
potentially sub-optimal but stable C-API usage.


Unfortunately, advertising a risk of breakage doesn't make the break 
less painful when it happens. We'd rather avoid that pain by 
preemptively breaking (at a major version update, e.g. 3.8) so that 
minor breaks (e.g. between 3.8.2 and 3.8.3) don't cause any problems at 
all. And if we preemptively break, then we can also preemptively add 
functions to cover what direct memory accesses previously used to do.


And it's not up to the users - it's up to the package developers. Most 
of whom optimise for their own ease of life (as someone who supports 
Windows users, I'm well aware of where package developers cut painful 
corners ;) ). The only choice users get in the matter is whether they 
ever update Python, or if they switch to a language that is more 
respectful toward them.


For what it's worth, the users I've been speaking to recently are *far* 
more concerned about being able to update Python without things breaking 
than they are about runtime performance.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Steve Dower]

On 19Feb2019 1141, Barry Warsaw wrote:

Steve Dower wrote on 2/16/19 14:34:>

This is mostly about being able to assign blame when things break, so
I'm totally okay with extension modules that want to play with internals
declaring Py_BUILD_CORE to get access to them (though I suspect that
won't work out of the box - maybe we should have a
Py_I_TOO_LIKE_TO_LIVE_DANGEROUSLY?).


Let's call it Py_POINTED_STICK of course!

http://www.montypython.net/scripts/fruit.php


+1, and instead of "using the internal API" we can call it "coming at 
CPython with a banana" :D


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making PyInterpreterState an opaque type

[Stefan Behnel]

Steve Dower schrieb am 19.02.19 um 21:40:
> On 19Feb2019 1212, Stefan Behnel wrote:
>> Then it's up to the users to decide
>> how much work they want to invest into keeping up with C-API changes vs.
>> potentially sub-optimal but stable C-API usage.
> [...]
> And it's not up to the users - it's up to the package developers.

I meant "users" as in "users of the C-API", i.e. package developers.

Stefan

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] new binary wheels PEP idea

[Brett Cannon]

Unfortunately you're still posted to the wrong list, Alexander. You want to
mail distutils-...@python.org where packaging discussions occur.

On Tue, Feb 19, 2019 at 8:19 AM Alexander Revin  wrote:

> Hi all,
>
> I have an idea regarding Python binary wheels on non-glibc platforms,
> and it seems that initially I've posted it to the wrong list ([1])
>
> Long story short, the proposal is to use platform tuples (like
> compiler ones) for wheel names, which will allow much broader platform
> support, for example:
>
> package-1.0-cp36-cp36m-amd64_linux_gnu.whl
> package-1.0-cp36-cp36m-amd64_linux_musl.whl
>
> So eventually only {platform tag} part will be modified. Glibc/musl
> detection is quite trivial and eventually will be based on existing
> one in PEP 513 [2].
>
> Let me know what you think.
>
> Best regards,
> Alex
>
> [1]
> https://mail.python.org/pipermail/python-list/2019-February/739524.html
> [2] https://www.python.org/dev/peps/pep-0513/#id49
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] new binary wheels PEP idea

[Steve Dower]

And for what it's worth, most of the really active contributors from 
distutils-sig seem to prefer the "Packaging" category at 
https://discuss.python.org/


If you'd prefer to use Discourse, I'd suggest posting there first and 
also email distutils-sig with a link to the discussion. Otherwise, go 
straight to distutils-sig (just don't be too surprised if you don't seem 
to get much traction there or if someone restarts the discussion on 
Discourse for you).


Cheers,
Steve

On 19Feb2019 1341, Brett Cannon wrote:
Unfortunately you're still posted to the wrong list, Alexander. You want 
to mail distutils-...@python.org  where 
packaging discussions occur.


On Tue, Feb 19, 2019 at 8:19 AM Alexander Revin > wrote:


Hi all,

I have an idea regarding Python binary wheels on non-glibc platforms,
and it seems that initially I've posted it to the wrong list ([1])

Long story short, the proposal is to use platform tuples (like
compiler ones) for wheel names, which will allow much broader platform
support, for example:

package-1.0-cp36-cp36m-amd64_linux_gnu.whl
package-1.0-cp36-cp36m-amd64_linux_musl.whl

So eventually only {platform tag} part will be modified. Glibc/musl
detection is quite trivial and eventually will be based on existing
one in PEP 513 [2].

Let me know what you think.

Best regards,
Alex

[1]
https://mail.python.org/pipermail/python-list/2019-February/739524.html
[2] https://www.python.org/dev/peps/pep-0513/#id49


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] new binary wheels PEP idea

[Alexander Revin]

Thank you guys! Will try it that way.

Best,
Alex

On Tue, Feb 19, 2019 at 10:57 PM Steve Dower  wrote:
>
> And for what it's worth, most of the really active contributors from
> distutils-sig seem to prefer the "Packaging" category at
> https://discuss.python.org/
>
> If you'd prefer to use Discourse, I'd suggest posting there first and
> also email distutils-sig with a link to the discussion. Otherwise, go
> straight to distutils-sig (just don't be too surprised if you don't seem
> to get much traction there or if someone restarts the discussion on
> Discourse for you).
>
> Cheers,
> Steve
>
> On 19Feb2019 1341, Brett Cannon wrote:
> > Unfortunately you're still posted to the wrong list, Alexander. You want
> > to mail distutils-...@python.org  where
> > packaging discussions occur.
> >
> > On Tue, Feb 19, 2019 at 8:19 AM Alexander Revin  > > wrote:
> >
> > Hi all,
> >
> > I have an idea regarding Python binary wheels on non-glibc platforms,
> > and it seems that initially I've posted it to the wrong list ([1])
> >
> > Long story short, the proposal is to use platform tuples (like
> > compiler ones) for wheel names, which will allow much broader platform
> > support, for example:
> >
> > package-1.0-cp36-cp36m-amd64_linux_gnu.whl
> > package-1.0-cp36-cp36m-amd64_linux_musl.whl
> >
> > So eventually only {platform tag} part will be modified. Glibc/musl
> > detection is quite trivial and eventually will be based on existing
> > one in PEP 513 [2].
> >
> > Let me know what you think.
> >
> > Best regards,
> > Alex
> >
> > [1]
> > https://mail.python.org/pipermail/python-list/2019-February/739524.html
> > [2] https://www.python.org/dev/peps/pep-0513/#id49
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com