[Python-Dev] Re: Question regarding the value of PyThreadState.thread_id

[Victor Stinner]

Hi,

There are two reasons:

* PyThread_get_thread_native_id() was only added recently (Python 3.8)
* PyThread_get_thread_native_id() is not portable and not available on
all platforms: the function is only declared if the
PY_HAVE_THREAD_NATIVE_ID macro is defined.

Maybe PyThreadState.thread_id could use
PyThread_get_thread_native_id() if avaialble, or
PyThread_get_thread_ident() otherwise. But that sounds like an
incompatible change.

Another approach would be to add a *new* structure member, like
thread_native_id. It would not exist if the PY_HAVE_THREAD_NATIVE_ID
macro is not defined.


Include/pythread.h:

#if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
|| defined(__OpenBSD__) || defined(__NetBSD__) || defined(_WIN32) ||
defined(_AIX)
#define PY_HAVE_THREAD_NATIVE_ID
PyAPI_FUNC(unsigned long) PyThread_get_thread_native_id(void);
#endif


thread_pthread.h implementation:

unsigned long
PyThread_get_thread_native_id(void)
{
if (!initialized)
PyThread_init_thread();
#ifdef __APPLE__
uint64_t native_id;
(void) pthread_threadid_np(NULL, &native_id);
#elif defined(__linux__)
pid_t native_id;
native_id = syscall(SYS_gettid);
#elif defined(__FreeBSD__)
int native_id;
native_id = pthread_getthreadid_np();
#elif defined(__OpenBSD__)
pid_t native_id;
native_id = getthrid();
#elif defined(_AIX)
tid_t native_id;
native_id = thread_self();
#elif defined(__NetBSD__)
lwpid_t native_id;
native_id = _lwp_self();
#endif
return (unsigned long) native_id;
}

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.


On Fri, Apr 16, 2021 at 10:47 PM Gabriele  wrote:
>
> Hi all.
>
> My apologies if this is a topic that's been discussed already, but I
> wasn't able to locate anything in the archive on the subject. I was
> wondering if there's a fundamental reason for using
> PyThread_get_thread_ident instead of PyThread_get_thread_native_id for
> the value of the thread_id field of the PyThreadState object.
>
> The reason why I'm asking is that I would like to have easy access to
> the native TID on Linux to identify the /proc/stat file associated
> with the thread from an external process. At the moment I have to
> resort to calling process_vm_readv to copy the struct pthread over to
> local VM and then guess where the tid field might be. So, if there's
> no fundamental reason for thread_id to be PyThread_get_thread_ident, I
> would like to propose to change it to PyThread_get_thread_native_id
> instead.
>
> Thanks,
> Gabriele
>
> --
> "Egli è scritto in lingua matematica, e i caratteri son triangoli,
> cerchi, ed altre figure
> geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola;
> senza questi è un aggirarsi vanamente per un oscuro laberinto."
>
> -- G. Galilei, Il saggiatore.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/3DOB6VUTAIJKK4SUGBYWL4QPWI2E5Q2T/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6U47ITBNRZWNWELWXBZNAOAELGBAP3T6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 644 Accepted -- Require OpenSSL 1.1.1 or newer

[Christian Heimes]

On 30/03/2021 13.46, Pablo Galindo Salgado wrote:
> Hi Christian,
> 
> Thank you for submitting PEP 644 (Require OpenSSL 1.1.1). After evaluating
> the situation and discussing the PEP, the Steering Council is happy with
> the PEP,
> and hereby accepts it. The SC is of the opinion that this change will
> make it considerable
> easier to maintain the extension modules that depend on OpenSSL while
> allowing us
> to leverage the new APIs and improvements in the future. As indicated in
> the PEP,
> OpenSSL 1.1.1 is the default variant and version of OpenSSL on almost
> all supported
> platforms and distributions almost all known Major CI providers provide
> images with
> OpenSSL 1.1.1, so we believe this has the right balance to improve the
> situation without
> causing major impact or breakage.
> 
> Nevertheless, the Steering Council would like to see this change
> reflected properly in the
> documentation, What's New (linking against the new instructions here: 
> https://docs.python.org/3.10/using/unix.html#custom-openssl
> ) and
> releasemanager notices
> so this is properly communicated to our users. 
> 
> The Steering Council also thanks Christian for his patience explaining
> different aspects
> of the PEP, including in the PEP some extra requested information and
> for considering
> the concerts raised.
> 
> Congratulations, Christian!
> 
> With thanks from the whole Python Steering Council,
> Pablo Galindo Salgado

Hi Pablo,

I have implemented PEP 644 in https://bugs.python.org/issue43669 and
marked the PEP as final.

Thanks!

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4BXOPVK5P3QIHTMFANKG2FLGI2U5OU4C/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Question regarding the value of PyThreadState.thread_id

[Gabriele]

Hi Victor

Thanks for your reply. Indeed, upon further investigation, I have realised
changing the value of thread_id is not wise as there are other methods that
expect this field to be a pthread_t. I have opened issue
https://bugs.python.org/issue43879https://bugs.python.org/issue43879 to
have the new field added. I'm happy to do the work myself if it gets
accepted.

Cheers,
Gabriele

On Sat, 17 Apr 2021, 11:07 Victor Stinner,  wrote:

> Hi,
>
> There are two reasons:
>
> * PyThread_get_thread_native_id() was only added recently (Python 3.8)
> * PyThread_get_thread_native_id() is not portable and not available on
> all platforms: the function is only declared if the
> PY_HAVE_THREAD_NATIVE_ID macro is defined.
>
> Maybe PyThreadState.thread_id could use
> PyThread_get_thread_native_id() if avaialble, or
> PyThread_get_thread_ident() otherwise. But that sounds like an
> incompatible change.
>
> Another approach would be to add a *new* structure member, like
> thread_native_id. It would not exist if the PY_HAVE_THREAD_NATIVE_ID
> macro is not defined.
>
>
> Include/pythread.h:
>
> #if defined(__APPLE__) || defined(__linux__) || defined(__FreeBSD__)
> || defined(__OpenBSD__) || defined(__NetBSD__) || defined(_WIN32) ||
> defined(_AIX)
> #define PY_HAVE_THREAD_NATIVE_ID
> PyAPI_FUNC(unsigned long) PyThread_get_thread_native_id(void);
> #endif
>
>
> thread_pthread.h implementation:
>
> unsigned long
> PyThread_get_thread_native_id(void)
> {
> if (!initialized)
> PyThread_init_thread();
> #ifdef __APPLE__
> uint64_t native_id;
> (void) pthread_threadid_np(NULL, &native_id);
> #elif defined(__linux__)
> pid_t native_id;
> native_id = syscall(SYS_gettid);
> #elif defined(__FreeBSD__)
> int native_id;
> native_id = pthread_getthreadid_np();
> #elif defined(__OpenBSD__)
> pid_t native_id;
> native_id = getthrid();
> #elif defined(_AIX)
> tid_t native_id;
> native_id = thread_self();
> #elif defined(__NetBSD__)
> lwpid_t native_id;
> native_id = _lwp_self();
> #endif
> return (unsigned long) native_id;
> }
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
>
> On Fri, Apr 16, 2021 at 10:47 PM Gabriele  wrote:
> >
> > Hi all.
> >
> > My apologies if this is a topic that's been discussed already, but I
> > wasn't able to locate anything in the archive on the subject. I was
> > wondering if there's a fundamental reason for using
> > PyThread_get_thread_ident instead of PyThread_get_thread_native_id for
> > the value of the thread_id field of the PyThreadState object.
> >
> > The reason why I'm asking is that I would like to have easy access to
> > the native TID on Linux to identify the /proc/stat file associated
> > with the thread from an external process. At the moment I have to
> > resort to calling process_vm_readv to copy the struct pthread over to
> > local VM and then guess where the tid field might be. So, if there's
> > no fundamental reason for thread_id to be PyThread_get_thread_ident, I
> > would like to propose to change it to PyThread_get_thread_native_id
> > instead.
> >
> > Thanks,
> > Gabriele
> >
> > --
> > "Egli è scritto in lingua matematica, e i caratteri son triangoli,
> > cerchi, ed altre figure
> > geometriche, senza i quali mezzi è impossibile a intenderne umanamente
> parola;
> > senza questi è un aggirarsi vanamente per un oscuro laberinto."
> >
> > -- G. Galilei, Il saggiatore.
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/3DOB6VUTAIJKK4SUGBYWL4QPWI2E5Q2T/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZSXLNWY3HIWYEHV7Z2UKHHSUEVTKYBQY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Memory use of PEP 563 vs PEP 649

[Mark Shannon]

Hi,

There has been some discussion on this mailing list about the memory use 
of PEP 563 vs PEP 649.


It doesn't matter.

The memory use of either is small, and any measurements are merely 
measuring artifacts of the current implementations, and the current 
on-disk representation of code objects.
In an ideal implementation, of either PEP, the underlying data for 
`__annotations__` will sit on disk at zero cost in memory and load time.


There are much more important differences between the two PEPs.

I support keeping the 3.9 behavior for 3.10 to give us time to resolve 
the discussion and maybe for us to implement some of the above "ideal 
implementation".


Cheers,
Mark.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3DF5P2SRAYIYSDWG7GXPUUSRAXCXGJ4M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Eric V. Smith]

On 4/17/2021 12:28 AM, Christopher Barker wrote:

I wonder if anyone has considered the impact of PEP 563 on dataclasses ?

I have!


I did find this SO post:

https://stackoverflow.com/questions/52171982/new-annotations-break-inspection-of-dataclasses 



which is related, but not quite the same -- THAT issue was fixed.

My issue does show up in this BPO:

https://bugs.python.org/issue39442 

Somewhere in the middle of that thread, Eric wrote: "At this point, 
this seems more like fodder for python-ideas."


Then there's a bit more to the thread, which peters out without 
resolution.


On to the issue:

dataclasses may be the only part of the stdlib that uses annotations.

dataclasses store the annotations in Field objects type attribute, in 
the __dataclass_fields__ attribute.


We can see that with this code:

@dataclass
class Simple:
    x : int
    y : float
    name : str

s = Simple(3, 4.0, "fred")

print("The field types:")
for f in s.__dataclass_fields__.values():
    print(f"name: {f.name }, type: {f.type}, type of 
type: {type(f.type)}")


Which results in:

The field types:
name: x, type: , type of type: 
name: y, type: , type of type: 
name: name, type: , type of type: 

with:

from __future__ import annotations

The result is:

The field types:
name: x, type: int, type of type: 
name: y, type: float, type of type: 
name: name, type: str, type of type: 

This of course is completely as expected.

I have no idea how dataclasses uses the Field type attribute -- as far 
as I can tell, for nothing at all. However, it is there, and it is 
called "type", rather than say, "annotation".
In retrospect, that field probably should have been named "annotation". 
Regardless, the intent was always "store what's in 
__annotations__[field_name]", or what the user specified in field(..., 
type=whatever, ...).


And I have a whole pile of code that fully expects the Fields' type 
attribute to be an actual type object that I can call to crate an 
instance of that type (or call a method on it, which is what I am 
actually doing)


So my code will very much break with this change.
True, unfortunately. To be clear to everyone not paying close attention, 
"this change" is PEP 563.


I fully understand that the __dataclass_fields__ attribute was 
probably never intended to be part of the public API, so I get what I 
deserve.


However, the Field object is documented, as such:

"""
class dataclasses.Field

Field objects describe each defined field. These objects are created 
internally, and are returned by the fields() module-level method (see 
below). Users should never instantiate a Field object directly. Its 
documented attributes are:


name: The name of the field.
type: The type of the field.
default, default_factory, init, repr, hash, compare, and metadata have 
the identical meaning and values as they do in the field() declaration.


Other attributes may exist, but they are private and must not be 
inspected or relied on.

"""

That last sentence implies that the type attribute CAN be inspected 
and relied upon, which is what I am doing.
Yes, Field.type is very much part of the public dataclasses API as 
available through dataclasses.fields(), not through 
cls.__dataclass_fields__.


And I haven't tried to solve this problem in my use case, but I'm not 
sure it can be done -- when I get around to inspecting the type of the 
Field objects, I have no idea what namespace they are in -- so I can't 
reconstruct them from the string. I really need the type object itself.
@dataclass pretty much has the same problem with respect to calling 
typing.get_type_hints().


So I'll probably need to re-write much of the dataclasses decorator, 
to call eval() early -- though even then I'm not sure I'll have access 
to the proper namespace.


Anyway -- one more data point:  PEP 563 changes the (semi-public?) API 
of dataclasses.


Though *maybe* that could be addressed with a dataclasses update -- 
again, I've only started to think about it -- there was some 
discussion of that in the BPO, though Eric didn't seem particularly 
interested.


I still think it's working as intended: it uses what's in 
__annotations__ (or passed in to field()). As everyone who has tried to 
call typing.get_type_hints() knows, it's hard to get right, if it's even 
possible, because, as you say "when I get around to inspecting the type 
of the Field objects, I have no idea what namespace they are in". My 
opinion is that the person who wants a real type object probably has a 
better idea of that namespace than @dataclass does, but there's a very 
good chance they don't know, either.


@dataclass goes out of its way to not call typing.get_type_hints(). The 
original reason for this is not wanting to force typing to be imported, 
if it wasn't already being used. That may have been addressed with PEP 
560, but 

[Python-Dev] f_lasti not correct with PREDICT (is it fixable?)

[Fabio Zadrozny]

Hi all,

I currently have a use-case where I need to rely on `f_lasti`, but it seems
that it's not really possible to rely on it as it doesn't accurately point
to the actual bytecode last executed when PREDICT is used.

So, I'd like to know: is it possible to change that so that the `f_lasti`
is indeed properly updated (even when PREDICT) is used (to actually match
what the docs say about it: https://docs.python.org/3/library/inspect.html)?

As a note, my use case is doing a step into a function in a debugger.

This is needed in order to know whether a given function call maps the
target being step into.

For instance, given something as:

def method(c):
return c+1

a = method
a(a(a(1)))

The user can choose to step into the `a` at the middle. The only
conceivable way out I found to implement it reliably was by identifying
that when the user enters the `method()` function the f_lasti on the parent
frame is the one which would result in that function call.

So, I have this working on a POC, but to have it working on all the cases
I'm having to handle all existing bytecode instructions and then going to
ceval.c and then identify which ones have a PREDICT and then having to do
the adjustments to match the `f_lasti` accordingly when the function call
is reached (which is a bit of an herculean task to support multiple
versions of CPython as that's definitely an implementation detail), so, I'd
really be interested in having the f_lasti work reliably.

Thanks,

Fabio
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HYTVMG4YTU3QLRGEE4LMLGFANQPO4PD3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 563 in light of PEP 649

[Larry Hastings]

Obviously that's a bug.  Can you send me this test case?  Anything 
works--Github, private email, whatever is most convenient for you.  
Thank you!



//arry/

On 4/16/21 11:22 PM, Inada Naoki wrote:

## memory error on co_annotations

I modifled py_compile to add `from __future__ import co_annotations`
automatically.

```
$ ../co_annotations/python -m compileall mypy
Listing 'mypy'...
Compiling 'mypy/checker.py'...
free(): corrupted unsorted chunks
Aborted

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x77c73859 in __GI_abort () at abort.c:79
#2  0x77cde3ee in __libc_message
(action=action@entry=do_abort, fmt=fmt@entry=0x77e08285 "%s\n") at
../sysdeps/posix/libc_fatal.c:155
#3  0x77ce647c in malloc_printerr
(str=str@entry=0x77e0a718 "free(): corrupted unsorted chunks") at
malloc.c:5347
#4  0x77ce81c2 in _int_free (av=0x77e39b80 ,
p=0x55d1db30, have_lock=) at malloc.c:4356
#5  0x55603906 in PyMem_RawFree (ptr=) at
Objects/obmalloc.c:1922
#6  _PyObject_Free (ctx=, p=) at
Objects/obmalloc.c:1922
#7  _PyObject_Free (ctx=, p=) at
Objects/obmalloc.c:1913
#8  0x5567caa9 in compiler_unit_free (u=0x55ef0fd0) at
Python/compile.c:583
#9  0x5568aea5 in compiler_exit_scope (c=0x7fffc3d0) at
Python/compile.c:760
#10 compiler_function (c=0x7fffc3d0, s=,
is_async=0) at Python/compile.c:2529
#11 0x5568837d in compiler_visit_stmt (s=,
c=0x7fffc3d0) at Python/compile.c:3665
#12 compiler_body (c=c@entry=0x7fffc3d0, stmts=0x56222450) at
Python/compile.c:1977
#13 0x55688e51 in compiler_class (c=c@entry=0x7fffc3d0,
s=s@entry=0x56222a60) at Python/compile.c:2623
#14 0x55687ce3 in compiler_visit_stmt (s=,
c=0x7fffc3d0) at Python/compile.c:3667
#15 compiler_body (c=c@entry=0x7fffc3d0, stmts=0x563014c0) at
Python/compile.c:1977
#16 0x5568db00 in compiler_mod (filename=0x772e6770,
mod=0x563017b0, c=0x7fffc3d0) at Python/compile.c:2001
```



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CJ3DQG3NUZ6P73BTQ27NALCOGUMYSHS7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Jelle Zijlstra]

El vie, 16 abr 2021 a las 21:32, Christopher Barker ()
escribió:

> ...
> dataclasses may be the only part of the stdlib that uses annotations.
>
>
> There's another one, actually: functools.singledispatch, at
https://github.com/python/cpython/blob/3.9/Lib/functools.py#L860. It uses
the annotations on the first argument to an `@register`ed function to
figure out how to dispatch. Currently it uses `typing.get_type_hints` to
get the type. Presumably, this would break with PEP 563 semantics if you
try to create a singledispatch function in a nested namespace.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BGC5WTYMO5EQBBNNR2YNZRJZHX5O3AH6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP-0467: Minor API improvements for binary sequences

[Nick Coghlan]

Thanks for picking this back up!

The deferral section can go away now that you're actively working on it
again, and +1 from me on the resolution of the previously open questions
(although I wouldn't be particularly upset if the SC considered bchr
redundant, given that "bchr = bytes.fromord" is a trivial alias in cases
where the shorter spelling is easier to read).

Cheers,
Nick.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6YQOPMNXINMFFU572EYXTV6SVWKBT2SQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Nick Coghlan]

On Fri, 16 Apr 2021, 3:14 pm Larry Hastings,  wrote:

>
> Anyway I assume it wasn't "fixable".  The compiler would presumably
> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
> would be cheaper every time for a global or builtin.  The fact that it
> already doesn't do so implies that it can't.
>

Metaclass __prepare__ methods can inject names into the class namespace
that the compiler doesn't know about, so yeah, it unfortunately has to be
conservative and use LOAD_NAME in class level code.

Cheers,
Nick.

>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZJYDHWJNMMMICUE32M3O7DGMSMVIOQ3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Jelle Zijlstra]

El sáb, 17 abr 2021 a las 8:30, Nick Coghlan ()
escribió:

>
>
> On Fri, 16 Apr 2021, 3:14 pm Larry Hastings,  wrote:
>
>>
>> Anyway I assume it wasn't "fixable".  The compiler would presumably
>> already prefer to generate LOAD_GLOBAL vs LOAD_NAME, because LOAD_GLOBAL
>> would be cheaper every time for a global or builtin.  The fact that it
>> already doesn't do so implies that it can't.
>>
>
> Metaclass __prepare__ methods can inject names into the class namespace
> that the compiler doesn't know about, so yeah, it unfortunately has to be
> conservative and use LOAD_NAME in class level code.
>
> But of course, most metaclasses don't. I wonder if there are cases where
the compiler can statically figure out that there are no metaclass
shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
when the class has no base classes and no metaclass=.


> Cheers,
> Nick.
>
>>
>> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/IZJYDHWJNMMMICUE32M3O7DGMSMVIOQ3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/W4LYALS5RMLTGIU5PE5YMF4L3MWL2HXY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Nick Coghlan]

On Sat, 17 Apr 2021, 3:51 am ,  wrote:

> Guys, the issue is that I most of the time see that somebody used C++ for
> one or two times, did not understand it and left with bad taste ...
>

I've got more than a decade and a half of experience with both C++ (dating
back to the relatively low quality VC6 C++ runtime) and CPython (getting on
towards 20 years now), and the value of adding a C++ runtime to the CPython
runtime would be vastly smaller than the value of adding it to an arbitrary
C project.

If folks want rich data structures in the CPython implementation, we want
them using the Python builtin types, not the C++ STL containers. If they
want a type hierarchy, we want them using Python types, not C++ types. If
they want automatic resource management, then we want them working out how
to lift the affected code out of C and into Python (for example, the import
system rewrite).

In a lot of ways, CPython's C API can be viewed as one of the many
competing approaches to enabling "C-with-objects" programming, just like
C++.

There are certainly nice features in modern C++ that make it easier to
scale to large projects than vanilla C code, but the bulk of the CPython
code base has never really been vanilla C code - it has always been able to
rely on the builtin Python containers and type hierarchy for a lot of the
heavy lifting.

And it is already a bigger barrier to entry than we would like to ask
potential contributors to the lower level code to learn two object models
(the Python one and the much simpler C one), let alone if we were to start
asking them to learn a 3rd one (C++) that has a deserved reputation as one
of the most complex and error prone object models in widespread use.

Cheers,
Nick.





>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SQKSB4OYXL5FPS72BNUJQO2ALRPW6YSX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 649: Deferred Evaluation Of Annotations Using Descriptors, round 2

[Nick Coghlan]

On Sun, 18 Apr 2021, 1:59 am Jelle Zijlstra, 
wrote:

> El sáb, 17 abr 2021 a las 8:30, Nick Coghlan ()
> escribió:.
>
>>
>> Metaclass __prepare__ methods can inject names into the class namespace
>> that the compiler doesn't know about, so yeah, it unfortunately has to be
>> conservative and use LOAD_NAME in class level code.
>>
>> But of course, most metaclasses don't. I wonder if there are cases where
> the compiler can statically figure out that there are no metaclass
> shenanigans going on, and emit LOAD_GLOBAL anyway. It seems safe at least
> when the class has no base classes and no metaclass=.
>

Aye, that particular case is one the symtable pass could at least
theoretically identify.

As soon as there is a name to resolve in the class header, though, it's no
longer safe for the compiler to make assumptions :(

Cheers,
Nick.

>


>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NDYKJTWUN4HT2BPNJVBUKYVD6HOTIBMY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Antoine Pitrou]

On Sun, 18 Apr 2021 02:13:57 +1000
Nick Coghlan  wrote:
> 
> If
> they want automatic resource management, then we want them working out how
> to lift the affected code out of C and into Python (for example, the import
> system rewrite).

There's a significant amount of wishful thinking here.  When CPython
code gets written in C, it's often because it's presumed to be
performance-critical.  The import system was indeed rewritten in
Python... but some parts were then written back in C (see
Python/import.c) to fix performance regressions.

> In a lot of ways, CPython's C API can be viewed as one of the many
> competing approaches to enabling "C-with-objects" programming, just like
> C++.

It's a clumsy API to use *from C*.  Witness the inconsistent behaviour
of those APIs wrt. borrowed references, for example, or the fact that
forgetting an INCREF or DECREF can lead to occasional leaks or crashes,
or the delicate task of teaching the cyclic GC about a particular type,
or all the special cases where those APIs deviate slightly in semantics
from their pure Python equivalents, or the fact that APIs are added in
adhoc manner, leading to an inconsistent set of primitives...
(extracting a C long from a Python object works a bit differently from
extracting a C unsigned long or long long... which then requires the
caller to learn about those subtle differences and work around them).

CPython's C API is probably fine for consumption by intermediate layers
such as Cython. It's a maze to navigate for direct use.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QJQHB2CLWMAV5DNZCRV7TTBR2CSKUPYN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[jacob . r . hayes]

Related, `inspect.Parameter.annotation` is affected too, but at least this 
attribute is called `annotation` instead of `type`.

I noticed this today with `multipledispatch` (though 
[reported](https://github.com/mrocklin/multipledispatch/issues/104) in 2019) 
and some other internal code, both using `inspect.signature`. 
`multipledispatch` could reasonably swap to `typing.get_type_hints`, but the 
other code I was working with also used `Parameter.default` and 
`Parameter.kind`, so would require both `inspect.signature` and 
`get_type_hints`. Not the worst and arguably necessary now/pre PEP 563 with 
explicit string annotations, but the PEP 649 semantics would be a welcome 
improvement.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RW7NUODC5YLSEXDNE4DBN6TGFGMGVF4L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Christopher Barker]

On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith  wrote:

> I wonder if anyone has considered the impact of PEP 563 on dataclasses ?
>
> I have!
>

Thanks Eric.


> In retrospect, that field probably should have been named "annotation".
> Regardless, the intent was always "store what's in
> __annotations__[field_name]", or what the user specified in field(...,
> type=whatever, ...).
>

intent is all well and good, but what was the intent of __annotations__
In the first place. Static typing was certainly one of the intents, but as
it DID store a value, not a string from the beginning, we can only assume
that there was some intent that the value might be used.

As for dataclasses -- why store it at all? It seems that dataclasses use
the fact that a class attribute has an annotation to determine what to add
to the field -- which is a nifty use of teh syntax. But presumably the
intent of storing the annotation in the Field object, and providing a
public attribute for it, was so that it could be used.

I suppose your point is that the type attribute stored the annotation, and
PEP 653 changes what the annotation will be so there's nothing specific at
all about dataclasses here. Which is true. But users of dataclasses may
have been less aware of all the implications as they didn't write that
annotation etrating code themselves. I know I wasn't.

> In any event, all of this mess is one reason I'd like to see PEP 649 get
> accepted:
>
Indeed -- that is the title of this thread, after all :-)

And see others' notes, there seems to be two other places in the stdlib
that will be affected.

-CHB


-- 
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3ORLF6IBGMXJMBC2FDGYKBHU445ILK7E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How about using modern C++ in development of CPython ?

[Nick Coghlan]

On Sun, 18 Apr 2021, 2:47 am Antoine Pitrou,  wrote:

> On Sun, 18 Apr 2021 02:13:57 +1000
> Nick Coghlan  wrote:
> >
> > If
> > they want automatic resource management, then we want them working out
> how
> > to lift the affected code out of C and into Python (for example, the
> import
> > system rewrite).
>
> There's a significant amount of wishful thinking here.  When CPython
> code gets written in C, it's often because it's presumed to be
> performance-critical.  The import system was indeed rewritten in
> Python... but some parts were then written back in C (see
> Python/import.c) to fix performance regressions.
>


Aye, I know, but the overall organisation of the import system still
changed to "Python with C acceleration of key parts" rather than the
previous "almost entirely C" approach.

Other implementations are now free to reuse the Python parts, and only have
to implement the accelerators separately.


> > In a lot of ways, CPython's C API can be viewed as one of the many
> > competing approaches to enabling "C-with-objects" programming, just like
> > C++.
>
> It's a clumsy API to use *from C*.  Witness the inconsistent behaviour
> of those APIs wrt. borrowed references, for example, or the fact that
> forgetting an INCREF or DECREF can lead to occasional leaks or crashes,
> or the delicate task of teaching the cyclic GC about a particular type,
> or all the special cases where those APIs deviate slightly in semantics
> from their pure Python equivalents, or the fact that APIs are added in
> adhoc manner, leading to an inconsistent set of primitives...
>

Right, but that clumsiness isn't any easier to manage from C++ than it is
from C. Things like RAII rely on consistent behaviour from the resources
being managed, and the current C API doesn't offer that.

For example, putting a borrowed or stolen reference into an RAII based
reference manager will result in an early free or a resource leak, just as
happens when you decref a borrowed reference or incref a stolen one in C.

Hence the need for wrapper APIs and tools that make the C API more C++
friendly rather than it being straightforward to use directly.

CPython's C API is probably fine for consumption by intermediate layers
> such as Cython. It's a maze to navigate for direct use.
>

That I also agree with, and hence I'd definitely be a fan if we ever
figured out a way to bootstrap Cython into the CPython build process so
that accelerated standard library modules could be written in Cython.

Unlike C++, Cython uses the same data structures and object model as Python
does, and it already smooths over the irregularities in the C API to allow
for fully automated resource management. The Python based syntax can even
help lower the barriers to entry for folks that don't already know C
(although you do still need to learn the underlying C memory model).

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N53JDKLNBQRGPSGLOK3HUIMJGD3FLSCV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Nick Coghlan]

On Mon, 12 Apr 2021, 1:48 pm Guido van Rossum,  wrote:

>
> At the very least I recommend that the SC take this into account when they
> consider PEP 649. Accepting it has some nice benefits when it comes to the
> scoping rules for annotations -- but it would forever close the door for
> the "relaxed annotation syntax" idea you brought up. (Isn't it fun to be on
> the SC. :-)
>

I may have missed someone else mentioning this, but I don't think this
concern is necessarily true, as even if PEP 649 were accepted, the only
pre-PEP-563 constraints it would reintroduce would be that all future type
annotation syntax:

* have a defined runtime effect;
* that runtime effect be consistent with normal expressions when reusing
existing syntax; and
* be explicitly quoted when using type hinting syntax from later Python
versions in code that needs to run on earlier versions

Any PEPs adding new type hinting specific syntax would be free to define
the runtime effect of the new syntax as "produces a string containing the
text of the part of the annotation using the new syntax, as if the new
syntax were explicitly quoted", even if we decided not to go ahead with the
idea of applying those "produces a string" semantics to *all* annotations.

Cheers,
Nick.


>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DLCZHSXCRBVGCXBAKG7SXI6LBFQH23X7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: In support of PEP 649

[Eric V. Smith]

On 4/17/2021 6:44 PM, Christopher Barker wrote:
On Sat, Apr 17, 2021 at 6:12 AM Eric V. Smith > wrote:



I wonder if anyone has considered the impact of PEP 563 on
dataclasses ?

I have!


Thanks Eric.

In retrospect, that field probably should have been named
"annotation". Regardless, the intent was always "store what's in
__annotations__[field_name]", or what the user specified in
field(..., type=whatever, ...).


intent is all well and good, but what was the intent of 
__annotations__ In the first place. Static typing was certainly one of 
the intents, but as it DID store a value, not a string from the 
beginning, we can only assume that there was some intent that the 
value might be used.
Yes, but wouldn't that say that PEP 563 could never be implemented then, 
since it would break that intent? We already knew it broke all code that 
was expecting a type object in __annotations__. I'm just saying my 
expectation is 563 will break any code expecting a type object in 
__annotations__, or anything that's derived from __annotations__ 
(specifically in my case, dataclass users looking at Field.type). My 
probably unarticulated intent was that Field.type agree with 
__annotations__[field_name].


As for dataclasses -- why store it at all? It seems that dataclasses 
use the fact that a class attribute has an annotation to determine 
what to add to the field -- which is a nifty use of teh syntax. But 
presumably the intent of storing the annotation in the Field object, 
and providing a public attribute for it, was so that it could be used.


I didn't want to predict what someone was using dataclasses for. 
Especially someone using field(..., metadata=something) might want 
access to any of the field() values, including .type. I supposed I could 
have said "for defaults, default_factory, repr, etc., look at the Field 
object, but for the type look at __annotations__[field_name]", but I 
decided it would be easier if everything about a field would be in the 
Field object. Not that it makes much difference: I believe that 
Field.type always agrees with __annotations__[field_name].


This allows someone to say (where some_dataclass_field_object is a Field):

process_field(some_dataclass_field_object)

instead of this, if the .type field weren't present:

process_field(some_dataclass_field_object, 
SomeDataclass.__annotations__[some_dataclass_field_object.name])




I suppose your point is that the type attribute stored the annotation, 
and PEP 653 changes what the annotation will be so there's nothing 
specific at all about dataclasses here. Which is true. But users of 
dataclasses may have been less aware of all the implications as they 
didn't write that annotation etrating code themselves. I know I wasn't.


In any event, all of this mess is one reason I'd like to see PEP
649 get accepted:

Indeed -- that is the title of this thread, after all :-)


Well, not everyone is not in support of the subject!

Eric



And see others' notes, there seems to be two other places in the 
stdlib that will be affected.


-CHB


--
Christopher Barker, PhD (Chris)

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython


--
Eric V. Smith

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JZALUYHIVQHM5GFZN7PIRS6G3CVNPYXX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] PEP 563 and 649: The Great Compromise

[Larry Hastings]

The heart of the debate between PEPs 563 and 649 is the question: what 
should an annotation be?  Should it be a string or a Python value?  It 
seems people who are pro-PEP 563 want it to be a string, and people who 
are pro-PEP 649 want it to be a value.


Actually, let me amend that slightly.  Most people who are pro-PEP 563 
don't actually care that annotations are strings, per se.  What they 
want are specific runtime behaviors, and they get those behaviors when 
PEP 563 turns their annotations into strings.


I have an idea--a rough proposal--on how we can mix together aspects of 
PEP 563 and PEP 649.  I think it satisfies everyone's use cases for both 
PEPs.  The behavior it gets us:


 * annotations can be stored as strings
 * annotations stored as strings can be examined as strings
 * annotations can be examined as values


The idea:

We add a new type of compile-time flag, akin to a "from __future__" 
import, but not from the future.  Let's not call it "from __present__", 
for now how about "from __behavior__".


In this specific case, we call it "from __behavior__ import 
str_annotations".  It behaves much like Python 3.9 does when you say 
"from __future__ import annotations", except: it stores the dictionary 
with stringized values in a new member on the function/class/module 
called "__str_annotations__".


If an object "o" has "__str_annotations__", set, you can access it and 
see the stringized values.


If you access "o.__annotations__", and the object has 
"o.__str_annotations__" set but "o.__annotations__" is not set, it 
builds (and caches) a new dict by iterating over o.__str_annotations__, 
calling eval() on each value in "o.__str_annotations__".  It gets the 
globals() dict the same way that PEP 649 does (including, if you compile 
a class with str_annotations, it sets __globals__ on the class).  It 
does /not/ unset "o.__str_annotations__" unless someone explicitly sets 
"o.__annotations__".  This is so you can write your code assuming that 
"o.__str_annotations__" is set, and it doesn't explode if somebody 
somewhere ever looks at "o.__annotations__".  (This could lead to them 
getting out of sync, if someone modified "o.__annotations__".  But I 
suspect practicality beats purity here.)


This means:

 * People who only want stringized annotations can turn it on, and only
   ever examine "o.__str_annotations__".  They get the benefits of PEP
   563: annotations don't have to be valid Python values at runtime,
   just parseable.  They can continue doing the "if TYPE_CHECKING:"
   import thing.
 * Library code which wants to examine values can examine
   "o.__annotations__".  We might consider amending library functions
   that look at annotations to add a keyword-only parameter,
   "str_annotations=False", and if it's true it uses
   o.__str_annotations__ instead etc etc etc.


Also, yes, of course we can keep the optimization where stringized 
annotations are stored as a tuple containing an even number of strings.  
Similarly to PEP 649's automatic binding of an unbound code object, if 
you set "o.__str_annotations__" to a tuple containing an even number of 
strings, and you then access "o.__str_annotations__", you get back a dict.



TBD: how this interacts with PEP 649.  I don't know if it means we only 
do this, or if it would be a good idea to do both this and 649.  I just 
haven't thought about it.  (It would be a runtime error to set both 
"o.__str_annotations__" and "o.__co_annotations__", though.)



Well, whaddya think?  Any good?


I considered calling this "PEP 1212", which is 563 + 649,


//arry/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WUZGTGE43T7XV3EUGT6AN2N52OD3U7AE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Relaxing the annotation syntax

[Guido van Rossum]

Hm, I was specifically thinking of things that introduce new keywords. For
example, TypeScript adds unary operators 'infer' and 'keyof'. It would be
rather difficult to have to define those as soft keywords throughout the
language. (We couldn't just make them unary keywords, since 'infer (x)'
should continue to call the function 'infer', for example. In an annotation
context that might not be a problem, since function calls in general aren't
valid types.)

IIRC Jukka also already brought up the possibility of using something like
'(int) => str' instead of 'Callable[[int], str]' -- but it would be
unpleasant if that syntax had a meaning like you propose outside
annotations.

On Sat, Apr 17, 2021 at 7:12 PM Nick Coghlan  wrote:

>
>
> On Mon, 12 Apr 2021, 1:48 pm Guido van Rossum,  wrote:
>
>>
>> At the very least I recommend that the SC take this into account when
>> they consider PEP 649. Accepting it has some nice benefits when it comes to
>> the scoping rules for annotations -- but it would forever close the door
>> for the "relaxed annotation syntax" idea you brought up. (Isn't it fun to
>> be on the SC. :-)
>>
>
> I may have missed someone else mentioning this, but I don't think this
> concern is necessarily true, as even if PEP 649 were accepted, the only
> pre-PEP-563 constraints it would reintroduce would be that all future type
> annotation syntax:
>
> * have a defined runtime effect;
> * that runtime effect be consistent with normal expressions when reusing
> existing syntax; and
> * be explicitly quoted when using type hinting syntax from later Python
> versions in code that needs to run on earlier versions
>
> Any PEPs adding new type hinting specific syntax would be free to define
> the runtime effect of the new syntax as "produces a string containing the
> text of the part of the annotation using the new syntax, as if the new
> syntax were explicitly quoted", even if we decided not to go ahead with the
> idea of applying those "produces a string" semantics to *all* annotations.
>
> Cheers,
> Nick.
>
>
>>
>>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6MY3UBK2J5G2WTKYEG7IWPLLCEVJMIGT/
Code of Conduct: http://python.org/psf/codeofconduct/