[Python-Dev] [RELEASE] Python 3.11.0a1 is available

[Pablo Galindo Salgado]

Now that we are on a release spree, here you have the first alpha release of
Python 3.11: Python 3.11.0a1. Let the testing and validation games begin!

https://www.python.org/downloads/release/python-3110a1/

*Major new features of the 3.11 series, compared with 3.10*

Python 3.11 is still in development. This release, 3.11.0a1 is the first of
six planned alpha releases.

Alpha releases are intended to make it easier to test the current state of
new features and bug fixes and to test the release process.

During the alpha phase, features may be added up until the start of the
beta phase (2022-05-06) and, if necessary, may be modified or deleted up
until the release candidate phase (2022-08-01). Please keep in mind that
this is a preview release and its use is not recommended for production
environments.

Many new features for Python 3.11 are still being planned and written.
Among the new major new features and changes so far:

   - PEP 657  – Include
   Fine-Grained Error Locations in Tracebacks
   - PEP 654  – PEP 654 –
   Exception Groups and except*
   - (Hey, fellow core developer, if a feature you find important is
   missing from this list, let Pablo know .)

The next pre-release of Python 3.11 will be 3.11.0a2, currently scheduled
for 2021-11-02.
*And now for something completely different*

Zero-point energy is the lowest possible energy that a quantum mechanical
system may have. Unlike in classical mechanics, quantum systems constantly
fluctuate in their lowest energy state as described by the Heisenberg
uncertainty principle. As well as atoms and molecules, the empty space of
the vacuum has these properties. According to quantum field theory, the
universe can be thought of not as isolated particles but as continuous
fluctuating fields: matter fields, whose quanta are fermions (i.e., leptons
and quarks), and force fields, whose quanta are bosons (e.g., photons and
gluons). All these fields have a non zero amount of energy called
zero-point energy. Physics currently lacks a full theoretical model for
understanding zero-point energy; in particular, the discrepancy between
theorized and observed vacuum energy is a source of major contention


*We hope you enjoy those new releases!*
Thanks to all of the many volunteers who help make Python Development and
these releases possible! Please consider supporting our efforts by
volunteering yourself or through organization contributions to the Python
Software Foundation.

Regards from cloudy London,

Your friendly release team,
Pablo Galindo @pablogsal
Ned Deily @nad
Steve Dower @steve.dower
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/T4YIZDQWU23RQVPWAINIHUDBPGR7N3YQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] RFC on Callable Type Syntax

[S Pradeep Kumar]

Hello all,

Typing-sig has been discussing user-friendly syntax for the type used to
represent callables. [1] Since this affects the Python language syntax, we
wanted to get some high-level feedback from you before putting up a
detailed PEP.

TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) ->
bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are
there concerns we need to keep in mind about such a syntax change? 2.
Should we propose syntax for additional features in the same PEP or in a
future PEP?


# Motivation

Guido has pointed out that `Callable` is one of the most frequently used
type forms but has the least readable syntax. For example, say we have a
callback `formatter` that accepts a record and a list of permissions:

```python
def print_purchases(
user,
formatter,  # <-- callback
):
<...>
output = formatter(record, permissions)
print(output)
```

To give it a type, we currently have to write:

```python
from typing import Callable

def print_purchases(
user: User,
formatter: Callable[[PurchaseRecord, List[AuthPermission]],
FormattedItem]  # Hard to read.
) -> None:

<...>
output = formatter(record, permissions)
print(output)
```

`Callable` can be hard to understand for new users since it doesn't
resemble existing function signatures and there can be multiple square
brackets. It also requires an import from `typing`, which is inconvenient.
Around 40% of the time [2], users just give up on precisely typing the
parameters and return type of their callbacks and just leave it as a blank
Callable. In other cases, they don't add a type for their callback at all.
Both mean that they lose the safety guarantees from typing and leave room
for bugs.

We believe that adding friendly syntax for Callable types will improve
readability and encourage users to type their callbacks more precisely.
Other modern, gradually-typed languages like TypeScript (JS), Hack (PHP),
etc. have special syntax for Callable types.

(As a precedent, PEP 604 recently added clean, user-friendly syntax for the
widely-used `Union` type. Instead of importing `Union` and writing `expr:
Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr |
SubtractExpr`.)


# Proposal and Questions

We have a two-part proposal and questions for you:

1. Syntax to replace Callable

After a lot of discussion, there is strong consensus in typing-sig about
adding syntax to replace Callable. So, the above example would be written
as:
```python
def print_purchases(
user: User,
formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem,
) -> None:

<...>
output = formatter(record, permissions)
print(output)
```
This feels more natural and succinct.

Async callbacks currently need to be written as
```
from typing import Callable

async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]]
```

With the new syntax, they would be written as
```
async_callback: async (HttpRequest) -> HttpResponse
```
which again seems more natural. There is similar syntax for the type of
decorators that pass on *args and **kwargs to the decorated function.

Note that we considered and rejected using a full def-signature syntax like

(record: PurchaseRecord, permissions: List[AuthPermission], /) ->
FormattedItem

because it would be more verbose for common cases and could lead to subtle
bugs; more details in [3].

The Callable type is also usable as an expression, like in type aliases
`IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.

**Question 1**: Are there concerns we should keep in mind about such a
syntax proposal?



2. Syntax for callback types beyond Callable

`Callable` can't express the type of all possible callbacks. For example,
it doesn't support callbacks where some parameters have default values:
`formatter(record)` (the user didn't pass in `permissions`). It *is*
possible to express these advanced cases using Callback Protocols (PEP 544)
[4] but it gets verbose.

There are two schools of thought on typing-sig on adding more syntax on top
of (1):

(a) Some, including Guido, feel that it would be a shame to not have syntax
for core Python features like default values, keyword arguments, etc.

One way to represent default values or optionally name parameters would
be:

```
# permissions is optional
formatter: (PurchaseRecord, List[AuthPermission]=...) -> FormattedItem

# permissions can be called using a keyword argument.
formatter: (PurchaseRecord, permissions: List[AuthPermission]) ->
FormattedItem
```

There are also alternative syntax proposals.

(b) Others want to wait till we have more real-world experience with the
syntax in (1).

The above cases occur < 5% of the time in typed or untyped code [5].
And the syntax in (1) is forward-compatible with the additional proposals.
So we could add them later if needed or leave them out, since we can always
use callback protocols.

**Questi

[Python-Dev] Python multithreading without the GIL

[Sam Gross]

Hi,

I've been working on changes to CPython to allow it to run without the
global interpreter lock. I'd like to share a working proof-of-concept that
can run without the GIL. The proof-of-concept involves substantial changes
to CPython internals, but relatively few changes to the C-API. It is
compatible with many C extensions: extensions must be rebuilt, but usually
require small or no modifications to source code. I've built compatible
versions of packages from the scientific Python ecosystem, and they are
installable through the bundled "pip".

Source code:
https://github.com/colesbury/nogil

Design overview:
https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit

My goal with the proof-of-concept is to demonstrate that removing the GIL
is feasible and worthwhile, and that the technical ideas of the project
could serve as a basis of such an effort.

I'd like to start a discussion about these ideas and gauge the community's
interest in this approach to removing the GIL.

Regards,
Sam Gross
colesb...@gmail.com / sgr...@fb.com
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Chris Angelico]

On Fri, Oct 8, 2021 at 1:45 PM S Pradeep Kumar  wrote:
> The Callable type is also usable as an expression, like in type aliases 
> `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
>
> **Question 1**: Are there concerns we should keep in mind about such a syntax 
> proposal?
>

Either I'm reading this wrongly, or this is oddly inconsistent with
tuples. With a tuple, it's the comma that defines it; with function
parameters here, it looks like the parentheses are mandatory, and they
are what define it?

Which of these will be valid?

x = int -> int
y = int, int -> int # tuple containing (int, Callable) or function
taking two args?
z = (int,) -> int

Given the examples you've shown, my intuition is that it should match
def statement syntax - namely, the args are always surrounded by
parentheses, a trailing comma is legal but meaningless, and it has
nothing to do with tuple display. It would be very convenient for
one-arg functions to be able to be defined the way x is, and I suspect
people will try it, but it introduces annoying ambiguities.

There are parallel proposals to support a lambda syntax like "x =>
x.spam" as equivalent to "lambda x: x.spam", so I'm curious what the
rules would be for the two types of arrows, and whether they'd be
consistent with each other.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KIQQEIOC2D4INNAAIYWC6HH7GXPZWLXH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Chris Angelico]

On Fri, Oct 8, 2021 at 1:51 PM Sam Gross  wrote:
>
> Hi,
>
> I've been working on changes to CPython to allow it to run without the global 
> interpreter lock. I'd like to share a working proof-of-concept that can run 
> without the GIL. The proof-of-concept involves substantial changes to CPython 
> internals, but relatively few changes to the C-API. It is compatible with 
> many C extensions: extensions must be rebuilt, but usually require small or 
> no modifications to source code. I've built compatible versions of packages 
> from the scientific Python ecosystem, and they are installable through the 
> bundled "pip".
>
> Source code:
> https://github.com/colesbury/nogil
>
> Design overview:
> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit
>
> My goal with the proof-of-concept is to demonstrate that removing the GIL is 
> feasible and worthwhile, and that the technical ideas of the project could 
> serve as a basis of such an effort.
>

Thanks for doing this, and thanks for the detailed writeup! I'd like
to offer a perspective from observing the ongoing project of a brother
of mine; he does not have the concurrency experience that I have, and
it's been instructive to see what he has trouble with. For reference,
the project involves GTK (which only works on the main thread),
multiple threads for I/O (eg a socket read/parse/process thread), and
one thread managed by asyncio using async/await functions.

At no point has he ever had a problem with performance, because the
project is heavily I/O based, spending most of its time waiting for
events. So this is not going to touch on the question of
single-threaded vs multi-threaded performance.

To him, an async function and a thread function are exactly
equivalent. He doesn't think in terms of yield points or anything;
they are simply two ways of doing parallelism and are, to his code,
equivalent.

Mutable shared state is something to get your head around with *any*
sort of parallelism, and nothing will change that. Whether it's
asyncio, GUI callbacks, or actual threads, the considerations have
been exactly the same. Threads neither gain nor lose compared to other
options.

Not being a low-level programmer, he has, I believe, an inherent
assumption that any operation on a built-in type will be atomic. He's
never stated this but I suspect he's assuming that. It's an assumption
that Python is never going to violate.

Concurrency is *hard*. There's no getting around it, there's no
sugar-coating it. There are concepts that simply have to be learned,
and the failures can be extremely hard to track down. Instantiating an
object on the wrong thread can crash GTK, but maybe not immediately.
Failing to sleep in one thread results in other threads stalling. I
don't think any of this is changed by different modes (with the
exception of process-based parallelism, which fixes a lot of
concurrency at the cost of explicit IPC), and the more work
programmers want their code to do, the more likely that they'll run
into this.

Glib.idle_add is really just a magic incantation to make the GUI work. :)

Spawning a thread for asyncio isn't too hard as long as you don't have
to support older Python versions sadly, not every device updated
at the same time. But in a few years, we will be able to ignore Python
versions pre-3.7.

Most likely, none of his code would be affected by the removal of the
GIL, since (as I understand it) the guarantees as seen in Python code
won't change. Will there be impact on lower-memory systems? As small
devices go, the Raspberry Pi is one of the largest, but it's still a
lot smaller than a full PC, and adding overhead to every object would
be costly (I'm not sure what the cost of local reference counting is,
but it can't be none). Threading is perfectly acceptable for a project
like this, so I'm hoping that GIL removal won't unnecessarily penalize
this kind of thread usage.

Speaking of local refcounting, how does that affect objects that get
passed as thread arguments? Initially, an object is owned by the
creating thread, which can relinquish ownership if its local refcount
drops to zero; does another thread then take it over?

I'm excited by anything that helps parallelism in Python, and very
curious to see where this effort will go. If you need a hand with
testing, I'd be happy to help out.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PGUFHTN4YY4X4OOTCXYPVEJVAWBZZS4K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Piotr Duda]

pt., 8 paź 2021 o 04:48 S Pradeep Kumar  napisał(a):
> ...
> Note that we considered and rejected using a full def-signature syntax like
> 
> (record: PurchaseRecord, permissions: List[AuthPermission], /) -> 
> FormattedItem
> 
> because it would be more verbose for common cases and could lead to subtle 
> bugs; more details in [3].

Did you considered full def-signature with optional argument names, so
the common cases would look like

(:PurchaseRecord, :List[AuthPermission]) -> FormattedItem

Bare name signatures like '(record) -> FormattedItem' could be
disallowed to prevent bugs.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/G2NONQGVY3V6OHHZ326M7MCQJYMNJFWH/
Code of Conduct: http://python.org/psf/codeofconduct/