[Python-Dev] Re: PEP 646 (Variadic Generics): final call for comments

[Matthew Rahtz via Python-Dev]

[Matthew]

> 1. The wording of the 'Multiple Type Variable Tuples: Not Allowed'
section - you're saying that we're being a bit imprecise here in saying
that we disallow multiple TypeVarTuples in a type parameter list, given
that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the
parameter list for the function f, but there it's unambiguous, so in fact
it is allowed.

[Guido]

> That looks like a syntax error. We only allow *Ts if the argument is of
the form *a. `def f(x: *Ts1)` is not allowed.
> Maybe you were thinking of `def f(x: Array[*Ts1], y: Array[*Ts2]) as the
example? That seems unambiguous since the two positional arguments are
given separately.

Sorry, yes!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3BW75IG7XINYKUNMWR35TJW2F5DGW6LQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 646 (Variadic Generics): final call for comments

[Kevin Millikin via Python-Dev]

Yes, exactly.

Specifically, the "wrong" example in section 'Multiple Type Variable
Tuples: Not Allowed' suggests that maybe what is wrong is that `Generic`
was given more than one unpacked type variable tuple.  The actual problem
is a consequence of that: `class Array` has more than one type variable
tuple as type parameters.  (But there are other ways that could happen and
all of them should be wrong.)

I think it might be good to say that explicitly in that section.

On Thu, Jan 13, 2022 at 4:18 PM Matthew Rahtz  wrote:

> Thanks also Kevin for this feedback!
>
> Good point about being careful to distinguish type parameters vs type
> arguments. If I understand correctly, you're making two points:
>
> 1. The wording of the 'Multiple Type Variable Tuples: Not Allowed' section
> - you're saying that we're being a bit imprecise here in saying that we
> disallow multiple TypeVarTuples in a type parameter list, given that in
> e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the
> parameter list for the function f, but there it's unambiguous, so in fact
> it *is* allowed.
>
> 2. Use of wrong terminology elsewhere in the PEP. Agreed.
>
> I'll draft a PR tweaking the wording to fix both these points.
>
>
> On Thu, 13 Jan 2022 at 11:28, Kevin Millikin 
> wrote:
>
>> The wording there probably should be improved.  I had a different
>> interpretation when I read that, so that suggests it needs to be clarified.
>>
>> We should ensure to draw a clear distinction between type parameters and
>> type arguments.  (Generic classes and functions are parameterized over type
>> parameters and they have a type parameter list that is implicit in the
>> syntax.  Generic classes can be explicitly instantiated by giving them type
>> arguments, and an instantiation has a (explicit or implicit) type argument
>> list.)
>>
>> So when I read:
>>
>> """
>> As of this PEP, only a single type variable tuple may appear in a type
>> parameter list:
>>
>> class Array(Generic[*Ts1, *Ts2]): ...  # Error\
>> """
>>
>> I interpreted it to mean that the error is that the type _parameters_ of
>> the generic class Array include *Ts1 and *Ts2 (not that they were used as
>> type arguments to Generic).  Similarly, this should be an error:
>>
>> class Array(dict[*Ts1], Generator[*Ts2]): ...
>>
>> even though there is only a single type variable tuple appearing in a
>> type _argument_ list.
>>
>> The reason for the restriction is that the tupling of Array's type
>> arguments is not explicit in an instantiation of Array, so we rely on this
>> restriction so that they can be unambiguously tupled.
>>
>> I don't think there is necessarily a similar restriction on a generic
>> function's type parameters, because we don't have the ability to explicitly
>> instantiate generic functions anyway.
>>
>> An alternative wording is along the lines of: "As of this PEP, only a
>> single type variable tuple may appear among a generic class's type
>> parameters."
>>
>> def foo(*args: tuple[*Ts1, *Ts2]) -> ...
>>
>> is already prohibited by "Multiple Unpackings in a Tuple: Not Allowed".
>>
>> There are three other occurrences of "type parameter list" in the PEP.
>> Two of them talk about instantiating generic type aliases and should be
>> changed to "type argument list".  The last one is less clear, I can't quite
>> parse out what it's trying to say.
>>
>> On Wed, Jan 12, 2022 at 5:04 PM Guido van Rossum 
>> wrote:
>>
>>> On Wed, Jan 12, 2022 at 4:57 AM Petr Viktorin 
>>> wrote:
>>>
 Matthew Rahtz wrote:
 > Hi everyone,
 >
 > We've got to the stage now with PEP 646 that we're feeling pretty
 happy
 > with it. So far though we've mainly been workshopping it in
 typing-sig, so
 > as PEP 1 requires we're asking for some feedback here too before
 submitting
 > it to the steering council.
 >
 > If you have time over the next couple of weeks, please take a look at
 the
 > current draft and let us know your thoughts:
 > https://www.python.org/dev/peps/pep-0646/ (Note that the final
 couple of
 > sections are out of date; https://github.com/python/peps/pull/1880
 > clarifies which grammar changes would be required, now that PEP 637
 has
 > been rejected. We also have a second PR in progress at
 > https://github.com/python/peps/pull/1881 clarifying some of the
 motivation.)
 >
 > Thanks!
 > Matthew and Pradeep

 Hi,
 I'm very late to the discussion -- I relied on the typing-sig and SC to
 handle this, but now that I'm on the SC, I no longer have that luxury :)
 This mail has my own opinions, not necessarily the SC's.


 I've read the PEP, and I quite like it! It's clear that typing-sig
 thought this through very well.
 The thing that surprised me is the proposed changes that affect more
 than typing annotations. Quite deep in the PEP, the "Grammar Changes"
 section explains the (quite exciting) change to make star-unpa

[Python-Dev] Re: PEP 646 (Variadic Generics): final call for comments

[Matthew Rahtz via Python-Dev]

*First point (indexing assignment)*

[Guido]
> Agreed. I just misremembered this, my bad! Please do the clarification
etc.

Will do.


*Second point (multiple TypeVarTuples)*

[Guido]
> I would love it for the cases where it's *not* ambiguous to just work
(once type checkers support it). I'd like the PEP to be written so as not
to disallow those. Type checkers that only support a single star even when
it's unambiguous should be considered out of compliance. (But could still
be fully compliant otherwise.) User pressure will then sort things out.
That seems better than requiring a follow-up PEP just to allow
non-ambiguous cases that follow the PEP's semantics.

I agree that would be nice, but I feel a bit worried this could result in
additional churn.

I was going to propose adding just a single sentence to the PEP along the
lines of "Multiple unpacking are ok when the result would be unambiguous",
but as you point out, there's no simple rule for deciding what would and
wouldn't be ambiguous - so we'd probably have to spell out it out with
examples in the PEP, which would a) make the PEP even longer, and b) maybe
put us at risk of the SC wanting to re-re-review.

So overall I'd strongly prefer to just add a sentence clarifying support at
the grammar level to satisfy Petr, but leave it at that. Wdyt?

To be clear, though:

> This should clearly be disallowed:
> def f(*args: *tuple[*Ts1, *Ts2])...
> This (and similar variants) should also be disallowed:
> def f(*args: *tuple[*Ts, *Ts])...
> This should be allowed:
> def f(*args: *tuple[int, *Ts, str, T])...
> But not this:
> def f(*args: *tuple[*Ts1, str, *Ts2])...

Right, I think the current wording in the PEP ("Multiple Unpackings in a
Tuple: Not Allowed") is sufficient to make it clear these are disallowed.

> It seems you are saying that this would be disallowed, even though
there's no ambiguity?
> def f(a: Array[*Ts1], b: Array[*Ts2])...

This should definitely be fine, and I don't think there's any wording in
the PEP which would suggest they be disallowed.


*Third point (aliases)*

[Guido]
> As a corollary, this would also be correct, right?
> SplitDataSet[Height, Width]
> and it would make Ts be (Height, Width). Right?

Right, exactly.

> TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
> TwoArrays[Height]
> Given the previous example and my corollary, this seems ambiguous, just
like `def f(*tuple[*Ts1, *Ts2])`.

Oh, right, good point. So it sounds like the algorithm I described for
aliases isn't *quite* accurate - it's not that we assign types to type
variables by going from left to right and assigning greedily, it's that we
assign types to type variables using the same mechanisms as in other
contexts.

So I guess the result is that multiple TypeVarTuples in aliases should be
disallowed, for the same reasons as disallowing multiple TypeVarTuples in
parameter lists elsewhere? (I'll wait for confirmation on this from you and
Pradeep, then draft a PR clarifying this.)

On Thu, 13 Jan 2022 at 16:44, Kevin Millikin  wrote:

> Yes, exactly.
>
> Specifically, the "wrong" example in section 'Multiple Type Variable
> Tuples: Not Allowed' suggests that maybe what is wrong is that `Generic`
> was given more than one unpacked type variable tuple.  The actual problem
> is a consequence of that: `class Array` has more than one type variable
> tuple as type parameters.  (But there are other ways that could happen and
> all of them should be wrong.)
>
> I think it might be good to say that explicitly in that section.
>
> On Thu, Jan 13, 2022 at 4:18 PM Matthew Rahtz  wrote:
>
>> Thanks also Kevin for this feedback!
>>
>> Good point about being careful to distinguish type parameters vs type
>> arguments. If I understand correctly, you're making two points:
>>
>> 1. The wording of the 'Multiple Type Variable Tuples: Not Allowed'
>> section - you're saying that we're being a bit imprecise here in saying
>> that we disallow multiple TypeVarTuples in a type parameter list, given
>> that in e.g. `def f(x: *Ts1, y: *Ts2)`, both Ts1 and Ts2 are members of the
>> parameter list for the function f, but there it's unambiguous, so in fact
>> it *is* allowed.
>>
>> 2. Use of wrong terminology elsewhere in the PEP. Agreed.
>>
>> I'll draft a PR tweaking the wording to fix both these points.
>>
>>
>> On Thu, 13 Jan 2022 at 11:28, Kevin Millikin 
>> wrote:
>>
>>> The wording there probably should be improved.  I had a different
>>> interpretation when I read that, so that suggests it needs to be clarified.
>>>
>>> We should ensure to draw a clear distinction between type parameters and
>>> type arguments.  (Generic classes and functions are parameterized over type
>>> parameters and they have a type parameter list that is implicit in the
>>> syntax.  Generic classes can be explicitly instantiated by giving them type
>>> arguments, and an instantiation has a (explicit or implicit) type argument
>>> list.)
>>>
>>> So when I read:
>>>
>>> """
>>> As of this PEP, only a single type variable 

[Python-Dev] Re: Fwd: PEP 646 (Variadic Generics): final call for comments

[Petr Viktorin]

On 13. 01. 22 16:23, Matthew Rahtz wrote:

Thanks for this feedback, Petr!

*First point (indexing assignment)*

Great catch; we hadn't thought about this. I agree it would be better to 
keep these in sync.


I just tested this in our current CPython implementation, and can 
confirm it looks like this already works fine. So as much as I agree 
with Guido in preferring not to make too many more updates to the PEP, I 
guess we can indeed just fix this with a small clarification. I'll also 
add some tests for this to our CPython implementation.


Thanks.


*Second point (multiple TypeVarTuples)*

In terms of the wording in the PEP - Guido, our intention actually 
/was/ to prohibit even the straightforward cases for now. Iirc, our 
reasoning was that we thought the decision boundary between 
"straightforward to infer type assignment" and "nontrivial to infer type 
assignment" (and of course "no unique type assignment") was tricky 
enough that we shouldn't complicate the already-long PEP by trying to 
describe all the cases where it was and wasn't ok.


Petr, do I understand that the crux for you is basically that we should 
commit to multiple-stars at the syntax level - the main implication 
being to make sure we've properly documented and tested it? I'm happy 
with this; we plan to follow up with another PEP that /does/ talk about 
when multiple unpackings are ok anyway. I guess we should just a) 
clarify in the PEP that allowing multiple unpackings in the grammar 
isn't accidental, and b) test this in our CPython implementation?


Even less, actually.
The PEP doesn't make a very clear distinction between invalid Python 
syntax vs. invalid type annotation, so I wanted to check if we're on the 
same page here: the newly valid syntax will be subject to PEP 387.
We clearly are on the same page, and I don't think you need to update 
the PEP.




*Third point (aliases)*
*
*
This is actually a great point - I had to think about this myself.

Since PEP 484 doesn't explicitly spell out how assignment of type to 
type variables in generic aliases works either, I had to try some things 
with mypy playground to figure out what the current rules are. I think 
the logic is, if we have an alias like


Foo = tuple[T1, T1, T2]
Foo[int, str]

then we construct a list of unique type variables in the alias (here 
[T1, T2]), then assign types to those variables by going left to right 
through the type argument list when the alias is instantiated. Guido, 
can you remember from your time with mypy whether this is correct? 
Pradeep, I guess you'll also know about this?


Based on that precedent, I believe that:

SplitDataset = Tuple[Array[*Ts], Array[*Ts]]
SplitDataset[Height]  # Valid; equivalent Tuple[Array[Height], 
Array[Height]]


(indeed, I just tested this with Pyre, and that matches our 
implementation there)


For this one:

TwoArrays = Tuple[Array[*Ts1], Array[*Ts2]]
TwoArrays[Height]

Since we we allow a TypeVarTuple to be bound to an /empty/ list of 
types, when we try to assign type arguments [Height] to the unique list 
of type variables [Ts1, Ts2], I think we should end up with Ts1 
containing Height and Ts2 being empty; thus:


TwoArrays[Height]  # Valid; equivalent to Tuple[Array[Height], Array[()]]

But I actually can't get this to type check correctly in Pyre. Pradeep, 
this is the test I was using: 
https://gist.github.com/mrahtz/cc86c29538de1d4a80a2e8958ae71c5a 
 Am I 
doing something wrong?


Also, by that logic, in this situation /all/ the type arguments would be 
assigned to Ts1, and Ts2 would always end up being empty. That seems a 
bit weird but.../shrug/ it also seems correct.


In any case, this is definitely something we should explain better in 
the PEP. I'll make a TODO for myself to write something on this once 
Pradeep and Guido have confirmed whether my understanding is correct.


When I asked my curious question, I thought I misread a piece of text, 
not that it's a detail that went unnoticed, and could delay the PEP.



I can't speak for the whole SC, but on the Monday meeting I'll suggest 
accepting the PEP with a note that

- index assignment is also affected, and
- the details around multiple unpackings in a type expression aren't 
specified precisely. This gives individual type checkers some leeway, 
but can be tightened in future PEPs.



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UWZ33C67QT2FV5RDUUHLRTMVF7HA2VEZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Mark Dickinson]

On Sun, Jan 2, 2022 at 10:35 AM Mark Dickinson  wrote:

> Division may still be problematic.
>

On that note: Python divisions are somewhat crippled even on x64. Assuming
30-bit digits, the basic building block that's needed for multi-precision
division is a 64-bit-by-32-bit unsigned integer division, emitting a 32-bit
quotient (and ideally also a 32-bit remainder). And there's an x86/x64
instruction that does exactly that, namely DIVL. But without using inline
assembly, current versions of GCC and Clang apparently can't be persuaded
to emit that instruction from the longobject.c source - they'll use DIVQ (a
128-bit-by-64-bit division, albeit with the top 64 bits of the dividend set
to zero) on x64, and the __udivti3 or __udivti4 intrinsic on x86.

I was curious to find out what the potential impact of the failure to use
DIVL was, so I ran some timings. A worst-case target is division of a large
(multi-digit) integer by a single-digit integer (where "digit" means digit
in the sense of PyLong digit, not decimal digit), since that involves
multiple CPU division instructions in a fairly tight loop.

Results: on my laptop (2.7 GHz Intel Core i7-8559U, macOS 10.14.6,
non-optimised non-debug Python build), a single division of 10**1000 by 10
takes ~1018ns on the current main branch and ~722ns when forced to use the
DIVL instruction (by inserting inline assembly into the inplace_divrem1
function). IOW, forcing use of DIVL instead of DIVQ, in combination
with getting the remainder directly from the DIV instruction instead of
computing it separately, gives a 41% speedup in this particular worst case.
I'd expect the effect to be even more marked on x86, but haven't yet done
those timings.

For anyone who wants to play along, here's the implementation of the
inplace_divrem1 (in longobject.c) that I was using:

static digit
inplace_divrem1(digit *pout, digit *pin, Py_ssize_t size, digit n)
{
digit remainder = 0;

assert(n > 0 && n <= PyLong_MASK);
while (--size >= 0) {
twodigits dividend = ((twodigits)remainder << PyLong_SHIFT) | pin[size];
digit quotient, high, low;
high = (digit)(dividend >> 32);
low = (digit)dividend;
__asm__("divl %2\n"
: "=a" (quotient), "=d" (remainder)
: "r" (n), "a" (low), "d" (high)
);
pout[size] = quotient;
}
return remainder;
}


I don't know whether we *really* want to open the door to using inline
assembly for performance reasons in longobject.c, but it's interesting to
see the effect.

-- 
Mark
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZWGPO3TMCI7WNLC3EMS26DIKI5D3ZWMK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2022-01-07 - 2022-01-14)
Python tracker at https://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open7202 ( +3)
  closed 50906 (+81)
  total  58108 (+84)

Open issues with patches: 2886 


Issues opened (48)
==

#32876: HTMLParser raises exception on some inputs
https://bugs.python.org/issue32876  reopened by iritkatriel

#46298: Automatically check for __slots__-mistakes in Lib
https://bugs.python.org/issue46298  opened by ariebovenberg

#46304: Unable to iterate over lines in a file without a block of code
https://bugs.python.org/issue46304  opened by jaraco

#46309: Task created by StreamReaderProtocol gets garbage collected.
https://bugs.python.org/issue46309  opened by simwr872

#46311: Clean up PyLong_FromLong and PyLong_FromLongLong
https://bugs.python.org/issue46311  opened by mark.dickinson

#46312: range() function could accept slice() objects as parameters
https://bugs.python.org/issue46312  opened by yota moteuchi

#46313: SSLObject does not raise SSLEOFError on OpenSSL 3
https://bugs.python.org/issue46313  opened by alex.gronholm

#46315: Add support for WebAssembly System Interface (wasm32-wasi)
https://bugs.python.org/issue46315  opened by christian.heimes

#46316: Optimize pathlib.Path.iterdir()
https://bugs.python.org/issue46316  opened by barneygale

#46317: Pathlib.rename isn't robust
https://bugs.python.org/issue46317  opened by Oz.Tiram

#46318: asyncio and ssl: ResourceWarning: unclosed transport
https://bugs.python.org/issue46318  opened by mdk

#46323: Use _PyObject_Vectorcall in Modules/_ctypes/callbacks.c
https://bugs.python.org/issue46323  opened by hydroflask

#46325: Documentation for SubprocessTransport.get_pipe_transport retur
https://bugs.python.org/issue46325  opened by xx11mz

#46326: 'venv --clear' should prompt user before nuking entire directo
https://bugs.python.org/issue46326  opened by alimpfard

#46329: Split up the CALL_NO_KW and CALL_KW instructions.
https://bugs.python.org/issue46329  opened by Mark.Shannon

#46330: Simplify the signature of __exit__
https://bugs.python.org/issue46330  opened by Jelle Zijlstra

#46333: ForwardRef.__eq__ does not respect module parameter
https://bugs.python.org/issue46333  opened by andreash

#46334: Glossary URLs with anchor link no longer jump to definitions
https://bugs.python.org/issue46334  opened by trey

#46335: asyncio.create_subprocess_exec throws RuntimeError yet still e
https://bugs.python.org/issue46335  opened by Clint Olsen

#46336: Sixth element of tuple from __reduce__(), inconsistency betwee
https://bugs.python.org/issue46336  opened by lev.bishop

#46337: urllib.parse: Allow more flexibility in schemes and URL resolu
https://bugs.python.org/issue46337  opened by lincolnauster

#46338: libc_ver() runtime error when sys.executable is empty
https://bugs.python.org/issue46338  opened by allie.hammond

#46339: PEG parser segfault from ast.literal_eval
https://bugs.python.org/issue46339  opened by gregory.p.smith

#46340: DeprecationWarning emitted when running asyncio tests
https://bugs.python.org/issue46340  opened by kumaraditya303

#46341: duplicate paragraphs - asyncio Coroutines and Tasks file
https://bugs.python.org/issue46341  opened by davem

#46343: Add PyErr_GetActiveException and PyErr_SetActiveException
https://bugs.python.org/issue46343  opened by iritkatriel

#46349: inspect.getdoc() should append parent class method docs when e
https://bugs.python.org/issue46349  opened by gregory.p.smith

#46350: re.sub, re.Match.expand, etc doesn't allow x, u, U, or N escap
https://bugs.python.org/issue46350  opened by bup

#46351: Makefile missing '/' for some path names
https://bugs.python.org/issue46351  opened by gwolski

#46353: 'pydoc -k' fails when some module's loader is not found
https://bugs.python.org/issue46353  opened by dlax

#46356: [C API] Enforce usage of PyFrame_GetBack()
https://bugs.python.org/issue46356  opened by vstinner

#46360: Inconsistent import behavior for (unusual) submodules
https://bugs.python.org/issue46360  opened by eric.snow

#46361: Small ints aren't always cached properly
https://bugs.python.org/issue46361  opened by brandtbucher

#46363: Two typos in versions 3.7 document translation of zh_CN
https://bugs.python.org/issue46363  opened by perlang

#46364: asyncio subprocess cannot read from /dev/stdin
https://bugs.python.org/issue46364  opened by xoph

#46367: multiprocessing's "spawn" doesn't actually use spawn
https://bugs.python.org/issue46367  opened by jakirkham

#46368: faulthandler: add the ability to dump all interpreters, not on
https://bugs.python.org/issue46368  opened by vstinner

#46369: get_type_hints does not evaluate ForwardRefs inside NewType
https://bugs.python.org/issue46369  opened by andreash

#46371: A better way to resolve ForwardRefs in type aliases across mod
https://bugs.python.org/issue46371  opened by andreas

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Tim Peters]

]Mark Dickinson ]
>> Division may still be problematic.

Heh. I'm half convinced that heavy duty bigint packages are so often
written in assembler because their authors are driven insane by trying
to trick C compilers into generating "the obvious" machine
instructions needed.

An alternative to HW division is to multiply by a scaled integer
reciprocal instead, but it's very delicate to get all cases right. GMP
heavyweights Niels Moller and Torbjorn Granlund wrote up the most
efficient I've seen of this ilk in their paper "Improved division by
invariant integers".

It requires a single x single -> double multiply, and another but only
the low bits from that one, to get the right quotient and remainder.

Ironically, it would run faster if CPython used all 32 bits of its
internal digits - some operations in their algorithm are expected to
overflow at times (including + and -!), andi it's crucial that they be
done modulo the base in use. That happens "by magic" if the base
matches the unsigned type's width.

They only need a single-width scaled reciprocal, which can be computed
with a HW double / single -> single divide if available, or via
excruciating division-free scaled integer Newton refinement. Of course
the cost of computing the reciprocal once pays off each time the same
denominator is used.

Curiously,

"""
It is curious that on these processors, the combination of our
reciprocal algorithm (Alg. 2) and division algorithm (Alg. 4) is
significantly faster than the built in assembler instruction
for 2/1 division.
"""

HW division may have gotten faster since then, though.

> On that note: Python divisions are somewhat crippled even on x64. Assuming
> 30-bit digits, the basic building block that's needed for multi-precision 
> division
> is a 64-bit-by-32-bit unsigned integer division, emitting a 32-bit quotient 
> (and
> ideally also a 32-bit remainder).

Which is an instance of what they mean by "2/1 division" above.

> ...
> IOW, forcing use of DIVL instead of DIVQ, in combination with getting the 
> remainder
> directly from the DIV instruction instead of computing it separately, gives a 
> 41%
> speedup in this particular worst case. I'd expect the effect to be even more 
> marked
> on x86, but haven't yet done those timings.

> ...
> I don't know whether we really want to open the door to using inline assembly
> for performance reasons in longobject.c, but it's interesting to see the 
> effect.

Indeed it is! But changing the problem to use multiplication instead
may be both faster and more portable, albeit at the cost of subtler
code.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QGCHH6B7HCQRARPPVELVROO6FMWQY3G4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] [RELEASE] Python 3.10.2, 3.9.10, and 3.11.0a4 are now available

[Łukasz Langa]

Hi there,
before we begin the usual round of release notes, please do note that the three 
new versions of Python released today do not contain Windows installers yet. 
This is temporary, due to a more complex than expected code signing certificate 
renewal.
We’ve held the releases all week while the situation is getting resolved but 
the urgency of 3.10.2 in particular made us release without the Windows 
installers after all. We apologize for the inconvenience and are doing 
everything we can to put the Windows installer in place as soon as possible.

We’re rooting for both Ee Durbin and Steve Dower who are helping us resolve 
this. Thanks for your hard work! Hopefully, by this time next week, this will 
only be a footnote in release management history.

The releases you’re looking at were all cursed in some way. What a way to start 
2022! Besides the certificate hold up, Python 3.10.2 is an expedited release 
(you’ll want to upgrade, read below!), Python 3.11.0a4 had almost 20 (sic, 
twenty!) release blockers before being finally green, and Python 3.9.10 was 
made from a new M1 Mac on macOS Monterey which made the usually boring process 
quite a ride. We’re hoping 2022 won’t be this intense all year!


 
Python
 3.10.2

Get it here: https://www.python.org/downloads/release/python-3102/ 
  

This is a special bugfix release ahead of schedule to address a memory leak 
that was happening on certain function calls when using Cython 
. The memory leak consisted of a small 
constant amount of bytes in certain function calls from Cython code. Although 
in most cases this was not very noticeable, it was very impactful for 
long-running applications and certain usage patterns. Check bpo-46347 
 for more information.

Upgrading existing Python 3.10 installations is highly recommended. Even though 
this is an expedited release, it still contains over 100 other bug fixes. See 
the change log  
for details.

The next Python 3.10 maintenance release will be 3.10.3, currently scheduled 
for 2022-04-04.


 
Python
 3.9.10

Get it here: https://www.python.org/downloads/release/python-3910/  

Python 3.9.10 is the ninth maintenance release of the legacy 3.9 series. Note: 
Python 3.10 is now the latest feature release series of Python 3.

Python 3.9 micro-releases enter double digits! There’s been 130 commits since 
3.9.9 which is a higher number of fixes for this stage of the life cycle 
compared to 3.8. See the changelog 
 for details on 
what changed.

As a reminder, on macOS, the default installer is now the new universal2 
variant. It’s compatible with Mac OS X 10.9 and newer, including macOS 11 Big 
Sur and macOS 12 Monterey. Python installed with this variant will work 
natively on Apple Silicon processors.

The next Python 3.9 maintenance release will be 3.9.11, currently scheduled for 
Pi Day '22 (2022-03-14).


 
Python
 3.11.0a4

Get it here: https://www.python.org/downloads/release/python-3110a4/  

Python 3.11 is still in development. This release, 3.11.0a4, is the fourth of 
seven planned alpha releases.

Alpha releases are intended to make it easier to test the current state of new 
features and bug fixes by the community, as well as to test the release process.

During the alpha phase, features may be added up until the start of the beta 
phase (2022-05-06) and, if necessary, may be modified or deleted up until the 
release candidate phase (2022-08-01). Please keep in mind that this is a 
preview release and its use is not recommended for production environments.

Many new features for Python 3.11 are still being planned and written. Among 
the new major new features and changes so far:

PEP 657  – Include Fine-Grained 
Error Locations in Tracebacks
PEP 654  – Exception Groups and 
except*
The Faster CPython Project  is already 
yielding some exciting results: this version of CPython 3.11 is ~ 19% faster on 
the geometric mean of the PyPerformance benchmarks <>, compared to 3.10.0.
(Hey, fellow core developer, if a feature you find important is missing from 
this list, let Pablo know .)
The next pre-release of Python 3.11 will be 3.11.0a5, currently scheduled for 
Wednesday, 2022-02-02.


 

[Python-Dev] Re: Is anyone using 15-bit PyLong digits (PYLONG_BITS_IN_DIGIT=15)?

[Gregory P. Smith]

On Fri, Jan 14, 2022 at 9:50 AM Mark Dickinson  wrote:

> On Sun, Jan 2, 2022 at 10:35 AM Mark Dickinson  wrote:
>
>> Division may still be problematic.
>>
>
> On that note: Python divisions are somewhat crippled even on x64. Assuming
> 30-bit digits, the basic building block that's needed for multi-precision
> division is a 64-bit-by-32-bit unsigned integer division, emitting a 32-bit
> quotient (and ideally also a 32-bit remainder). And there's an x86/x64
> instruction that does exactly that, namely DIVL. But without using inline
> assembly, current versions of GCC and Clang apparently can't be persuaded
> to emit that instruction from the longobject.c source - they'll use DIVQ (a
> 128-bit-by-64-bit division, albeit with the top 64 bits of the dividend set
> to zero) on x64, and the __udivti3 or __udivti4 intrinsic on x86.
>
> I was curious to find out what the potential impact of the failure to use
> DIVL was, so I ran some timings. A worst-case target is division of a large
> (multi-digit) integer by a single-digit integer (where "digit" means digit
> in the sense of PyLong digit, not decimal digit), since that involves
> multiple CPU division instructions in a fairly tight loop.
>
> Results: on my laptop (2.7 GHz Intel Core i7-8559U, macOS 10.14.6,
> non-optimised non-debug Python build), a single division of 10**1000 by 10
> takes ~1018ns on the current main branch and ~722ns when forced to use the
> DIVL instruction (by inserting inline assembly into the inplace_divrem1
> function). IOW, forcing use of DIVL instead of DIVQ, in combination
> with getting the remainder directly from the DIV instruction instead of
> computing it separately, gives a 41% speedup in this particular worst case.
> I'd expect the effect to be even more marked on x86, but haven't yet done
> those timings.
>
> For anyone who wants to play along, here's the implementation of the
> inplace_divrem1 (in longobject.c) that I was using:
>
> static digit
> inplace_divrem1(digit *pout, digit *pin, Py_ssize_t size, digit n)
> {
> digit remainder = 0;
>
> assert(n > 0 && n <= PyLong_MASK);
> while (--size >= 0) {
> twodigits dividend = ((twodigits)remainder << PyLong_SHIFT) | pin[size];
> digit quotient, high, low;
> high = (digit)(dividend >> 32);
> low = (digit)dividend;
> __asm__("divl %2\n"
> : "=a" (quotient), "=d" (remainder)
> : "r" (n), "a" (low), "d" (high)
> );
> pout[size] = quotient;
> }
> return remainder;
> }
>
>
> I don't know whether we *really* want to open the door to using inline
> assembly for performance reasons in longobject.c, but it's interesting to
> see the effect.
>

That only appears true in default boring -O2 builds.  Use `./configure
--enable-optimizations` and the C version is *much* faster than your asm
one...

250ns for C vs 370ns for your asm divl one using old gcc 9.3 on my zen3
when compiled using --enable-optimizations.

tested using ` -m timeit -n 150 -s 'x = 10**1000; r=x//10; assert r ==
10**999, r' 'x//10' `

Use of __asm__ appears to have prevented the compiler from being able to
fully optimize that code in PGO/FDO mode.

I trust the compiler toolchain to know what's close to best here.
Optimizing use of a div instruction isn't entirely straight forward as on
many microarchitectures the time required varies based on the inputs as
they'll internally implement looping when values exceed the bits with which
their hw operates.  there's probably interesting ways to optimize bignum
division using opencl and vector hardware as well - for the case when you
know you've got over a dozen digits; but that's what numpy et. al. are for.
Bignums in python are a convenience. Most values normal code deals with are
less than 2**100.

-Greg


>
> --
> Mark
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZWGPO3TMCI7WNLC3EMS26DIKI5D3ZWMK/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WAZGSHPS7LBN4QNYVVSUG2RC26322L5D/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Changing PySequence and PyMapping checks

[Bar Harel]

There's a long time issue of trying to differentiate mappings and sequences
in the C-API in a fast and reliable way.

Due to recent changes, we might be able to do so at last, by checking
tp_flags + str/bytes/bytearray which are considered unique.

This might be a breaking change in the Stable ABI promise but one that can
be considered a bug fix as the current behavior is lacking.

What do you think?

bpo: https://bugs.python.org/issue46376

Best regards,
Bar Harel
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WTIKR5DGMZRKRRAZGFTD4EPG5ITFHOVS/
Code of Conduct: http://python.org/psf/codeofconduct/