[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Roundup Robot]

Roundup Robot added the comment:

New changeset 94d630a02a81 by Serhiy Storchaka in branch 'default':
Issue #29368: The extend() method is now called instead of the append()
https://hg.python.org/cpython/rev/94d630a02a81

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29418] inspect.isroutine does not return True for some bound builtin methods

[Manuel Krebber]

New submission from Manuel Krebber:

Some of the builtin methods are not recognized as such by inspect.isroutine(). 
inspect.ismethoddescriptor() only returns True for the unbound versions of the 
methods but not the bound ones.

For example:

>>> inspect.isroutine(object.__str__)
True
>>> inspect.isroutine(object().__str__)
False

The second one should also return True as it does for user-defined classes:

>>> class A:
... def f(self):
... pass
>>> inspect.isroutine(A.f)
True
>>> inspect.isroutine(A().f)
True

The types needed for that have already been added to the types module in issue 
#29377. Here is the commit: 
https://github.com/python/cpython/commit/1947d33a0a1c2ba006397579ec6235528faea9fd

--
components: Library (Lib)
messages: 286752
nosy: Wheerd, levkivskyi, yselivanov
priority: normal
severity: normal
status: open
title: inspect.isroutine does not return True for some bound builtin methods
type: behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Roundup Robot]

Roundup Robot added the comment:

New changeset 328147c0edc3 by Victor Stinner in branch 'default':
Issue #29368: Fix _Pickle_FastCall() usage in do_append()
https://hg.python.org/cpython/rev/328147c0edc3

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Roundup Robot]

Changes by Roundup Robot :


--
stage: patch review -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Roundup Robot]

Changes by Roundup Robot :


--
resolution:  -> fixed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Roundup Robot]

Changes by Roundup Robot :


--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Roundup Robot]

Roundup Robot added the comment:


New changeset f89fdc29937139b55dd68587759cadb8468d0190 by Serhiy Storchaka in 
branch 'master':
Issue #29368: The extend() method is now called instead of the append()
https://github.com/python/cpython/commit/f89fdc29937139b55dd68587759cadb8468d0190

New changeset 4d7e63d9773a766358294593fc00b1f8c8f41b5d by Victor Stinner in 
branch 'master':
Issue #29368: Fix _Pickle_FastCall() usage in do_append()
https://github.com/python/cpython/commit/4d7e63d9773a766358294593fc00b1f8c8f41b5d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[STINNER Victor]

STINNER Victor added the comment:

> Even if I don't see any refleak, you might just run "./python -m test -R 3:3 
> test_pickle" just to be sure :-)

Change 94d630a02a81 introduced a crash in test_pickle:

Fatal Python error: ..\Modules\_pickle.c:5847 object at 02B7F7BED2F8 has 
negative ref count -1

Seen on buildbots, but can always be reproduce on Linux as well.

It seems like you was biten by the surprising _Pickle_FastCall() API which 
decreases the reference counter of its second parameter. Don't ask me why it 
does that :-) (I don't know.)

I fixed the bug in the change 328147c0edc3 to repair buildbots.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

This doesn't make the patch much bigger. Usually such kind of changes are made 
in one patch.

Here is a patch that also changes the type of the self parameter to 
PyStructObject*.

struct_fastcall-6.patch LGTM, but I prefer struct_fastcall-7.patch.

--
Added file: http://bugs.python.org/file46488/struct_fastcall-7.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29263] Implement LOAD_METHOD/CALL_METHOD for C functions

[STINNER Victor]

STINNER Victor added the comment:

I tried to benchmark b''.decode(encoding='ascii'): CALL_METHOD is not used for 
this call, but CALL_FUNCTION_KW :-/ So the call is not affected by your patch.

I also ran a quick benchmark on loadmethod-methoddescr.patch:

b''.decode(): 71.1 ns +- 0.5 ns -> 65.4 ns +- 0.2 ns: 1.09x faster (-8%)
b''.decode('utf8'): 92.8 ns +- 0.4 ns -> 85.5 ns +- 0.3 ns: 1.09x faster (-8%)

I confirm the speedup.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Thanks Victor.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29263] Implement LOAD_METHOD/CALL_METHOD for C functions

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27659] Prohibit implicit C function declarations

[Chi Hsuan Yen]

Chi Hsuan Yen added the comment:

Hello, any updates here? I hope this merged soon so that potential issues on 
obscure platforms can be fixed as soon as possible.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29079] pathlib.resolve() causes infinite loop on Windows

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I'm not sure the fix is correct. os.path.dirname(s) can point to different 
place than os.path.abspath(os.path.join(s, os.pardir)) if the last component of 
s is "..", "." or a symbolic link.

Would be nice to add tests.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

-if (PyObject_GetBuffer(input, &vbuf, PyBUF_SIMPLE) < 0)
-return NULL;

Oh by the way, I forgot to mention a subtle change. 
PyObject_GetBuffer(PyBUF_SIMPLE) is less strict that PyArg_Parse("y#") / 
"buffer" converter of Argument Clinic: getargs.c also checks that the buffer is 
contiguous, extract of getbuffer():

if (!PyBuffer_IsContiguous(view, 'C')) {
PyBuffer_Release(view);
*errmsg = "contiguous buffer";
return -1;
}

I don't know well the buffer protocol. I don't know any object which provide a 
non-contiguous buffer. At least, I can say that the last time I looked at this 
dark part of Python, the documentation was between tiny and non-existent :-/ 
The buffer protocol is complex but not well documented :-(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29263] Implement LOAD_METHOD/CALL_METHOD for C functions

[INADA Naoki]

INADA Naoki added the comment:

I confirmed bm_mako performance degrade is caused by L1 cache miss.
(I didn't use --with-optimization)
https://gist.github.com/methane/b33edbf1f123ae026e704b0e005c3606

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29414] Change 'the for statement is such an iterator' in Tutorial

[Marco Buttu]

Marco Buttu added the comment:

I agree with you that the current sentence:

"We have seen that the for statement is such an iterator"

is wrong. But also the new sentence IMHO is confusing, because it stills 
compare statementes with objects:

"the for statement expects an object that is iterable. The function list is 
another; it creates lists from iterables". 

Also list is a class, not a function.
IMHO the goal of the sentence you want to patch is to complete the previous one 
[1], adding an example of "construct" that operates with iterables, and of 
function that takes an iterable. If you want to follow that purpose, I suggest 
somethink like this:

"We have seen that the for statement is such a construct, while examples of 
functions that take an iterable are ``sum()`` and ``max()``::"

Written in better English than mine...


[1] "We say such an object is iterable, that is, suitable as a target for 
functions and constructs that expect something from which they can obtain 
successive items until the supply is exhausted."

--
nosy: +marco.buttu

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29263] Implement LOAD_METHOD/CALL_METHOD for C functions

[STINNER Victor]

STINNER Victor added the comment:

Naoki> I confirmed bm_mako performance degrade is caused by L1 cache miss.

I know this performance instability very well, the issue is called "code 
placement":
https://haypo.github.io/journey-to-stable-benchmark-deadcode.html

I tried to fight it with GCC __attribute__((hot)) in the issue #28618, but it 
doesn't fix the issue (at least, not completely):
http://bugs.python.org/issue28618#msg281459

In my experience, the best fix is PGO. Slowly, I consider that it's worthless 
to try to "fight" against code placement, and that benchmark results are only 
reliable when PGO compilation was used. Otherwise, you should ignore small 
performance differences. Problem: What is the threshold? 5%? 10%? I already 
noticed a difference up to 70% only caused by code placement!
https://haypo.github.io/analysis-python-performance-issue.html

--

I ran benchmarks on loadmethod-methoddescr.patch on haypo@speed-python with 
LTO+PGO. If you only show performance difference of at least 5%, only 3 
benchmarks are significant and are faster:
---
haypo@speed-python$ python3 -m perf compare_to ~/benchmarks/*762a93935afd*json 
loadmethod-methoddesc_ref_762a93935afd.json  -G --min-speed=5
Faster (3):
- regex_v8: 50.3 ms +- 0.4 ms -> 43.2 ms +- 0.3 ms: 1.17x faster (-14%)
- scimark_monte_carlo: 230 ms +- 6 ms -> 208 ms +- 4 ms: 1.11x faster (-10%)
- scimark_lu: 390 ms +- 17 ms -> 370 ms +- 13 ms: 1.05x faster (-5%)

Benchmark hidden because not significant (61): (...)
---

In my experience, regex_v8 and scimark_* benchmarks are not really reliable. 
I'm not surprised to not see a major speedup on performance, since the speedup 
on microbenchmarks is only 10% faster.

IHMO 10% faster on method calls is significant enough, since it's a core, very 
common, and widely used Python feature.

--

To be more explicitl:  loadmethod-methoddescr.patch LGTM except of minor 
comments on the review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29400] Instruction level tracing via sys.settrace

[Xavier de Gaye]

Xavier de Gaye added the comment:

> The motivation for this suggestion was that for tracers that *are* interested 
> in instructions, it *might* be simpler to get two calls, one of 'line', and 
> one of 'instruction'. Maybe it's a bad idea though.

Thanks for the clarification. It's ok if instruction tracing remains an opt-in 
feature. As for this suggestion, I am not sure it is useful for debuggers based 
on the bdb module as trace_dispatch() (the bdb trace function) receives 
notification of the trace event type as the 'event' argument.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:

New changeset f3ff4a3ce77c by Victor Stinner in branch 'default':
Issue #29300: Convert _struct module to Argument Clinic
https://hg.python.org/cpython/rev/f3ff4a3ce77c

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

I also prefer Serhiy's struct_fastcall-7.patch over my struct_fastcall-6.patch.

Serhiy Storchaka: "This doesn't make the patch much bigger. Usually such kind 
of changes are made in one patch."

Well, it's just that you prefer to reduce the number of commits and so fold 
tiny changes into a single big commit, whereas I now prefer multiple tiny 
commits. I have not strong preferences between the two ways to commit, so I 
pushed your patch.

Here is the full commit message, since Roundbot bot only shows the first line:
---
Issue #29300: Convert _struct module to Argument Clinic

* The struct module now requires contiguous buffers.
* Convert most functions and methods of the _struct module to Argument Clinic
* Use "Py_buffer" type for the "buffer" argument. Argument Clinic is
  responsible to create and release the Py_buffer object.
* Use "PyStructObject *" type for self to avoid explicit conversions.
* Add an unit test on the _struct.Struct.unpack_from() method to test passing
  arguments as keywords.
* Rephrase docstrings.
* Rename "fmt" argument to "format" in docstrings and the documentation.

As a side effect, functions and methods which used METH_VARARGS calling
convention like struct.pack() now use the METH_FASTCALL calling convention
which avoids the creation of temporary tuple to pass positional arguments and
so is faster. For example, struct.pack("i", 1) becomes 1.56x faster (-36%)::

$ ./python -m perf timeit \
-s 'import struct; pack=struct.pack' 'pack("i", 1)' \
--compare-to=../default-ref/python
Median +- std dev: 119 ns +- 1 ns -> 76.8 ns +- 0.4 ns: 1.56x faster (-36%)
Significant (t=295.91)

Patch co-written with Serhiy Storchaka.
---

So yeah, the "side effect" is that struct.pack("i", 1) becomes 1.56x faster 
(-36%). Ok, maybe it was my main goal ;-) I also mentioned the "new" (?) 
contiguous requirement on buffers.

Thanks Serhiy for your multiple reviews, very useful as usual. I like the final 
result: better documentation, better docstrings and better code!

--
resolution:  -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27715] call-matcher breaks if a method is mocked with spec=True

[jordan-pittier]

jordan-pittier added the comment:

I stumbled onto this today. I can confirm the issue.

--
nosy: +jordan-pittier

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:


New changeset e552e185f3087204d326409e8631eb33dd0e7958 by Victor Stinner in 
branch 'master':
Issue #29300: Convert _struct module to Argument Clinic
https://github.com/python/cpython/commit/e552e185f3087204d326409e8631eb33dd0e7958


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


--
stage:  -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Martin Panter]

Martin Panter added the comment:

Shouldn’t the top-level unpack() parameter be called “buffer” like the other 
functions and methods, not “inputstr”?

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Propose for your consideration a patch that uses Argument Clinic for getting 
possible cached struct object from the format. This simplifies the 
implementation of module level functions.

--
resolution: fixed -> 
stage: resolved -> patch review
status: closed -> open
Added file: http://bugs.python.org/file46489/cache_struct.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Martin Panter]

Martin Panter added the comment:

FYI Victor, you can make non-C-contiguous buffers by slicing memoryview:

>>> struct.unpack(">L", memoryview(b"1234")[::-1])
Traceback (most recent call last):
  File "", line 1, in 
BufferError: memoryview: underlying buffer is not C-contiguous

Can also use the built-in _testbuffer module to create stranger buffers.

--
resolution:  -> fixed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

Martin> Shouldn’t the top-level unpack() parameter be called “buffer” like the 
other functions and methods, not “inputstr”?

Hum. I reopen the issue.

Attached patch renames unpack() "inputstr" argument to "buffer" and uses the 
Py_buffer type for it. I had to fix unit tests which passed str instead of 
bytes to buffer. Before this error was missed because the unit test is written 
to test the format string value, not the type of arguments.

--
resolution: fixed -> 
Added file: http://bugs.python.org/file46490/unpack_buffer.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> Oh by the way, I forgot to mention a subtle change.
> PyObject_GetBuffer(PyBUF_SIMPLE) is less strict that PyArg_Parse("y#") /
> "buffer" converter of Argument Clinic: getargs.c also checks that the
> buffer is contiguous, extract of getbuffer():

We already made such changes in the past. The difference is subtle and I have 
doubts that choosing any of ways was deliberate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> So yeah, the "side effect" is that struct.pack("i", 1) becomes 1.56x faster
> (-36%). Ok, maybe it was my main goal ;-) I also mentioned the "new" (?)
> contiguous requirement on buffers.

struct.pack() always was faster than int.to_bytes(). I wanted to speed up 
int.to_bytes(), and after converting to Argument Clinic in issue20185 it have 
became faster than struct.pack(). But after converting the struct module to 
Argument Clinic struct.pack() is faster than int.to_bytes() again! Now I need 
to find other ways to make int.to_bytes() even faster to win this chase.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29404] "TypeError: 'int' does not have the buffer interface" on memoryview over bytearray

[Vincent Pelletier]

Vincent Pelletier added the comment:

My original point of view was that:
- python3 is right to only accept integers, consistently with "str != bytes"
- python2 is rather right to accept str, consistently with "str == bytes"
- maybe the bug is that python2 should not reject integers, making the upgrade 
path to python3 (or the backward compatibility with python2, same thing) easy.

The context is that I develop a (pure-python) module interfacing with a C 
library, and somewhere in there I want to expose a read/write memory area (the 
bytearray) which first few bytes must not be accessible from the application 
using my module (because the application does not care about these bytes, and 
slicing everywhere is not convenient). Also, I do not want to expose ctypes 
instances (I'm supposed to be the python/C interface so the application does 
not have to care). So exposing that memory chunk via a memoryview slice over 
the original bytearray seems (and please do correct me if I am wrong) the right 
way to implement this:

>>> b = bytearray(16) # the "real" buffer
>>> c = (ctypes.c_char * 16).from_buffer(b) # for C library
>>> v = memoryview(b)[8:] # for host program

But because of this memoryview behaviour difference my API will behave 
inconsistently between python2 and python3.

My (naïve) initial idea submitting this bug report was that python2 would be 
modified to tolerate integers passed to memoryview.__setitem__. But I later 
realised such change would not be sufficient: python2's memoryview.__getitem__ 
returns strings where python3's returns integers. I agree that changing such 
visible behaviour in python2 would be bad.

Am I stuck with developing my own proxy class then (likely wrapping memoryview 
with type-casting glue) ?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

unpack_buffer.patch LGTM. Please backport test changes and other changes 
discussed before to other branches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29419] Argument Clinic: inline PyArg_UnpackTuple and PyArg_ParseStack(AndKeyword)?

[STINNER Victor]

New submission from STINNER Victor:

Argument Clinic calls one the following functions depending on parameters:

* PyArg_UnpackTuple(), _PyArg_UnpackStack()
* PyArg_ParseTuple(), _PyArg_ParseStack()
* PyArg_ParseTupleAndKeywords(), _PyArg_ParseStackAndKeywords()
* etc.

Would it make sense to emit C code instead of calling complex and slow 
PyArg_ParseXXX() functions? It would emit the most efficient C code to parse 
arguments.

I don't recall where this idea comes from. Maybe Larry Hastings told me once 
that he wants to implement this idea :-) I'm sure that Larry has a big plan but 
lacks time to implement all of his cool ideas.

Using profiled guided optimization (PGO), the compiler should be able to easily 
detect that error cases are unlikely and mark these code paths as unlikely.

We should probably experiment an implementation to be able to measure the 
speedup, to be able to say if the idea is worth it or not, in term of 
performance, since the motivation here is clearly performance.



We can begin with format strings only made of "O" format. Most simple example 
with divmod(), replace:

if (!_PyArg_UnpackStack(args, nargs, "divmod", 2, 2, &x, &y)) { return 
NULL; }

with something like:

if (nargs != 2) { _PyArg_ErrNumArgs(nargs, 2, 2, "divmod"); return NULL; }
x = args[0];
y = args[1];


The next question is if we should go further with more complex formats. Example 
with the format() function, replace:

if (!_PyArg_ParseStack(args, nargs, "O|U:format", &value, &format_spec)) { 
... }

with:

if (nargs < 1 || nargs > 2) { _PyArg_ErrNumArgs(nargs, 1, 2, "format"); 
return NULL; }

value = args[0];

if (nargs == 2) {
format_spec = args[1];
if (!PyUnicode_Check(format_spec)) { .. raise an exception ...; return 
NULL; }
/* getargs.c calls PyUnicode_READY(), we should also do it here */
if (PyUnicode_READY(format_spec) == -1) { return NULL; }
}
else {
format_spec = NULL;
}

--
components: Argument Clinic
messages: 286778
nosy: haypo, inada.naoki, larry, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Argument Clinic: inline PyArg_UnpackTuple and 
PyArg_ParseStack(AndKeyword)?
type: performance
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29419] Argument Clinic: inline PyArg_UnpackTuple and PyArg_ParseStack(AndKeyword)?

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

See issue23867 for the first step (unfinished). After that I planned to inline 
parsing code for PyArg_ParseTuple() with multiple arguments. Then split 
PyArg_ParseTupleAndKeywords() on two parts: first unpack keywords to linear 
sparse array and then handle it as positional arguments.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29404] "TypeError: 'int' does not have the buffer interface" on memoryview over bytearray

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> - maybe the bug is that python2 should not reject integers, making the 
> upgrade path to python3 (or the backward compatibility with python2, same 
> thing) easy.

It is too late for this.

You can use the helper that converts the argument to appropriate depending on 
the version.

if PY2:
def helper(arg):
if isinstance(ar, (int, long)):
return chr(arg)
return arg
else:
def helper(arg):
return arg
...
v[0] = helper(42)

Or just have different branches for performance critical code:

if PY2:
v[0] = b'\x2a'
else:
v[0] = 42

I don't think Python interpreter should be changed here.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "But after converting the struct module to Argument Clinic 
struct.pack() is faster than int.to_bytes() again!"

Sorry about that ;-)


Serhiy Storchaka: "Now I need to find other ways to make int.to_bytes() even 
faster to win this chase."

I ran a microbenchmark:

$ ./python -m perf timeit -s 'to_bytes=int.to_bytes' 'to_bytes(1, 4, "little")'

Reference: ~154 ns

Replace int_to_bytes_impl() body with:
   PyBytes_FromStringAndSize("1", 1)
=> ~120 ns (-34 ns)

Replace int_to_bytes() body with:
   return int_to_bytes_impl(self, 4, NULL, 1);
=> ~76 ns (-44 ns)

_PyArg_ParseStackAndKeywords() with _PyArg_Parser{"nU|$p:to_bytes"} takes 44 ns 
on a total of 154 ns. 29% of the runtime is spent on parsing arguments.

If you want to optimize further int.to_bytes(), IMHO we should explore the 
issue #29419: "Argument Clinic: inline PyArg_UnpackTuple and 
PyArg_ParseStack(AndKeyword)?".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:

New changeset 32380d41e788 by Victor Stinner in branch '3.5':
Issue #29300: test_struct tests unpack_from() with keywords
https://hg.python.org/cpython/rev/32380d41e788

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:

New changeset faa1e4f4b156 by Victor Stinner in branch 'default':
Rename struct.unpack() 2nd parameter to "buffer"
https://hg.python.org/cpython/rev/faa1e4f4b156

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "We already made such changes in the past. The difference is 
subtle and I have 
doubts that choosing any of ways was deliberate."

Right. IMHO it's safe to make sure that the buffer is contiguous. I'm quite 
sure that the code doesn't support non-contiguous buffers.


Serhiy Storchaka: "Please backport test changes and other changes discussed 
before to other branches."

Done. Sorry, I forgot this part.


Serhiy Storchaka: "unpack_buffer.patch LGTM."

Thanks for the review, it's now merged. It was a regression in unpack() 
docstring, Python 3.5 docstring contains "unpack(fmt, buffer)".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

Martin Panter: """FYI Victor, you can make non-C-contiguous buffers by slicing 
memoryview:

>>> struct.unpack(">L", memoryview(b"1234")[::-1])
Traceback (most recent call last):
  File "", line 1, in 
BufferError: memoryview: underlying buffer is not C-contiguous"""

Oh, it means that the Argument Clinic change doesn't add new checks on the 
buffer? In Python 3.6, memory_getbuf() raises an exception in the following 
code:

if (!REQ_STRIDES(flags)) {
if (!MV_C_CONTIGUOUS(baseflags)) {
PyErr_SetString(PyExc_BufferError,
"memoryview: underlying buffer is not C-contiguous");
return -1;
}
view->strides = NULL;
}

I undersrtand that memory_getbuf() is smart enough to raise an exception 
becaues the buffer is not contiguous. But a weaker implementation of getbuffer 
may not implement such check, whereas getbuffer() double check that the buffer 
is C-contiguous. Am I right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

I like the overall cache_struct.patch change, but I have questions: see my 
review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29394] Cannot tunnel TLS connection through TLS connection

[Maximilian Blochberger]

Maximilian Blochberger added the comment:

Okay, I see, thanks for the hint. That worked perfectly – I found 
`asyncio.sslproto._SSLPipe` very useful for that purpose.

I personally consider the behaviour of `ssl.SSLContext.wrap_socket()` 
unexpected and would raise an exception if that method call is tried on an 
instance of `ssl.SSLSocket`. But as this would be a change that could lead to 
backwards compatibility issues (if developers depend on that behaviour) this is 
probably not a good idea.

I think that the documentation for `ssl.SSLContext.wrap_socket()` has this 
behaviour to avoid future confusion – including a hint to make use of 
`ssl.SSLContext.wrap_bio()` instead.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23867] Argument Clinic: inline parsing code for 1-argument functions

[STINNER Victor]

STINNER Victor added the comment:

I like the idea since I just proposed something similar in the issue #29419, 
but it seems like your change removes the function name from error messages 
which can become much more obscure.

Maybe we should wrap all exceptions into a new TypeError which contains at 
least the function name, or even the parameter name/position. I mean chained 
exception to keep the original exception which contains more information.

The best would be to have all information in a single error message, but it is 
likely to be much more complex to implement, especially if you want to support 
arbitrary converter function, not only simple formats like  i". So I think that 
two chained exceptions is a reasonable compromise. What do you think?

If we succeed to get the function name and the parameter name or position, the 
error messages will be MUCH MORE better than currently! And Argument Clinic 
allows us to implement this feature.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


--
stage: patch review -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


--
resolution:  -> fixed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:


New changeset a88eb4fa9e7fdf1a1050786223044c6bb7949784 by Victor Stinner in 
branch '3.5':
Issue #29300: test_struct tests unpack_from() with keywords
https://github.com/python/cpython/commit/a88eb4fa9e7fdf1a1050786223044c6bb7949784


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:


New changeset a88eb4fa9e7fdf1a1050786223044c6bb7949784 by Victor Stinner in 
branch 'master':
Issue #29300: test_struct tests unpack_from() with keywords
https://github.com/python/cpython/commit/a88eb4fa9e7fdf1a1050786223044c6bb7949784

New changeset dcd4b1af2c59b0aae33cbac00d9f6fb47782ac57 by Victor Stinner in 
branch 'master':
Rename struct.unpack() 2nd parameter to "buffer"
https://github.com/python/cpython/commit/dcd4b1af2c59b0aae33cbac00d9f6fb47782ac57


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Roundup Robot added the comment:


New changeset a88eb4fa9e7fdf1a1050786223044c6bb7949784 by Victor Stinner in 
branch '3.6':
Issue #29300: test_struct tests unpack_from() with keywords
https://github.com/python/cpython/commit/a88eb4fa9e7fdf1a1050786223044c6bb7949784


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Stefan Krah]

Stefan Krah added the comment:

PyBUF_SIMPLE implies C-contiguous for a conforming buffer provider.

--
nosy: +skrah

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29404] "TypeError: 'int' does not have the buffer interface" on memoryview over bytearray

[Stefan Krah]

Stefan Krah added the comment:

I agree it's too late to change 2.7, and 3.x cannot (and should not) be changed 
at this stage.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29420] Python 3.6 change in dict iteration when inserting keys

[Matthew Brett]

New submission from Matthew Brett:

The behavior of dict iteration has changed in Python 3.6, in that inserting 
keys during iteration has a different and unpredictable affect.  For this code:

d = {'foo': 1}
for key in d:
   print(key)
   d.pop(key)
   d[key] = 1

Python 3.5 prints a single 'foo' (one pass through the loop).  Python 3.6 
generates five lines of 'foo' (five passes through the loop).  Of course this 
code is pathological, but I found this behavior from a bug in code where the 
pathology was a lot less obvious - see https://github.com/nipy/nipy/issues/420

--
messages: 286794
nosy: matthew.brett
priority: normal
severity: normal
status: open
title: Python 3.6 change in dict iteration when inserting keys
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

Stefan Krah: "PyBUF_SIMPLE implies C-contiguous for a conforming buffer 
provider."

Oh ok, thanks for the confirmation. So my change didn't add new checks and my 
commit message is wrong :-) Only non-conforming objects are impacted, but in 
such case, it's more a bug in the object than a bug in struct. I guess that 
non-conforming non-contiguous objects are very rare in the wild ;-)

The new code is more strict, so is safe in all corner cases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Proposed patch makes int.to_bytes() and int.from_bytes() slightly faster.

$ ./python -m perf timeit '1 .to_bytes(4, "little")' --compare-to 
"`pwd`/python0"
Median +- std dev: [python0] 1.28 us +- 0.07 us -> [python] 1.17 us +- 0.07 us: 
1.09x faster (-8%)

$ ./python -m perf timeit '1 .to_bytes(4, "big")' --compare-to "`pwd`/python0"
Median +- std dev: [python0] 1.30 us +- 0.07 us -> [python] 1.19 us +- 0.05 us: 
1.10x faster (-9%)

$ ./python -m perf timeit 'int.from_bytes(b"\1\0\0\0", "little")' --compare-to 
"`pwd`/python0"
Median +- std dev: [python0] 1.70 us +- 0.13 us -> [python] 1.41 us +- 0.06 us: 
1.20x faster (-17%)

$ ./python -m perf timeit 'int.from_bytes(b"\0\0\0\1", "big")' --compare-to 
"`pwd`/python0"
Median +- std dev: [python0] 1.71 us +- 0.15 us -> [python] 1.40 us +- 0.05 us: 
1.22x faster (-18%)

--
components: Interpreter Core
files: int-to_bytes-from_bytes.patch
keywords: patch
messages: 286796
nosy: haypo, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Make int.to_bytes() and int.from_bytes() slightly faster
type: performance
versions: Python 3.7
Added file: http://bugs.python.org/file46491/int-to_bytes-from_bytes.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29420] Python 3.6 change in dict iteration when inserting keys

[R. David Murray]

R. David Murray added the comment:

This "bug" has always existed, its just that its manifestation has changed.  
See issue 19332 for background.

--
nosy: +r.david.murray
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Guard against changing dict during iteration

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[STINNER Victor]

STINNER Victor added the comment:

Ah yes, I began with the exact same change when I experimented optimizing 
to_bytes():
http://bugs.python.org/issue29300#msg286781

I was going to suggest you the same, but the speedup was low compared to the 
cost of parsing arguments.

Anyway, int-to_bytes-from_bytes.patch is safe, LGTM, and the speedup is 
significant.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Dear Roundup Robot, please don't close issues.

--
resolution: fixed -> 
stage: resolved -> patch review
status: closed -> open
Added file: http://bugs.python.org/file46492/cache_struct-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29300] Modify the _struct module to use FASTCALL and Argument Clinic

[STINNER Victor]

STINNER Victor added the comment:

Serhiy Storchaka: "Dear Roundup Robot, please don't close issues."

For what it's worth, I reported issues of this robot to python-dev:

https://mail.python.org/pipermail/python-dev/2017-February/147317.html

https://mail.python.org/pipermail/python-dev/2017-February/147325.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Roundup Robot]

Roundup Robot added the comment:

New changeset 52c5a7c6 by Serhiy Storchaka in branch 'default':
Issue #29421: Make int.to_bytes() and int.from_bytes() slightly faster
https://hg.python.org/cpython/rev/52c5a7c6

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

I tried this optimization few years ago. But with slow 
PyArg_ParseTupleAndKeywords() the effect was smaller and without 
_PyUnicode_EqualToASCIIId() the code was more cumbersome.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29414] Change 'the for statement is such an iterator' in Tutorial

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee: docs@python -> rhettinger
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Roundup Robot]

Roundup Robot added the comment:


New changeset f0c1bbce09dc21d82965f1706b30b69e285ca4d3 by Serhiy Storchaka in 
branch 'master':
Issue #29421: Make int.to_bytes() and int.from_bytes() slightly faster
https://github.com/python/cpython/commit/f0c1bbce09dc21d82965f1706b30b69e285ca4d3


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29420] Python 3.6 change in dict iteration when inserting keys

[Matthew Brett]

Matthew Brett added the comment:

To clarify from comments on issue 19332:

"""
* The normal rule (not just for Python) is that a data structures have 
undefined behavior for mutating while iterating, unless there is a specific 
guarantee
"""

The change in Python 3.6 is just a different undefined behavior.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29421] Make int.to_bytes() and int.from_bytes() slightly faster

[STINNER Victor]

STINNER Victor added the comment:

3 empty notifications from Roundup Robot :-/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29079] pathlib.resolve() causes infinite loop on Windows

[Steve Dower]

Steve Dower added the comment:

At the point this code is running, it doesn't matter. The path doesn't exist, 
so trimming irrelevant segments from it will just cause a few extra iterations 
through resolve until we clear out enough of the absent segments to find 
something that does exist.

abspath just prepends the current working directory unless the path is rooted, 
so we essentially have unbounded concatenation of "\.." in that case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29263] Implement LOAD_METHOD/CALL_METHOD for C functions

[INADA Naoki]

Changes by INADA Naoki :


Added file: http://bugs.python.org/file46493/loadmethod-methoddescr-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29422] add socket independent timeout to httplib/http.client read

[Patrick Michaud]

New submission from Patrick Michaud:

Using python's httplib, there is no way to set a timeout for the full request 
cycle.  A timeout can be given, which will apply to the connection attempt and 
each blocking socket operation.  However, a malicious (or poorly performing) 
server can keep a connection open for an indefinite amount of time by slowly 
sending data.  Consider this server process:

https://gist.github.com/vegitron/bc883ddc88fe9253adc3e0bccea6445e

and this client:

https://gist.github.com/vegitron/4ee269b6492ff80d350e108363689d5c

With a timeout of 0.5, the client takes 0.501363039017 seconds.  With a timeout 
of 2.5, it takes 10.0041370392 seconds.

This is explained in the documentation, but it's a problem.  A commonly 
suggested solution is to use SIGALRM to set a timeout, but that doesn't work in 
a multi-threaded environment.  Moving to multi-process introduces big slow 
downs as I can't use connection pools, and data needs to be serialized and 
deserialized for the parent process.

I would like to propose an addition to httplib that would add a hook to 
httplib.HTTPResponse._read_chunked (python 2) and 
http.client.HTTPResponse._readall_chunked (python 3) that would either:

1) support an overall read timeout, by tracking a per-response start time and 
raising a timeout exception if that chunk read finishes after the given timeout
2) support a per-chunk callback function, so a client of httplib/http.client 
could define that logic for themselves.


Current possible timeouts, and where they can happen:

  connect  | read chunk | read chunk | read chunk
[ timeout ] [ timeout ]  [ timeout ]  [ timeout ]

Proposed addition:

  connect  | read chunk | read chunk | read chunk
[ timeout ] [ timeout ]  [ timeout ]  [ timeout ]
[total read time out]

--
messages: 286807
nosy: Patrick Michaud
priority: normal
severity: normal
status: open
title: add socket independent timeout to httplib/http.client read
type: enhancement
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29324] test_aead_aes_gcm fails on Kernel 4.9

[jan matejek]

jan matejek added the comment:

The "'0' * taglen" part is now considered part of plaintext. Which makes a lot 
of sense :) 
Removing the "empty taglen" fixes the encryption part of the tests for me. 
Similarly, for the decryption test, we must only read and check the message 
without the tag.

--
nosy: +matejcik

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29263] Implement LOAD_METHOD/CALL_METHOD for C functions

[STINNER Victor]

STINNER Victor added the comment:

loadmethod-methoddescr-2.patch LGTM.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29404] "TypeError: 'int' does not have the buffer interface" on memoryview over bytearray

[Brett Cannon]

Brett Cannon added the comment:

As Serhiy and Stefan pointed out, it's too late to change Python 2.7 semantics. 
Closing as "not a bug".

--
resolution:  -> not a bug
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29324] test_aead_aes_gcm fails on Kernel 4.9

[jan matejek]

jan matejek added the comment:

the attached patch fixes the test for me

--
keywords: +patch
Added file: http://bugs.python.org/file46494/test_socket_aead_kernel49.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29347] Python 2.7.8 is crashing while creating weakref for a given object.

[Saida Dhanavath]

Changes by Saida Dhanavath :


--
versions: +Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28749] Fixed the documentation of the mapping codec APIs

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee: docs@python -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24875] pyvenv doesn´t install PIP inside a new venv with --system-site-package

[Roundup Robot]

Roundup Robot added the comment:

New changeset 1bf1d36463b4 by Vinay Sajip in branch '3.5':
Fixes #24875: pip can now be installed in a venv with --system-site-packages.
https://hg.python.org/cpython/rev/1bf1d36463b4

New changeset 7d4701b4210f by Vinay Sajip in branch '3.6':
Fixes #24875: Merged fix from 3.5.
https://hg.python.org/cpython/rev/7d4701b4210f

New changeset 7fd931727764 by Vinay Sajip in branch 'default':
Closes #24875: Merged fix from 3.6.
https://hg.python.org/cpython/rev/7fd931727764

--
nosy: +python-dev
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Roundup Robot added the comment:

New changeset 2b9e5cbdb0b1 by Vinay Sajip in branch '3.5':
Fixes #29213: regularised EOLs of venv scripts.
https://hg.python.org/cpython/rev/2b9e5cbdb0b1

New changeset 0f3ebeb389fe by Vinay Sajip in branch '3.6':
Fixes #29213: merged fix from 3.5.
https://hg.python.org/cpython/rev/0f3ebeb389fe

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14376] sys.exit documents argument as "integer" but actually requires "subtype of int"

[Roundup Robot]

Roundup Robot added the comment:

New changeset 14682d00b09a by Mark Dickinson in branch '2.7':
Issue #14376: sys.exit now accepts longs as well as ints. Thanks Gareth Rees.
https://hg.python.org/cpython/rev/14682d00b09a

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14376] sys.exit documents argument as "integer" but actually requires "subtype of int"

[Mark Dickinson]

Changes by Mark Dickinson :


--
assignee:  -> mark.dickinson
resolution:  -> fixed
stage: test needed -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Roundup Robot added the comment:

New changeset c23224ebc9e5 by Vinay Sajip in branch 'default':
Closes #29213: Merged fix from 3.6.
https://hg.python.org/cpython/rev/c23224ebc9e5

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14376] sys.exit documents argument as "integer" but actually requires "subtype of int"

[Gareth Rees]

Gareth Rees added the comment:

Thank you, Mark (and everyone else who helped).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20185] Derby #17: Convert 49 sites to Argument Clinic across 13 files

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updated patch for the gc module is synchronized with current sources and 
addresses Martin's comments.

The only problem with this patch is that it exposes the default value for 
gc.collect() argument as constant 2 in Python instead of actual 
NUM_GENERATIONS-1. The latter value can't be exposed in Python.

On other side it is 2 in the documentation, and it is not so easy to change the 
value of NUM_GENERATIONS.

--
Added file: http://bugs.python.org/file46495/clinic_gc_v5.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Roundup Robot added the comment:


New changeset ef0264af4d1a1844c2feb32714870d636e583b3a by Vinay Sajip in branch 
'master':
Closes #29213: Merged fix from 3.6.
https://github.com/python/cpython/commit/ef0264af4d1a1844c2feb32714870d636e583b3a


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24875] pyvenv doesn´t install PIP inside a new venv with --system-site-package

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29213] python -m venv activate.bat has weird mix of line endings

[Roundup Robot]

Changes by Roundup Robot :


___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue24875] pyvenv doesn´t install PIP inside a new venv with --system-site-package

[Roundup Robot]

Roundup Robot added the comment:


New changeset fca720360f87f6dd2cd2030f7f98f689809c45d9 by Vinay Sajip in branch 
'master':
Fixes #24875: pip can now be installed in a venv with --system-site-packages.
https://github.com/python/cpython/commit/fca720360f87f6dd2cd2030f7f98f689809c45d9

New changeset e3b461cf4327bc933b2421121daa5f938d929db3 by Vinay Sajip in branch 
'master':
Fixes #24875: Merged fix from 3.5.
https://github.com/python/cpython/commit/e3b461cf4327bc933b2421121daa5f938d929db3

New changeset 2a2f5a741c509381ea8d0da09944aafdfc40b670 by Vinay Sajip in branch 
'master':
Closes #24875: Merged fix from 3.6.
https://github.com/python/cpython/commit/2a2f5a741c509381ea8d0da09944aafdfc40b670


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com