[issue33597] Compact PyGC_Head

[INADA Naoki]

New submission from INADA Naoki :

Currently, PyGC_Head takes three words; gc_prev, gc_next, and gc_refcnt.

gc_refcnt is used when collecting, for trial deletion.
gc_prev is used for tracking and untracking.

So if we can avoid tracking/untracking while trial deletion, gc_prev and 
gc_refcnt can share same memory space.

This idea reduces PyGC_Head size to two words.

--
components: Interpreter Core
messages: 317262
nosy: inada.naoki, pitrou
priority: normal
severity: normal
status: open
title: Compact PyGC_Head
type: resource usage
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[INADA Naoki]

Change by INADA Naoki :


--
keywords: +patch
pull_requests: +6682
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33596] fix memory leak in lib/json/scanner.py py_make_scanner

[yzongyue]

Change by yzongyue :


--
keywords: +patch
pull_requests: +6683
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5945] PyMapping_Check returns 1 for lists

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset f5b1183610d5888db3bbd639b1a0c945dbd8f8dd by Serhiy Storchaka in 
branch 'master':
bpo-5945: Improve mappings and sequences C API docs. (GH-7029)
https://github.com/python/cpython/commit/f5b1183610d5888db3bbd639b1a0c945dbd8f8dd


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5945] PyMapping_Check returns 1 for lists

[miss-islington]

Change by miss-islington :


--
pull_requests: +6684

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5945] PyMapping_Check returns 1 for lists

[miss-islington]

miss-islington  added the comment:


New changeset e1a78cacf65f007b1000966ce3aac6ac2eaa5cfc by Miss Islington (bot) 
in branch '3.7':
bpo-5945: Improve mappings and sequences C API docs. (GH-7029)
https://github.com/python/cpython/commit/e1a78cacf65f007b1000966ce3aac6ac2eaa5cfc


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33598] ActiveState Recipes links in docs, and the apparent closure of Recipes

[Alex Walters]

New submission from Alex Walters :

ActiveState has stopped accepting new recipes on their website, and migrated 
all current recipes to a Github repo.  I have seen no official announcement of 
a shutdown date for the code.activestate.com website, but it's future has to be 
in question considering the migration.

I propose we go through the docs for all the recipes, and either rescue them to 
a section of the docs (perhaps to a new "Code Examples" section), incorporate 
the recipes into the doc pages where appropriate, or excise them entirely.

Another option would be to update the links to the GitHub repo, but I am less 
enthusiastic about that option - GitHub repos come and go, and mutate structure 
easily.  I don't think it would be a good idea to deep link into a repository 
not controlled by the Python project.

It would stink to have a bunch of suddenly dead links in the docs.

--
assignee: docs@python
components: Documentation
messages: 317265
nosy: docs@python, rhettinger, tritium
priority: normal
severity: normal
status: open
title: ActiveState Recipes links in docs, and the apparent closure of Recipes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33599] Copying objects subclassed from SimpleNamespace doesn't work

[Sascha]

New submission from Sascha :

Try 
from types import SimpleNamespace
import copy
class Person(SimpleNamespace):
def __init__(self, name, **kwargs):
self.name = name
super().__init__(**kwargs)

bob = Person('Bob', job='tester')
clone = copy.copy(bob)

For me this results in 
TypeError: __init__() missing 1 required positional argument: 'name'

--
components: Library (Lib)
messages: 317266
nosy: stereobutter
priority: normal
severity: normal
status: open
title: Copying objects subclassed from SimpleNamespace doesn't work
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Sorry for not fixing this issue earlier. The fix is trivial, but it needed 
tests. PR 7048 adds a simpler test.

--
versions: +Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
keywords: +patch
pull_requests: +6685

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33596] fix memory leak in lib/json/scanner.py py_make_scanner

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Thank you for your report and pull request, but it is a duplicate of 
issue30877. This bug was not fixed before because we wanted a test.

--
nosy: +serhiy.storchaka
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
superseder:  -> possibe typo in json/scanner.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33599] Copying objects subclassed from SimpleNamespace doesn't work

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Right, this is because your subclass is not completely compatible with 
SimpleNamespace. The SimpleNamespace constructor accepts only keyword 
arguments, but your class requires a positional argument. You have to implement 
the __copy__ method for supporting shallow copying and the __deepcopy__ method 
for supporting deep copying. Or the __reduce__ method for supporting both 
shallow and deep copying and pickling.

--
nosy: +serhiy.storchaka
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5945] PyMapping_Check returns 1 for lists

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
pull_requests: +6686

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5945] PyMapping_Check returns 1 for lists

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:


New changeset 93e9fb5664e56c02c9aa89098b556929735b35db by Serhiy Storchaka in 
branch '3.6':
[3.6] bpo-5945: Improve mappings and sequences C API docs. (GH-7029). (GH-7049)
https://github.com/python/cpython/commit/93e9fb5664e56c02c9aa89098b556929735b35db


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[miss-islington]

Change by miss-islington :


--
pull_requests: +6687

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[miss-islington]

Change by miss-islington :


--
pull_requests: +6688

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33580] Make binary/text file glossary entries follow most common "see also" style

[Andrés Delfino]

Andrés Delfino  added the comment:

This one can be closed, right?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33580] Make binary/text file glossary entries follow most common "see also" style

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33531] test_asyncio: test_subprocess test_stdin_broken_pipe() failure on Travis CI

[STINNER Victor]

STINNER Victor  added the comment:

> Fixed failed sendfile tests on Windows (at least I hope so).

I guess that Andrew means the following fix:

commit e2537521916c5bf88fcf54d4654ff1bcd332be4a
Author: Andrew Svetlov 
Date:   Mon May 21 12:03:45 2018 +0300

Fix asyncio flaky tests (#7023)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

[STINNER Victor]

STINNER Victor  added the comment:

> the old TLS API is only available if PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT.

PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined on most (pthread) platforms, 
no? I understood that the PEP 539 is mostly designed for Cygwin, a platform 
which is not officially supported by Python. At least, 
PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is set to 1 on my Fedora 27 (Linux).

I propose to cast pthread_key_create() result to int, but only define 
PyThread_create_key() in Python/thread_pthread.h if 
PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined.

It means that the pthread implementation of Python would still have this bug 
(race condition) if PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is not defined. But 
backporting the PEP 539 to Python 2.7 doesn't seem worth it.

What do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[miss-islington]

miss-islington  added the comment:


New changeset 25fd6cc5b0ad311bb771ae47ae8173417730bb6a by Miss Islington (bot) 
in branch '3.7':
bpo-30877: Fix clearing a cache in the the JSON decoder. (GH-7048)
https://github.com/python/cpython/commit/25fd6cc5b0ad311bb771ae47ae8173417730bb6a


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33600] [EASY DOC] Python 2: document that platform.linux_distribution() has been removed

[STINNER Victor]

New submission from STINNER Victor :

The platform.linux_distribution() function has been removed from the future 
Python 3.8. The removal should be documented in Python 2.7 documentation, and 
mention the alternative.

Python 2 documenation:
https://docs.python.org/2/library/platform.html#platform.linux_distribution

Python 3 documentation with the deprecation warning:
https://docs.python.org/3/library/platform.html#platform.linux_distribution

IMHO it's too late to deprecate the function from Python 2. We should only add 
a quick note.

Function removed in: bpo-28167.

--
assignee: docs@python
components: Documentation
messages: 317275
nosy: docs@python, vstinner
priority: normal
severity: normal
status: open
title: [EASY DOC] Python 2: document that platform.linux_distribution() has 
been removed
versions: Python 2.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33600] [EASY DOC] Python 2: document that platform.linux_distribution() has been removed

[STINNER Victor]

Change by STINNER Victor :


--
keywords: +easy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

[Petr Viktorin]

Petr Viktorin  added the comment:

> PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined on most (pthread) platforms, 
> no?

> I propose to cast pthread_key_create() result to int, but only define 
> PyThread_create_key() in Python/thread_pthread.h if 
> PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined.

I don't think that's a good idea. Changing API, even for platforms that aren't 
officially supported, sounds very harsh this late in the release cycle.

But! I suppose we could fix the bug only for platforms with 
PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT. Other platforms would keep the current 
implementation -- they'd still have the bug, but the API would stay unchanged.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[miss-islington]

miss-islington  added the comment:


New changeset 2baee0aa77055755ac50e92e64bbccfea4108621 by Miss Islington (bot) 
in branch '3.6':
bpo-30877: Fix clearing a cache in the the JSON decoder. (GH-7048)
https://github.com/python/cpython/commit/2baee0aa77055755ac50e92e64bbccfea4108621


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20941] pytime.c:184 and pytime.c:218: runtime error, outside the range of representable values of type 'long'

[STINNER Victor]

STINNER Victor  added the comment:

I close the issue as a duplicate of bpo-31373. Reopen/comment bpo-31373 if the 
issue is not completely fixed.

--
resolution: out of date -> duplicate
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

[STINNER Victor]

STINNER Victor  added the comment:

> Wait, I didn't notice the change to the format of raw timings. It looks as a 
> regression to me.

Do you mean that some applications may run timeit as a CLI and parse stdout to 
get raw values? Why doing so? timeit is a Python module, it's trivial to use 
its API to avoid using the CLI, no?

I don't think that the CLI output must not change.

master branch:

vstinner@apu$ ./python -m timeit -v '[1,2]*1000'
1 loop -> 1.73e-05 secs
2 loops -> 6.49e-05 secs
5 loops -> 0.000107 secs
10 loops -> 0.000173 secs
20 loops -> 0.000331 secs
50 loops -> 0.000798 secs
100 loops -> 0.00159 secs
200 loops -> 0.00304 secs
500 loops -> 0.00777 secs
1000 loops -> 0.0163 secs
2000 loops -> 0.0315 secs
5000 loops -> 0.0775 secs
1 loops -> 0.154 secs
2 loops -> 0.311 secs

raw times: 310 msec, 313 msec, 308 msec, 303 msec, 304 msec

2 loops, best of 5: 15.2 usec per loop

Python 3.6:

vstinner@apu$ python3 -m timeit -v '[1,2]*1000'
10 loops -> 3.41e-05 secs
100 loops -> 0.000345 secs
1000 loops -> 0.00327 secs
1 loops -> 0.0332 secs
10 loops -> 0.325 secs
raw times: 0.319 0.316 0.319
10 loops, best of 3: 3.16 usec per loop

Hum, the timings of the calibration (xx loops -> ...) should use the same 
function to format time to use ns, ms, etc.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33518] Add PEP to glossary

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 8cbde8a43769641373a681af4b0d72944af43f95 by Victor Stinner 
(Andrés Delfino) in branch '2.7':
[2.7] bpo-33518: Add PEP entry to documentation glossary (GH-6860). (#6936)
https://github.com/python/cpython/commit/8cbde8a43769641373a681af4b0d72944af43f95


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33518] Add PEP to glossary

[STINNER Victor]

STINNER Victor  added the comment:

Thank you Andrés Delfino. I applied your PR to 2.7, 3.6, 3.7 and master.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33521] Add 1.32x faster C implementation of asyncio.isfuture().

[STINNER Victor]

STINNER Victor  added the comment:

> @pitrou We'll measure the wins of gather when we implement it in C. Before 
> that, we need to get all helpers ready in C.

You don't have to provide _asyncio.isfuture() (in C) to implement gather() in C.

If your goal is to optimize gather(), write a change to only implement gather() 
no?

What's the point of optimizing isfuture() is gather() is implemented in C and 
doesn't call the Python implementation anymore?

Implement isfuture() is C and expose it as _asyncio.isfuture() are two 
different things.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28167] remove platform.linux_distribution()

[STINNER Victor]

STINNER Victor  added the comment:

FYI I created bpo-33600: "[EASY DOC] Python 2: document that 
platform.linux_distribution() has been removed".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23860] Windows: Failure to check return value from lseek() in Modules/mmapmodule.c

[STINNER Victor]

STINNER Victor  added the comment:

"At the moment, I'm not sure if it's needed or not, but if it's only an
issue with XP, then it might not be worth fixing...:)"

If the workaround is no longer needed, the lseek() call should be removed.

--
components: +Windows
nosy: +paul.moore, steve.dower, tim.golden, zach.ware
title: Failure to check return value from lseek() in Modules/mmapmodule.c -> 
Windows: Failure to check return value from lseek() in Modules/mmapmodule.c

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23860] Windows: Failure to check return value from lseek() in Modules/mmapmodule.c

[STINNER Victor]

STINNER Victor  added the comment:

Oh right, PR 7017 does that: it removes the lseek() call ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33601] Py_UTF8Mode is not documented

[Serhiy Storchaka]

New submission from Serhiy Storchaka :

Py_UTF8Mode was added to the limited API in 3.7, but it is not documented 
anywhere.

--
assignee: docs@python
components: Documentation
messages: 317286
nosy: docs@python, ncoghlan, serhiy.storchaka, vstinner
priority: normal
severity: normal
status: open
title: Py_UTF8Mode is not documented
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33470] Changes from GH-1638 (GH-3575, bpo-28411) are not documented in Porting to Python 3.7

[STINNER Victor]

STINNER Victor  added the comment:

> Also, the actual removal of the "modules" field was reverted.

Oh... I didn't understand that part :-) Ok. In this case it's fine to close 
this documentation issue. Nothing should be documented for 3.7 ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33197] Confusing error message when constructing invalid inspect.Parameters

[Dong-hee Na]

Dong-hee Na  added the comment:

Can I get a code review for PR 6636?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33565] strange tracemalloc results

[STINNER Victor]

STINNER Victor  added the comment:

A memory leak is when each iteration adds a fixed number of bytes of the 
memory: I'm talking about tracemalloc.get_traced_memory()[0] value. For 
example, if you compare the total traced memory between your iteration 30 and 
iteration 20, you get a value 10N. If you compare iteration 100 and 20, I would 
expect the value 80N. Is it the case?

You can get the number of allocated bytes from a snapshot using:

>>> sum(trace.size for trace in snapshot.traces)
2620350

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28240] Enhance the timeit module: display average +- std dev instead of minimum

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Yes, it was my thought. But seems you are right, it is easier to use Python as 
a programming language. In the past I used the CLI because the programming 
interface didn't supported autoranging.

Although I would change the human-readable output to

raw times (msec): 310 313 308 303 304

But it may be too later for 3.7.

--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29640] _PyThreadState_Init and fork race leads to inconsistent key list

[STINNER Victor]

STINNER Victor  added the comment:

> But! I suppose we could fix the bug only for platforms with 
> PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT.

Yes, this is my proposal.

>> I propose to cast pthread_key_create() result to int, but only define 
>> PyThread_create_key() in Python/thread_pthread.h if 
>> PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined.

> I don't think that's a good idea. Changing API, even for platforms that 
> aren't officially supported, sounds very harsh this late in the release cycle.

Which API change? I don't propose to modify the existing public C API "int 
PyThread_create_key(void)". I only propose to change it's implementation to the 
native pthread API when PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT is defined.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33601] [EASY DOC] Py_UTF8Mode is not documented

[STINNER Victor]

STINNER Victor  added the comment:

I wasn't sure if I should document it, but after talking with Serhiy on IRC, I 
now agree that the new variable should be documented.

It should be documented at:
https://docs.python.org/dev/c-api/init.html#global-configuration-variables

--
keywords: +easy
title: Py_UTF8Mode is not documented -> [EASY DOC] Py_UTF8Mode is not documented

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[Andrés Delfino]

New submission from Andrés Delfino :

Data Types mentions sets (which are now built-in) and synchronized queues (now 
mentioned in Concurrent Execution). I'm proposing fixing this.

PR also adds mention to bytearray.

--
assignee: docs@python
components: Documentation
messages: 317293
nosy: adelfino, docs@python
priority: normal
severity: normal
status: open
title: Remove set and queue references from Data Types
type: enhancement
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33602] Remove set and queue references from Data Types

[Andrés Delfino]

Change by Andrés Delfino :


--
keywords: +patch
pull_requests: +6689
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33601] [EASY DOC] Py_UTF8Mode is not documented

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

If there are reasons of including it in the limited API, it should be 
documented.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[INADA Naoki]

INADA Naoki  added the comment:

$ ./python -m perf compare_to master.json twogc.json -G  --min-speed=2
Slower (3):
- scimark_monte_carlo: 268 ms +- 9 ms -> 278 ms +- 8 ms: 1.04x slower (+4%)
- fannkuch: 1.03 sec +- 0.02 sec -> 1.06 sec +- 0.02 sec: 1.03x slower (+3%)
- spectral_norm: 285 ms +- 9 ms -> 291 ms +- 6 ms: 1.02x slower (+2%)

Faster (13):
- unpickle_pure_python: 980 us +- 13 us -> 886 us +- 11 us: 1.11x faster (-10%)
- pickle_dict: 71.9 us +- 6.8 us -> 67.2 us +- 0.4 us: 1.07x faster (-7%)
- mako: 40.5 ms +- 1.1 ms -> 38.7 ms +- 0.4 ms: 1.05x faster (-5%)
- scimark_lu: 396 ms +- 18 ms -> 381 ms +- 16 ms: 1.04x faster (-4%)
- genshi_text: 89.3 ms +- 1.2 ms -> 86.3 ms +- 1.2 ms: 1.03x faster (-3%)
- xml_etree_generate: 270 ms +- 5 ms -> 262 ms +- 5 ms: 1.03x faster (-3%)
- regex_dna: 286 ms +- 1 ms -> 279 ms +- 1 ms: 1.03x faster (-3%)
- scimark_sor: 511 ms +- 6 ms -> 497 ms +- 10 ms: 1.03x faster (-3%)
- xml_etree_iterparse: 220 ms +- 6 ms -> 214 ms +- 5 ms: 1.03x faster (-3%)
- xml_etree_process: 231 ms +- 4 ms -> 225 ms +- 4 ms: 1.02x faster (-2%)
- genshi_xml: 191 ms +- 2 ms -> 187 ms +- 2 ms: 1.02x faster (-2%)
- unpack_sequence: 131 ns +- 2 ns -> 129 ns +- 2 ns: 1.02x faster (-2%)
- richards: 180 ms +- 4 ms -> 176 ms +- 2 ms: 1.02x faster (-2%)

Benchmark hidden because not significant (44)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33238] AssertionError on await of Future returned by asyncio.wrap_future

[Jason Haydaman]

Change by Jason Haydaman :


--
keywords: +patch
pull_requests: +6690
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends

[GranPrego]

New submission from GranPrego :

On windows 7 / 10 I'm using subprocess to launch a dos cmdline executable and 
returning the results, which is all working fine.
However, each time I make a call, the Python handle count is gradually 
increasing, jumping up , back a few, then jumping up and so on.

All the handles are released when the script exits, but quite often python just 
hangs after a few hours.  If I use process explorer to investigate I can see 
that python has an increasing number of Thread handles, even though I can see 
the process being called and cleanly exiting.

Unfortunately I'm stuck with the dos executable and it's always a one shot of 
sending it a single command each time and the script calls it a lot. The 
executable is just taking a string cmdline and returning a couple of lines of 
text and then exiting.  It only runs for a couple of seconds at most.

I've tried two variants of calling the process, I was hoping that the with 
variant would clean up, but there is no difference.

Each handle object that gets left behind has a single reference and a non paged 
quota of 1192, 0 paged.

The script is long running and I've seen the handle count reach 46K.


result = ""
with Popen ([fcptool, parameters],  stdout=PIPE, universal_newlines=True, 
bufsize=1) as process:
for line in process.stdout:
result = result + line

return result
or 
p = subprocess.run([fcptool, parameters], stdout=subprocess.PIPE,  
stderr=subprocess.STDOUT, universal_newlines=True, shell=True).stdout


return p

I can reproduce this on 3 different machines, 2 windows 7 and one windows 10, 
all Python 3.6.   I can't see a way around this at the moment and as far as I 
can tell, I'm using the call to subprocess correctly.

--
components: Interpreter Core, Windows
messages: 317296
nosy: GranPrego, paul.moore, steve.dower, tim.golden, zach.ware
priority: normal
severity: normal
status: open
title: Subprocess Thread handles  grow with each call and aren't released until 
script ends
type: resource usage
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Interesting.  Do you have any comparisons on memory footprint too?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[INADA Naoki]

INADA Naoki  added the comment:

$ ./python-gc -c 'import asyncio,sys; sys._debugmallocstats()'

master:

# bytes in allocated blocks=4,011,368
# bytes in available blocks=  136,640
50 unused pools * 4096 bytes   =  204,800
# bytes lost to pool headers   =   49,824
# bytes lost to quantization   =   53,816
# bytes lost to arena alignment=0
Total  =4,456,448

patched:

# bytes in allocated blocks=3,852,432
# bytes in available blocks=  132,664
27 unused pools * 4096 bytes   =  110,592
# bytes lost to pool headers   =   47,856
# bytes lost to quantization   =   50,760
# bytes lost to arena alignment=0
Total  =4,194,304

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

This is an interesting idea.

The other problem with the garbage collecting is that it modifies the memory of 
all collectable objects. This leads to deduplicating virtually all memory 
blocks after the fork, even if these objects are not used in the child.

If gc_refcnt is used only when collecting, what if allocate a linear array for 
them for that time? This will save less memory (only one word per object in the 
peak), but avoid modifying the memory of not collected objects (except pointers 
at the ends of generation lists and neighbors of collected objects).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2504] Add gettext.pgettext() and variants support

[Éric Araujo]

Éric Araujo  added the comment:

It was an answer to «Is there anything I can do to help get this into the 
codebase»

Feel free to take this on!  I’ll try to review.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30877] possibe typo in json/scanner.py

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[INADA Naoki]

INADA Naoki  added the comment:

@Serhiy

php implemented similar idea recently.
https://react-etc.net/entry/improvements-to-garbage-collection-gc-php-7-3-boosts-performance-in-benchmark

In short, each tracked object have only "index" of GC struct, not "pointer".
GC struct is in array and it can be resized.

I tried to copy it, but there are some challenges:

* _PyObject_GC_TRACK() will resize GC array and cause MemoryError.  It's not 
API compatible.
* php's GC is not generational. This design may slow down moving objects 
between generation.
* We need one word (index) for object header and two words (refcnt, pointer to 
the object) for GC struct.  It means we can reduce memory footprint only for 
untracked dicts and tuples.

And this is my first time GC hack.  So I gave up PHP way and choose easier way.

Anyway, we have gc.freeze() now which can be used for avoid CoW after fork.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Le 22/05/2018 à 17:31, INADA Naoki a écrit :
> 
> INADA Naoki  added the comment:
> 
> $ ./python-gc -c 'import asyncio,sys; sys._debugmallocstats()'

Thanks.  You can also collect peak memory stats during the benchmark
suite, though the numbers may be approximate.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33593] Support heapq on typed arrays?

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I don't think we should go down this path.  The efficiency of the C 
implementation depends on it being tightly coupled to lists.  This tool is used 
in the schedulers of various async tools (such as Tornando), used for merge(), 
nsmallest(), and nlargest() all of which depend on this foundational tool being 
very fast.

Also, I question whether it makes sense at all to be heapifying numpy arrays 
using standard library tooling.  It numpy arrays actually needed this and 
needed for it to be efficient, it would need to be implemented natively in 
numpy.

--
nosy: +rhettinger
versions: +Python 3.8 -Python 2.7, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33590] sched.enter priority has no impact on execution

[Raymond Hettinger]

Raymond Hettinger  added the comment:

It would be nice to either modify the example or add another example to show 
the use of enterabs() and of the priority field being used as a tie breaker for 
two events scheduled at the same time.

--
nosy: +rhettinger
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33565] strange tracemalloc results

[Alexander Mohr]

Alexander Mohr  added the comment:

I believe your method is flawed, when enabling tracemalloc it's going to be 
using memory as well to store the traces.  I still believe you need to use the 
method I mentioned and further even if we don't take into account the total 
memory the traces I mentioned need to be explained.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[INADA Naoki]

INADA Naoki  added the comment:

In Doc folder:

  make clean
  make PYTHON=../python venv
  /usr/bin/time make html

master:

  113.15user 0.41system 1:55.46elapsed 98%CPU (0avgtext+0avgdata 
205472maxresident)k
  18800inputs+223544outputs (1major+66066minor)pagefaults 0swaps

  111.07user 0.44system 1:51.72elapsed 99%CPU (0avgtext+0avgdata 
205052maxresident)k
  0inputs+223376outputs (0major+65855minor)pagefaults 0swaps

patched:

  109.92user 0.44system 1:50.43elapsed 99%CPU (0avgtext+0avgdata 
195832maxresident)k
  0inputs+223376outputs (0major+63206minor)pagefaults 0swaps

  110.70user 0.40system 1:51.50elapsed 99%CPU (0avgtext+0avgdata 
195516maxresident)k
  0inputs+223376outputs (0major+62723minor)pagefaults 0swaps

It seems reduced 5% memory footprint, and performance difference is very small.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23188] Provide a C helper function to chain raised (but not yet caught) exceptions

[Eric Snow]

Eric Snow  added the comment:

FTR, see PEP 490 ("Chain exceptions at C level") which proposed implicitly 
chaining exceptions in the PyErr_* API.

While that PEP was rejected (not all exceptions should be chained), it does 
make a good point about the clunkiness of using _PyErr_ChainExceptions():

PyObject *exc, *val, *tb;
PyErr_Fetch(&exc, &val, &tb);
PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
_PyErr_ChainExceptions(exc, val, tb);

So if we are going to add a public helper function, let's consider adding one 
that simplifies usage.  For instance, how about one that indicates the next 
exception set should chain:

PyErr_ChainNext();
PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);

Or perhaps we should revive PEP 490 with an opt out mechanism for the cases 
where we don't want chaining:

PyErr_NoChainNext();
PyErr_Format(PyExc_RuntimeError, "uh-oh");

--
nosy: +eric.snow, vstinner
versions: +Python 3.8 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33565] strange tracemalloc results

[STINNER Victor]

STINNER Victor  added the comment:

Oh. Usually, I strip traces allocated by tracemalloc using filters:

http://pytracemalloc.readthedocs.io/examples.html#pretty-top

snapshot = snapshot.filter_traces((
tracemalloc.Filter(False, ""),
tracemalloc.Filter(False, ""),
))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33565] strange tracemalloc results

[Alexander Mohr]

Alexander Mohr  added the comment:

that's not going to affect 
http://pytracemalloc.readthedocs.io/api.html#get_traced_memory.  There is no 
filter for that :)

as to your sum that's exactly what my original callstack lists:
21 memory blocks: 4.7 KiB

this means 21 blocks were not released, and in this case leaked because nothing 
should be held onto after the first iteration (creating the initial connector 
in the connection pool.  In the head object case that's going to be a new 
connector per iteration, however the old one should go away.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33592] Document contextvars C API

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset b2f5f59ae15564b991f3ca4850e6ad28d9faacbc by Yury Selivanov (Elvis 
Pranskevichus) in branch 'master':
bpo-33592: Document the C API in PEP 567 (contextvars) (GH-7033)
https://github.com/python/cpython/commit/b2f5f59ae15564b991f3ca4850e6ad28d9faacbc


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33592] Document contextvars C API

[miss-islington]

Change by miss-islington :


--
pull_requests: +6691

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33593] Support heapq on typed arrays?

[Diego Argueta]

Diego Argueta  added the comment:

I was referring to the C arrays in the Python standard library: 
https://docs.python.org/3/library/array.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends

[Eryk Sun]

Eryk Sun  added the comment:

The thread handle that CreateProcess returns gets immediately closed in 
Popen._execute_child, so I can't see how this is due to subprocess. Please 
check to make sure these thread handles aren't for threads in your own process. 
Set the lower pane of Process Explorer to show the handle list. For a thread 
handle, the name field shows the executable name and PID of the process to 
which the thread is attached.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33597] Compact PyGC_Head

[Yury Selivanov]

Yury Selivanov  added the comment:

This is such a great idea. +1 from me.

--
nosy: +yselivanov

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23188] Provide a C helper function to chain raised (but not yet caught) exceptions

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

There is usually more complex code between PyErr_Fetch() and 
_PyErr_ChainExceptions():

PyObject *exc, *val, *tb, *close_result;
PyErr_Fetch(&exc, &val, &tb);
close_result = _PyObject_CallMethodId(result, &PyId_close, NULL);
_PyErr_ChainExceptions(exc, val, tb);
Py_XDECREF(close_result);

Many exceptions can be raised and silenced or overridden between, but we are 
interesting in chaining the only latest exception (or restoring the original 
exception if all exceptions between were silenced).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33593] Support heapq on typed arrays?

[Diego Argueta]

Diego Argueta  added the comment:

However I do see your point about the speed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33592] Document contextvars C API

[miss-islington]

miss-islington  added the comment:


New changeset afec2d583a06849c5080c6cd40266743c8e04b3e by Miss Islington (bot) 
in branch '3.7':
bpo-33592: Document the C API in PEP 567 (contextvars) (GH-7033)
https://github.com/python/cpython/commit/afec2d583a06849c5080c6cd40266743c8e04b3e


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends

[Antoine Pitrou]

Change by Antoine Pitrou :


--
nosy: +giampaolo.rodola, gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33593] Support heapq on typed arrays?

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Workaround:

alist = list(a)
heapq.heapify(alist)
a[:] = alist

And it should be not much slower than using heapq.heapify() directly if it 
could support general sequences. Using it with array.array would add 
significant overhead due to boxing.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends

[GranPrego]

GranPrego  added the comment:

Process explorer is showing the handles as belonging to the python executable. 
I can see the cmd process start,then the executable which terminates cleanly.  
I can see thread handles appearing under the python process, with some 
terminating, but more green than red, hence the increase.  I can post a 
screenshot tomorrow. Thanks

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30940] Documentation for round() is incorrect.

[Serhiy Storchaka]

Change by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Ned Deily]

Ned Deily  added the comment:

> Considering the huge popularity of these SO questions, I don't think this 
> should be reverted [...]

As I understand it (and, again, I make no claim to be an argparse expert), 
there does not seem to be one absolutely correct answer here; there has to be a 
tradeoff.  If we revert the change in default as in PR 6919, users porting code 
from 2.7 will continue to run into the unfortunate change in behavior 
introduced in 3.3.  But, with the reversion, those users are no worse off than 
they were before: the existing workarounds, like those in the cited SO answers, 
still apply.  And it's a one-time annoyance for them, along with all the other 
changes they may need to make to port to a current Python 3.x.  Whereas, if the 
change is not reverted, then we introduce a new incompatibility to a new class 
of users, that is, those upgrading from Python 3.3 through 3.6 to 3.7, 
generating a new set of SO questions, etc.  That seems to be making a 
less-than-ideal situation worse.  So, as release manager, I continue to think 
that the reversion (PR 6919) should go in to 3.7.0.  (For 3.8 and beyond, it 
 would be great to have at least one core developer take responsibility for 
argparse enhancements.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33531] test_asyncio: test_subprocess test_stdin_broken_pipe() failure on Travis CI

[Terry J. Reedy]

Terry J. Reedy  added the comment:

The patch seems to have worked.  The last AppVeyor failure was a day ago, when 
testing the 3.7 backport of the fix.
https://ci.appveyor.com/project/python/cpython/history

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Anthony Sottile]

Anthony Sottile  added the comment:

Is there then no pathway for actually fixing the bug?  aka how can I get 
`required=True` to be the default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33604] HMAC default to MD5 marked as to be removed in 3.6

[Matthias Bussonnier]

New submission from Matthias Bussonnier :

HMAC docs says digestmod=md5 default will be removed in 3.6, but was not. 

We should likely bumpt that to 3.8 in the documentation, and actually remove it 
in 3.8

--
messages: 317322
nosy: mbussonn
priority: normal
severity: normal
status: open
title: HMAC default to MD5 marked as to be removed in 3.6
versions: Python 3.6, Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33604] HMAC default to MD5 marked as to be removed in 3.6

[Matthias Bussonnier]

Change by Matthias Bussonnier :


--
keywords: +patch
pull_requests: +6692
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33603] Subprocess Thread handles grow with each call and aren't released until script ends

[Eryk Sun]

Eryk Sun  added the comment:

The 2nd example with subprocess.run() creates two threads in the Python 
process, since you're redirecting both stdout and stderr to pipes and run() 
calls communicate(). The first example with subprocess.Popen() shouldn't create 
any threads. In either case, nothing in subprocess should be opening a handle 
for a thread.

Please attach a minimal script that reproduces the problem, preferably running 
a command everyone can test such as "python.exe -V" and preferably with 
shell=False if the problem can be reproduced without the shell. Also, describe 
your Python setup, i.e. the installed distribution and packages. Something 
could be monkey patching the subprocess module.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33605] Detect accessing event loop from a different thread outside of _debug

[Hrvoje Nikšić]

New submission from Hrvoje Nikšić :

Looking at StackOverflow's python-asyncio tag[1], it appears that it's a very 
common mistake for users to invoke asyncio functions or methods from a thread 
other than the event loop thread. In some cases this happens because the user 
is careless with threads and hasn't read the documentation. But in many cases 
what happens is that a third-party library invoked a callback in a different 
thread without making it transparent that that's what it's doing.

The trouble is that in many cases doing these things, e.g. calling 
loop.call_soon() or loop.create_task() from the wrong thread, will *appear to 
work*. The typical event loop is busy servicing different coroutines, so a 
function or task enqueued without a proper wakeup gets picked up soon enough. 
This is, of course, a disaster waiting to happen because it could easily lead 
to corruption of event loop's data structures. But users don't know that, and 
many of them become aware of the problem only after wondering "why does my code 
start working when I add a coroutine that does nothing but asyncio.sleep(0.1) 
in an infinite loop?" Some may never even fix their code, just assuming that 
asyncio takes a long time to process a new task or something like that.

I suggest that asyncio should be stricter about this error and that methods and 
functions that operate on the event loop, such as call_soon, call_later, 
create_task, ensure_future, and close, should all call _check_thread() even 
when not in debug mode. _check_thread() warns that it "should only be called 
when self._debug == True", hinting at "performance reasons", but that doesn't 
seem justified. threading.get_ident() is efficiently implemented in C, and 
comparing that integer to another cached integer is about as efficient an 
operation as it gets.

The added benefit would be a vast improvement of robustness of asyncio-based 
programs, saving many hours of debugging.


[1]
Here is an incomplete list of questions where the users stumbled on this 
problem, and that's only from the last three months or so:

https://stackoverflow.com/questions/49906034/python-asyncio-run-forever-and-tasks
https://stackoverflow.com/questions/49851514/python-websockets-and-gtk-confused-about-asyncio-queue
https://stackoverflow.com/questions/49533612/using-asyncio-loop-reference-in-another-file
https://stackoverflow.com/questions/49093623/strange-behaviour-when-task-added-to-empty-loop-in-different-thread
https://stackoverflow.com/questions/48836285/python-asyncio-event-wait-not-responding-to-event-set
https://stackoverflow.com/questions/48833644/how-to-wait-for-asynchronous-callback-in-the-background-i-e-not-invoked-by-us
https://stackoverflow.com/questions/48695670/running-asynchronous-code-synchronously-in-separate-thread

--
components: asyncio
messages: 317324
nosy: asvetlov, hniksic, yselivanov
priority: normal
severity: normal
status: open
title: Detect accessing event loop from a different thread outside of _debug
type: enhancement
versions: Python 3.7, Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

I tried to use add_subparsers() with required=True and have found it not usable.

import argparse
parser = argparse.ArgumentParser(prog='PROG')
subparsers = parser.add_subparsers(required=True)
parser_a = subparsers.add_parser('a')
parser_b = subparsers.add_parser('b')
parser.parse_args([])

The result:

Traceback (most recent call last):
  File "", line 1, in 
  File "/home/serhiy/py/cpython/Lib/argparse.py", line 1745, in parse_args
args, argv = self.parse_known_args(args, namespace)
  File "/home/serhiy/py/cpython/Lib/argparse.py", line 1777, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
  File "/home/serhiy/py/cpython/Lib/argparse.py", line 2012, in 
_parse_known_args
', '.join(required_actions))
TypeError: sequence item 0: expected str instance, NoneType found

Seems that not only the default value should be changed, but the whole PR 3027 
should be reverted.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Anthony Sottile]

Anthony Sottile  added the comment:

That's a separate issue (also a bug introduced by the bad 3.3 patch): 
https://bugs.python.org/issue29298

I have an open PR to fix it as well but it has not seen review action: 
https://github.com/python/cpython/pull/3680

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33593] Support heapq on typed arrays?

[Raymond Hettinger]

Raymond Hettinger  added the comment:

As noted by Serhiy, the interaction with the Array type would incur significant 
overhead.  Your fastest approach will be to follow his suggest to first convert 
to a list and then perform heap manipulations.

Marking this as closed.  Thank you for the suggestion.

--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33603] Subprocess Thread handles grow with each call and aren't released [Windows]

[Gregory P. Smith]

Change by Gregory P. Smith :


--
title: Subprocess Thread handles  grow with each call and aren't released until 
script ends -> Subprocess Thread handles grow with each call and aren't 
released [Windows]

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33606] Improve logging performance when logger disabled

[Vinay Sajip]

New submission from Vinay Sajip :

If a logger is disabled (by setting it's disabled attribute to True), the check 
for this is done late in the dispatch of the logging event - during the 
handle() call - rather than isEnabledFor(), which would short-circuit some 
processing. So the check for logger.disabled should be moved to isEnabledFor().

Credit to Abhijit Gadgil for raising this:

https://stackoverflow.com/questions/50453121/logger-disabled-check-much-later-in-python-logging-module-whats-the-rationale

--
assignee: vinay.sajip
components: Library (Lib)
messages: 317328
nosy: vinay.sajip
priority: normal
severity: normal
stage: needs patch
status: open
title: Improve logging performance when logger disabled
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Wouldn't be better to first fix this bug, and only after that add the 
'required' parameter?

Adding it introduced yet one bug: when pass arguments as positional, the 'help' 
argument will be swallowed. You can add new parameters only after existing 
positional parameters.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33607] Explicitly track object ownership (and allocator).

[Eric Snow]

New submission from Eric Snow :

When an object is created it happens relative to the current
thread (ergo interpreter) and the current allocator (part of
global state).  We do not track either of these details for
the object.  It may make sense to start doing so (reasons next).

Regarding tracking the interpreter, that originating interpreter
can be thought of as the owner.  Any lifecycle operations should
happen relative to that interpreter.  Furthermore, the object
should be used in C-API calls only in that interpreter (i.e.
when the current thread's Py_ThreadState belongs to that
interpreter).  This hasn't been an issue since currently all
interpreters in the process share the GIL, as well as the fact
that subinterpreters haven't been heavily used historically.
However, the possibility of no longer sharing the GIL suggests
that tracking the owning interpreter (and perhaps even other
"sharing" interpreters) would be important.  Furthermore,
in the last few years subinterpreters have seen increasing usage
(see Openstack Ceph), and knowing the originating interpreter
for an object can be useful there.  Regardless, even in the
single interpreter case knowing the owning interpreter is
important during runtime finalization (which is currently
slightly broken), which impacts CPython embedders.

Regarding the allocator, there used to be just a single global
one that the runtime used from start to finish.  Now the C-API
offers a way to switch the allocator, so there's no guarantee
that the right allocator is used in PyMem_Free().  This has
already had a negative impact on efforts to clean up CPython's
runtime initialization.  It also results in problems during
finalization.  Additionally, we are looking into moving the
allocator from the global runtime state to the per-interpreter
(or even per-thread or per-context) state value.  In that world
it would be essential to know which allocator was used when
creating the object.  There are other possible applications
based on knowing an object's allocator, but I'll stop there.

To sort all this out we would need to track per-object:

* originating allocator (pointer or id)
* owning interpreter (pointer or id)
* (possibly) "sharing" interpreters (linked list?)

Either we'd add 2 pointer-size fields to PyObject or we would
keep a separate hash table (or two) pointing from each object
to the info (similar to how we've considered doing for
refcounts).  To alleviate impact on the common case (not
embedded, single interpreter, same allocator), we could default
to not tracking interpreter/allocator and take a lookup failure
to mean "main interpreter, default allocator".

--
messages: 317330
nosy: eric.snow, ncoghlan, vstinner
priority: normal
severity: normal
status: open
title: Explicitly track object ownership (and allocator).
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Anthony Sottile]

Anthony Sottile  added the comment:

The bug is orthogonal, you can trigger it without the `required=` keyword 
argument via the (currently suggested) monkeypatch workaround which restores 
the pre-3.3 behaviour:

import argparse

parser = argparse.ArgumentParser()
subp = parser.add_subparsers()
subp.add_parser('test')
subp.required = True
parser.parse_args()


$ python3 test.py
Traceback (most recent call last):
  File "test.py", line 7, in 
parser.parse_args()
  File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py",
 line 1730, in parse_args
args, argv = self.parse_known_args(args, namespace)
  File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py",
 line 1762, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
  File 
"/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/argparse.py",
 line 1997, in _parse_known_args
', '.join(required_actions))
TypeError: sequence item 0: expected str instance, NoneType found


Also note that when `dest` is specified it works fine:


import argparse

parser = argparse.ArgumentParser()
subp = parser.add_subparsers(dest='cmd')
subp.add_parser('test')
subp.required = True
parser.parse_args()

$ python3 test.py
usage: test.py [-h] {test} ...
test.py: error: the following arguments are required: cmd

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33109] argparse: make new 'required' argument to add_subparsers default to False instead of True

[Serhiy Storchaka]

Serhiy Storchaka  added the comment:

Please ignore the last paragraph. It was my mistake, all add_subparsers() 
parameters are keyword-only, and _SubParsersAction is a privale class.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

[Eric Snow]

New submission from Eric Snow :

In order to keep subinterpreters properly isolated, objects
from one interpreter should not be used in C-API calls in
another interpreter.  That is relatively straight-forward
except in one case: indicating that the other interpreter
doesn't need the object to exist any more (similar to
PyBuffer_Release() but more general).  I consider the
following solutions to be the most viable.  Both make use
of recounts to protect cross-interpreter usage (with incref
before sharing).

1. manually switch interpreters (new private function)
  a. acquire the GIL
  b. if refcount > 1 then decref and release the GIL
  c. switch
  d. new thread (or re-use dedicated one)
  e. decref
  f. kill thread
  g. switch back
  h. release the GIL
2. change pending call mechanism (see Py_AddPendingCall) to
   per-interpreter instead of global (add "interp" arg to
   signature of new private C-API function)
  a. queue a function that decrefs the object
3. new cross-interpreter-specific private C-API function
  a. queue the object for decref (a la Py_AddPendingCall)
 in owning interpreter

I favor #2, since it is more generally applicable.  #3 would
probably be built on #2 anyway.  #1 is relatively inefficient.
With #2, Py_AddPendingCall() would become a simple wrapper
around the new private function.

--
messages: 317333
nosy: eric.snow, ncoghlan, serhiy.storchaka, vstinner, yselivanov
priority: normal
severity: normal
stage: needs patch
status: open
title: Add a cross-interpreter-safe mechanism to indicate that an object may be 
destroyed.
versions: Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33608] Add a cross-interpreter-safe mechanism to indicate that an object may be destroyed.

[Eric Snow]

Eric Snow  added the comment:

As a lesser (IMHO) alternative, we could also modify Py_DECREF
to respect a new "shared" marker of some sort (perhaps relative
to #33607), but that would probably still require one of the
refcount-based solutions (and add a branch to Py_DECREF).

--
versions: +Python 3.8 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23188] Provide a C helper function to chain raised (but not yet caught) exceptions

[Eric Snow]

Eric Snow  added the comment:

good point :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33516] unittest.mock: Add __round__ to supported magicmock methods

[STINNER Victor]

STINNER Victor  added the comment:


New changeset 6c4fab0f4b95410a1a964a75dcdd953697eff089 by Victor Stinner (John 
Reese) in branch 'master':
bpo-33516: Add support for __round__ in MagicMock (GH-6880)
https://github.com/python/cpython/commit/6c4fab0f4b95410a1a964a75dcdd953697eff089


--
nosy: +vstinner

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33516] unittest.mock: Add __round__ to supported magicmock methods

[STINNER Victor]

STINNER Victor  added the comment:

Thank you Martijn Pieters for the feature request/bug report, and thanks John 
Reese for the implementation!

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33565] strange tracemalloc results

[STINNER Victor]

STINNER Victor  added the comment:

> this means 21 blocks were not released, and in this case leaked because 
> nothing should be held onto after the first iteration (creating the initial 
> connector in the connection pool.  In the head object case that's going to be 
> a new connector per iteration, however the old one should go away.

I'm not sure that I understand properly. If you call the function many times, 
does the memory usage increase?

I'm not sure that this issue is a bug in Python.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33607] Explicitly track object ownership (and allocator).

[STINNER Victor]

STINNER Victor  added the comment:

"Either we'd add 2 pointer-size fields to PyObject or we would keep a separate 
hash table (or two) pointing from each object to the info (...)"

The expect a huge impact on the memory footprint. I dislike the idea.

Currently, the smallest Python object is:

>>> sys.getsizeof(object())
16

It's just two pointers. Adding two additional pointers would simply double the 
size of the object.

"Now the C-API offers a way to switch the allocator, so there's no guarantee
that the right allocator is used in PyMem_Free()."

I would expect that either all interpreters use the same memory allocator, or 
that each interpreter uses its own allocator. If you use one allocator per 
interpreter, calling PyMem_Free() from the wrong interpreter would just crash. 
As you get a crash when you call free() on an object allocated by PyMem_Free(). 
You can extend PYTHONMALLOC=debug to detect bugs. This builtin debugger is 
already able to catch misuses of allocators. Adding extra pointers to this 
debugger is acceptable since it doesn't modify the footprint of the default 
mode.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33605] Detect accessing event loop from a different thread outside of _debug

[Yury Selivanov]

Yury Selivanov  added the comment:

> I suggest that asyncio should be stricter about this error and that methods 
> and functions that operate on the event loop, such as call_soon, call_later, 
> create_task, ensure_future, and close, should all call _check_thread() even 
> when not in debug mode. _check_thread() warns that it "should only be called 
> when self._debug == True", hinting at "performance reasons", but that doesn't 
> seem justified. threading.get_ident() is efficiently implemented in C, and 
> comparing that integer to another cached integer is about as efficient an 
> operation as it gets.

I'd be OK with this if the performance penalty is within 0.5% in 
microbenchmarks for asyncio & uvloop.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33565] strange tracemalloc results

[Alexander Mohr]

Alexander Mohr  added the comment:

yes, memory does go up.  If you click the botocore bug link you'll see a graph 
of memory usage over time.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33604] HMAC default to MD5 marked as to be removed in 3.6

[Matthias Bussonnier]

Change by Matthias Bussonnier :


--
pull_requests: +6693

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32400] inspect.isdatadescriptor false negative

[Aaron Hall]

Change by Aaron Hall :


--
nosy: +Aaron Hall

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com