[issue38019] AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'
New submission from Marat Sharafutdinov : I'm trying to use dwdiff (https://os.ghalkes.nl/dwdiff.html) to compare two text files. While running subprocess I'm getting the following errors: ``` Traceback (most recent call last): File "test_dwdiff.py", line 17, in asyncio.run(main()) File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/runners.py", line 43, in run return loop.run_until_complete(main) File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete return future.result() File "test_dwdiff.py", line 14, in main await asyncio.gather(*(dwdiff() for __ in range(25))) File "test_dwdiff.py", line 10, in dwdiff await process.communicate() File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 187, in communicate loop=self._loop) File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 166, in _read_stream output = await stream.read() File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 633, in read block = await self.read(self._limit) File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 646, in read self._maybe_resume_transport() File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 417, in _maybe_resume_transport self._transport.resume_reading() File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 498, in resume_reading self._loop._add_reader(self._fileno, self._read_ready) AttributeError: 'NoneType' object has no attribute '_add_reader' ``` and ``` Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n') handle: Traceback (most recent call last): File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 71, in pipe_data_received reader.feed_data(data) File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 440, in feed_data self._transport.pause_reading() File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 495, in pause_reading self._loop._remove_reader(self._fileno) AttributeError: 'NoneType' object has no attribute '_remove_reader' ``` The code is the following: ``` import asyncio async def dwdiff(): process = await asyncio.create_subprocess_exec( '/usr/local/bin/dwdiff', 'f1.txt', 'f2.txt', stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) await process.communicate() async def main(): await asyncio.gather(*(dwdiff() for __ in range(25))) asyncio.run(main()) ``` Two text files f1.txt and f2.txt are attached as archive. Perhaps it relates to https://bugs.python.org/issue29704 -- components: asyncio files: f1_and_f2.zip messages: 351090 nosy: asvetlov, decaz, yselivanov priority: normal severity: normal status: open title: AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' type: behavior versions: Python 3.7 Added file: https://bugs.python.org/file48589/f1_and_f2.zip ___ Python tracker <https://bugs.python.org/issue38019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38019] asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'
Change by Marat Sharafutdinov : -- title: AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' -> asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader' versions: +Python 3.6, Python 3.8, Python 3.9 ___ Python tracker <https://bugs.python.org/issue38019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38019] asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'
Marat Sharafutdinov added the comment: Andrew, it would be nice to have it fixed within 3.8. Hope you'll find time for this. -- ___ Python tracker <https://bugs.python.org/issue38019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue38019] asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'
Marat Sharafutdinov added the comment: Andrew, thank you! -- ___ Python tracker <https://bugs.python.org/issue38019> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue35974] os.DirEntry.inode() returns invalid value within Docker container
New submission from Marat Sharafutdinov :
I'm trying to build Python 3.7.2 within official CentOS 7.6.1810 image
(https://hub.docker.com/_/centos) and getting the following error during
testing:
==
FAIL: test_attributes (test.test_os.TestScandir)
--
Traceback (most recent call last):
File "/usr/src/python/Lib/test/test_os.py", line 3367, in test_attributes
self.check_entry(entry, 'dir', True, False, False)
File "/usr/src/python/Lib/test/test_os.py", line 3319, in check_entry
os.stat(entry.path, follow_symlinks=False).st_ino)
AssertionError: 28093768 != 85098458
I guess this bug applies to Docker containers in general. For instance it's
reproduced with the official Python 3.7.2-stretch image based on the Debian
Stretch (https://hub.docker.com/_/python):
$ docker run --rm -it python:3.7.2-stretch
Python 3.7.2 (default, Feb 6 2019, 12:04:03)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.mkdir('/test_dir')
>>> for entry in os.scandir('/'):
... if entry.name == 'test_dir':
... break
...
>>> print(entry, entry.inode(), os.stat(entry.path,
>>> follow_symlinks=False).st_ino)
23898155 85118011
>>> assert entry.inode() == os.stat(entry.path, follow_symlinks=False).st_ino
Traceback (most recent call last):
File "", line 1, in
AssertionError
>>>
In case of using host volume when running container it works ok, - the problem
occurs when using default Docker volume:
$ docker run --rm -it -v /home/decaz/workspace:/host_dir python:3.7.2-stretch
Python 3.7.2 (default, Feb 6 2019, 12:04:03)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.mkdir('/host_dir/test_dir')
>>> for entry in os.scandir('/host_dir'):
... if entry.name == 'test_dir':
... break
...
>>> print(entry, entry.inode(), os.stat(entry.path,
>>> follow_symlinks=False).st_ino)
12873222 12873222
>>> assert entry.inode() == os.stat(entry.path, follow_symlinks=False).st_ino
>>>
Similar issue - https://bugs.python.org/issue32811.
--
components: Build
messages: 335275
nosy: decaz
priority: normal
severity: normal
status: open
title: os.DirEntry.inode() returns invalid value within Docker container
type: behavior
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue35974>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36629] imaplib test fails with errno 101
New submission from Marat Sharafutdinov : == FAIL: test_imap4_host_default_value (test.test_imaplib.TestImaplib) -- Traceback (most recent call last): File "/home/python/Lib/test/test_imaplib.py", line 94, in test_imap4_host_default_value self.assertIn(cm.exception.errno, expected_errnos) AssertionError: 101 not found in [111, 99] -- I guess `errno.ENETUNREACH` should be added to the `expected_errnos` as it done within `test_create_connection` (test.test_socket.NetworkConnectionNoServer). -- components: Tests messages: 340212 nosy: decaz priority: normal severity: normal status: open title: imaplib test fails with errno 101 type: behavior versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue36629> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36629] imaplib test fails with errno 101
Marat Sharafutdinov added the comment: I see this error on my local Bamboo CI (runs on CentOS 7). -- ___ Python tracker <https://bugs.python.org/issue36629> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36629] imaplib test fails with errno 101
Marat Sharafutdinov added the comment: Thank you for the patch, Victor! -- ___ Python tracker <https://bugs.python.org/issue36629> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33115] Asyncio loop blocks with a lot of parallel tasks
New submission from Marat Sharafutdinov :
I want to schedule a lot of parallel tasks, but it becomes slow with loop
blocking:
```python
import asyncio
task_count = 1
async def main():
for x in range(1, task_count + 1):
asyncio.ensure_future(f(x))
async def f(x):
if x % 1000 == 0 or x == task_count:
print(f'Run f({x})')
await asyncio.sleep(1)
loop.call_later(1, lambda: asyncio.ensure_future(f(x)))
loop = asyncio.get_event_loop()
loop.set_debug(True)
loop.run_until_complete(main())
loop.run_forever()
```
Outputs:
```
Executing
result=None created at /usr/lib/python3.6/asyncio/base_events.py:446> took
0.939 seconds
...
Executing , None) at
/usr/lib/python3.6/asyncio/futures.py:339 created at
/usr/lib/python3.6/asyncio/tasks.py:480> took 0.113 seconds
...
Executing wait_for=()] created at
/usr/lib/python3.6/asyncio/base_events.py:275> created at test_aio.py:13> took
0.100 seconds
...
```
What can be another way to schedule a lot of parallel tasks?
--
components: asyncio
messages: 314207
nosy: asvetlov, decaz, yselivanov
priority: normal
severity: normal
status: open
title: Asyncio loop blocks with a lot of parallel tasks
type: resource usage
versions: Python 3.6
___
Python tracker
<https://bugs.python.org/issue33115>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33115] Asyncio loop blocks with a lot of parallel tasks
Marat Sharafutdinov added the comment:
But why if I use multiprocessing (run 100 tasks by 100 workers) it still
continue blocking loop within some workers? Are 100 tasks "a lot of work" for
asyncio loop?
```python
import asyncio
from multiprocessing import Process
worker_count = 100
task_count = 100
def worker_main(worker_id):
async def main():
for x in range(1, task_count + 1):
asyncio.ensure_future(f(x))
async def f(x):
if x % 1000 == 0 or x == task_count:
print(f'[WORKER-{worker_id}] Run f({x})')
await asyncio.sleep(1)
loop.call_later(1, lambda: asyncio.ensure_future(f(x)))
loop = asyncio.get_event_loop()
loop.set_debug(True)
loop.run_until_complete(main())
loop.run_forever()
if __name__ == '__main__':
for worker_id in range(worker_count):
worker = Process(target=worker_main, args=(worker_id,), daemon=True)
worker.start()
while True:
pass
```
--
___
Python tracker
<https://bugs.python.org/issue33115>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33115] Asyncio loop blocks with a lot of parallel tasks
Marat Sharafutdinov added the comment: Does this mean that GC uses most part of CPU time so the loop blocks? And another question: do you have any plans to optimize the loop so it would be possible to run really lot of tasks in parallel? Thanks. -- ___ Python tracker <https://bugs.python.org/issue33115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33115] Asyncio loop blocks with a lot of parallel tasks
Marat Sharafutdinov added the comment: Concerning the example adding a jitter is useful, thanks! But anyway in case there will be something not constant as sleeping for 1 sec is, the problem will continue to appear. -- ___ Python tracker <https://bugs.python.org/issue33115> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31943] Add asyncio.Handle.cancelled() and asyncio.TimerHandle.when
New submission from Marat Sharafutdinov : It would be handy to access the loop's time the Handle's callback will be called at if you are using delayed calls with "delay" (not "when") argument. Also it is useful to know whether the call was cancelled through the special attribute. -- assignee: docs@python components: Documentation, Library (Lib), asyncio messages: 305561 nosy: decaz, docs@python, yselivanov priority: normal pull_requests: 4238 severity: normal status: open title: Add asyncio.Handle.cancelled() and asyncio.TimerHandle.when type: enhancement versions: Python 3.6 ___ Python tracker <https://bugs.python.org/issue31943> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31943] Add asyncio.Handle.cancelled() method
Change by Marat Sharafutdinov : -- title: Add asyncio.Handle.cancelled() and asyncio.TimerHandle.when -> Add asyncio.Handle.cancelled() method ___ Python tracker <https://bugs.python.org/issue31943> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31943] Add asyncio.Handle.cancelled() method
Change by Marat Sharafutdinov : -- versions: +Python 3.7 -Python 3.6 ___ Python tracker <https://bugs.python.org/issue31943> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26795] Fix PEP 344 Python version
New submission from Marat Sharafutdinov: Should be 3.0 instead of 2.5 -- assignee: docs@python components: Documentation files: pep-0344-version.patch keywords: patch messages: 263667 nosy: decaz, docs@python priority: normal severity: normal status: open title: Fix PEP 344 Python version type: enhancement Added file: http://bugs.python.org/file42505/pep-0344-version.patch ___ Python tracker <http://bugs.python.org/issue26795> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
