[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: New changeset 07627e9a6a5f418354ff3dc99a0f36bc5b79dcd8 by Yury Selivanov in branch 'master': bpo-32251: Fix docs (#5408) https://github.com/python/cpython/commit/07627e9a6a5f418354ff3dc99a0f36bc5b79dcd8 -- ___ Pyt

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- pull_requests: +5241 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed type: -> enhancement ___ Python tracker ___ _

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: New changeset 631fd38dbf04dbf0127881f3977982e401a849e4 by Yury Selivanov in branch 'master': bpo-32251: Implement asyncio.BufferedProtocol. (#4755) https://github.com/python/cpython/commit/631fd38dbf04dbf0127881f3977982e401a849e4 --

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: >> Looks nice. Can it speed up aiohttp too? > Yes. > aiohttp uses own streams but public API and internal implementation are > pretty close to asyncio streams. > Moreover C accelerated HTTP parser should work with proposed BufferedProtocol > seamlessly. I d

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: > See https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage for an > interesting discussion of the drawbacks of some buffer handling idioms. Thanks for the link! It does make sense to use a pool of buffers for the proposed BufferedProtocol when you

[issue32251] Add asyncio.BufferedProtocol

Antoine Pitrou added the comment: See https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage for an interesting discussion of the drawbacks of some buffer handling idioms. -- ___ Python tracker

[issue32251] Add asyncio.BufferedProtocol

Andrew Svetlov added the comment: Yes. aiohttp uses own streams but public API and internal implementation are pretty close to asyncio streams. Moreover C accelerated HTTP parser should work with proposed BufferedProtocol seamlessly. -- ___ Python

[issue32251] Add asyncio.BufferedProtocol

INADA Naoki added the comment: Looks nice. Can it speed up aiohttp too? -- nosy: +inada.naoki ___ Python tracker ___ ___ Python-bug

[issue32251] Add asyncio.BufferedProtocol

Łukasz Langa added the comment: +1 on the idea, I would use this. -- nosy: +lukasz.langa ___ Python tracker ___ ___ Python-bugs-list

[issue32251] Add asyncio.BufferedProtocol

Change by Guido van Rossum : -- nosy: -gvanrossum ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mai

[issue32251] Add asyncio.BufferedProtocol

Andrew Svetlov added the comment: New protocol may speed up not only `reader.readexactly()`. Every `reader.feed_data()` appends a new chunk to existing buffer. We can try to read into unused tail of the buffer instead. -- ___ Python tracker

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: > Do you think it's possible to fold BufferedProtocol into Protocol? It would be a backwards incompatible change :( Coincidentally, there might be protocols that already implement 'get_buffer()' and 'buffer_updated()' methods that do something completely diff

[issue32251] Add asyncio.BufferedProtocol

Antoine Pitrou added the comment: Do you think it's possible to fold BufferedProtocol into Protocol? i.e., either `get_buffer()` returns None (the default) and `data_received()` is called with a bytes object, or `get_buffer()` returns a writable buffer and `buffer_updated()` is called with the

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: > 1. What happens if size of read data is greater than pre-allocated buffer? Let's say we have 2Kb of data in the socket's network buffer, and we only preallocated 0.5Kb in our buffer. We will the receive 0.5Kb in our buffer, 'Protocol.buffer_updated()' will

[issue32251] Add asyncio.BufferedProtocol

Antoine Pitrou added the comment: I have another question: what happens if there is a partial read? For example, let's says I return a 1024-bytes buffer in get_buffer(), but recv_into() receives data in 512 chunks. Is it: 1. getbuffer() is called, returns 1024 bytes buffer 2. recv_into() rec

[issue32251] Add asyncio.BufferedProtocol

Andrew Svetlov added the comment: Numbers are great! Couple questions. 1. What happens if size of read data is greater than pre-allocated buffer? 2. Is flow control logic changed or not? If I understand correctly pause_reading() / resume_reading() continue to work as earlier. -- ___

[issue32251] Add asyncio.BufferedProtocol

Yury Selivanov added the comment: I've made a PR that implements the change for selector_events.py. With the change: vanilla asyncio:120-135 Mb/s vanilla asyncio/get_buffer: 220-230 Mb/s uvloop: 320-330 Mb/s uvloop/get_buffer: 600-650 Mb/s If we decide

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- keywords: +patch pull_requests: +4658 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-li

[issue32251] Add asyncio.BufferedProtocol

New submission from Yury Selivanov : A couple emails from async-sig for the context: 1. https://mail.python.org/pipermail/async-sig/2017-October/000392.html 2. https://mail.python.org/pipermail/async-sig/2017-December/000423.html I propose to add another Protocol base class to asyncio: Buffered

[issue32251] Add asyncio.BufferedProtocol

Change by Yury Selivanov : -- assignee: yselivanov components: asyncio nosy: asvetlov, gvanrossum, pitrou, yselivanov priority: normal severity: normal status: open title: Add asyncio.BufferedProtocol versions: Python 3.7 ___ Python tracker