[issue6028] Interpreter aborts when chaining an infinite number of exceptions

[Yury]

Yury  added the comment:

Rather than aborting with a stack overflow, I feel it is more natural to raise 
an exception. If it is not too difficult to implement, perhaps another type of 
exception should be raised. Since chained exceptions are new to 3.x, there 
should be a new exception to describe errors that happen in chaining. Perhaps 
stopping chaining at a certain depth and truncating all further exceptions with 
a blanket "ChainingException" exception. Or perhaps truncating the chain and 
wrapping it in another exception before returning it to user code.

While this is my proposed solution, I apologize I cannot volunteer to write the 
patch. The time investment on my part would involve learning most of the 
working of the interpreter. Perhaps someone who is already familiar with it 
would be interested.

--

___
Python tracker 
<http://bugs.python.org/issue6028>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28874] test_logging fails and freezes

[Yury]

Yury  added the comment:

I am observing this issue while building 3.6.8 version on CentOS 7.6. Shall we 
reopen the bug? Thanks, Yury.

--
nosy: +scrutari

___
Python tracker 
<https://bugs.python.org/issue28874>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28874] test_logging fails and freezes

[Yury]

Yury  added the comment:

And same happens to me when building version 3.7.3.
I'm building with gcc (GCC) 4.8.5.

--

___
Python tracker 
<https://bugs.python.org/issue28874>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16184] Attack against the pseudorandom number generator

[Yury]

New submission from Yury:

---[ Vulnerability description]

Positive Research experts have detected a vulnerability "Predictable 
pseudorandom numbers in Python" caused by random.random() output in Python.

By default, Mersenne Twister initialization is done during random module import 
and then after 312 random.random() calls that allows attackers to predict 313 
random number with 2^8 accuracy based on 1, 2, 199, 200, 511, 625 random 
numbers.

--[Solution]
 
We recommend you to initialize random.seed() after every 312 calls that 
guarantees enough entropy of Mersenne Twister element statuses.

---[Credits]

Vulnerability was detected by Timur Yunusov, Positive Research Center (Positive 
Technologies Company)

---

1. In spite of the fact that MT states are initialized correctly, there is a 
practical method to exploit the vulnerability because of Python web framework 
peculiarities.
2. The possibility that a random number is among described 256 variants is 
12,5% because there are two methods to round off numbers to the certain 
accuracy. This provides 8 possible variants of existing state_1[2], 
state_1[397], state_2[397].
You can find the code necessary to demonstrate the vulnerability (vuln.py) and 
exploit it (brute.py) attached.

--
components: None
files: POC.zip
messages: 172566
nosy: ymaryshev
priority: normal
severity: normal
status: open
title: Attack against the pseudorandom number generator
type: security
versions: Python 3.3
Added file: http://bugs.python.org/file27514/POC.zip

___
Python tracker 
<http://bugs.python.org/issue16184>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16184] Attack against the pseudorandom number generator

[Yury]

Yury added the comment:

Thank you, you are right. Please close the issue.

--

___
Python tracker 
<http://bugs.python.org/issue16184>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue7567] Messed up terminal after calling curses.initscr() twice.

[Yury]

New submission from Yury :

Calling curses.initscr after curses.endwin() results in a messed up
terminal the second time curses.endwind() is called. For example:

import curses
curses.initscr(); curses.endwin(); curses.initscr(); curses.endwin()

This behaves correctly in python 2.5. It does not work in python 2.6.

--
components: Library (Lib)
messages: 96826
nosy: yury
severity: normal
status: open
title: Messed up terminal after calling curses.initscr() twice.
versions: Python 2.6

___
Python tracker 
<http://bugs.python.org/issue7567>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6028] Interpreter crashes when chaining an infinite number of exceptions

[Yury]

New submission from Yury :

def error_handle():
try:
print(5/0)
except:
error_handle()

error_handle()

Fatal Python error: Cannot recover from stack overflow.
Aborted

The interpreter should not crash. Perhaps a RuntimeError should be
thrown instead.

--
components: Interpreter Core
messages: 87797
nosy: yury
severity: normal
status: open
title: Interpreter crashes when chaining an infinite number of exceptions
type: crash
versions: Python 3.0

___
Python tracker 
<http://bugs.python.org/issue6028>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6028] Interpreter crashes when chaining an infinite number of exceptions

[Yury]

Yury  added the comment:

The code you posted causes an infinite loop in the 2.x branch as well.
Anyway, I do not see how crashing is a desired result. An infinite loop
means the programmer made a mistake somewhere. A crash means the
interpreter did.

--

___
Python tracker 
<http://bugs.python.org/issue6028>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6028] Interpreter crashes when chaining an infinite number of exceptions

[Yury]

Yury  added the comment:

I knew that python handles infinite recursion and gracefully errors out,
and I knew that exception chaining was new to 3.0, so I wanted to see if
they would work together. Apparently, they do not. Yet, the code works
fine in the 2.x branch. So, the 3.0 branch introduces a bug. I am not
sure how much clearer I can make this.

The code is in no way trying to circumvent anything. If there is about
to be a stack overflow, an exception show be raised. It should then
float up the stack. This is the desired result. In fact, this is the
only sane result. In an operating system, a program should not be able
to crash the entire system. In X, a client should not be able to crash
the server. Same principle applies here.

I am sorry I simply do not see your point. This seems so obvious. There
is a correct way of handling this error. You can safely recover from it.
If you insist on a general case, I have outlined it above.

I am not terribly thrilled about your attitude of "we might as well
leave it broken." Why bother working on the project at all?

--

___
Python tracker 
<http://bugs.python.org/issue6028>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

New submission from Yury Selivanov :

I'm investigating a memory leak in 3.10, and while looking at the offending 
commit I stumbled upon this: in ceval.c:PyEval_EvalCodeEx, there's this 
allocation:

PyObject **kwargs = PyMem_Malloc(sizeof(PyObject *)*kwcount);

The problem is that this isn't ever freed. And `kwargs` isn't used anywhere in 
the function body. It seems to me that this is silently leaking memory.

--
messages: 410329
nosy: Elvis.Pranskevichus, Mark.Shannon, pablogsal, vstinner, yselivanov
priority: normal
severity: normal
status: open
title: memory leak in PyEval_EvalCodeEx

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:

(this is the context of our current investigation btw: 
https://github.com/MagicStack/asyncpg/issues/874#issuecomment-1009383262)

--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:

This is the questionable place in the code: 
https://github.com/python/cpython/blob/dce642f24418c58e67fa31a686575c980c31dd37/Python/ceval.c#L6131-L6166

See that `kwargs` is allocated but never freed or used.

--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Change by Yury Selivanov :


--
keywords: +patch
pull_requests: +28746
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/30546

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 607d8a838f29ad3c4c4e85b39f338dade5f9cafe by Yury Selivanov in 
branch 'main':
bpo-46347: Fix memory leak in PyEval_EvalCodeEx. (#30546)
https://github.com/python/cpython/commit/607d8a838f29ad3c4c4e85b39f338dade5f9cafe


--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +28752
pull_request: https://github.com/python/cpython/pull/30551

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:

> The function will still leak "kwnames" and "default" if creating the "func" 
> object fails.

Yeah, here's another PR to address that: 
https://github.com/python/cpython/pull/30551

--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 20b5791ce9b47195ce51cfd5acb223b1ca59cdf0 by Yury Selivanov in 
branch 'main':
bpo-46347: Fix PyEval_EvalCodeEx to correctly cleanup in error paths (#30551)
https://github.com/python/cpython/commit/20b5791ce9b47195ce51cfd5acb223b1ca59cdf0


--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Change by Yury Selivanov :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +28754
pull_request: https://github.com/python/cpython/pull/30553

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +28755
pull_request: https://github.com/python/cpython/pull/30554

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 6f9ca53a6ac343a5663cc5c52546acf9a63b605a by Yury Selivanov in 
branch '3.10':
bpo-46347: Fix PyEval_EvalCodeEx to correctly cleanup in error paths (#30553)
https://github.com/python/cpython/commit/6f9ca53a6ac343a5663cc5c52546acf9a63b605a


--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46347] memory leak in PyEval_EvalCodeEx

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset be578e0c063dad1dbb273f86d5bc77e4e6f14583 by Yury Selivanov in 
branch 'main':
bpo-46347: Yet another fix in the erorr path of PyEval_EvalCodeEx (#30554)
https://github.com/python/cpython/commit/be578e0c063dad1dbb273f86d5bc77e4e6f14583


--

___
Python tracker 
<https://bugs.python.org/issue46347>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46752] Introduce task groups to asyncio and change task cancellation semantics

[Yury Selivanov]

Yury Selivanov  added the comment:

> There's one thing that gather() does that TaskGroup doesn't: it gives us the 
> return values from the tasks.

That's easy to do with task groups too:

  async with TaskGroup() as g:
  r1 = g.create_task(coro1())
  r2 = g.create_task(coro2())

  print(r1.result())

  # or
  print(await r2)  # I *think* this should work

--

___
Python tracker 
<https://bugs.python.org/issue46752>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46771] Add some form of cancel scopes

[Yury Selivanov]

Yury Selivanov  added the comment:

Couple thoughts:

I'm +1 for adding TaskGroup.cancel() method.

I'd be -1 on abusing `Task.cancel()` to signal something with a nonce. Whatever 
problem we are trying to solve here, it should be solvable without resorting to 
hacks like this. It should be trivial to implement simple tracking of whether a 
child task was cancelled by the group or not to decide on how to handle a rogue 
CancelledError.

--

___
Python tracker 
<https://bugs.python.org/issue46771>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46771] Add some form of cancel scopes

[Yury Selivanov]

Yury Selivanov  added the comment:

> * `async with` vs `with`: if Andrew thinks `async with` is easier to teach 
> and less error-prone,

It has to be 'async with' like most asyncio apis to avoid user confusion.

--

___
Python tracker 
<https://bugs.python.org/issue46771>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46844] Context-based TaskGroup for legacy libraries

[Yury Selivanov]

Yury Selivanov  added the comment:

> I believe that this approach would allow more control over tasks implicitly 
> spawned by 3rd-party libraries that cannot control.

Please elaborate. I'm not sure what are the benefits of this.

--

___
Python tracker 
<https://bugs.python.org/issue46844>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13062] Introspection generator and function closure state

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +Yury.Selivanov

___
Python tracker 
<http://bugs.python.org/issue13062>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13241] llvm-gcc-4.2 miscompiles Python (XCode 4.1 on Mac OS 10.7)

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +Yury.Selivanov

___
Python tracker 
<http://bugs.python.org/issue13241>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11682] PEP 380 reference implementation for 3.3

[Yury Selivanov]

Yury Selivanov  added the comment:

It looks like there is a memory leak bug (on StopIteration exception instances).

Attached is the test to expose it.

It seems that adding 'Py_DECREF(e);' after 
'PyErr_SetObject(PyExc_StopIteration, e);' in 'genobject.c' fixes the leak.

--
Added file: http://bugs.python.org/file23639/test.py

___
Python tracker 
<http://bugs.python.org/issue11682>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10882] Add os.sendfile()

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +Yury.Selivanov

___
Python tracker 
<http://bugs.python.org/issue10882>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9666] 'hasattr' fix to suppress only AttributeError

[Yury Selivanov]

New submission from Yury Selivanov :

As discussed on python-dev mailing list 
(http://mail.python.org/pipermail/python-dev/2010-August/103178.html), 
'hasattr' default behaviour should be changed to suppress only AttributeError 
exceptions.  Other should pass through.

The fix, however, shouldn't change behaviour of existing C API, functions 
PyObject_HasAttr and PyObject_HasAttrString in particular.

I'm targeting this issue on Python 3.2 version, but probably it may be 
introduced in the next Python 3.1 maintenance release.

--
assignee: d...@python
components: Documentation, Library (Lib)
files: hasattr.patch
keywords: patch
messages: 114767
nosy: Yury.Selivanov, d...@python
priority: normal
severity: normal
status: open
title: 'hasattr' fix to suppress only AttributeError
type: behavior
versions: Python 3.2
Added file: http://bugs.python.org/file18622/hasattr.patch

___
Python tracker 
<http://bugs.python.org/issue9666>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9695] Return from generators in Python 3.2

[Yury Selivanov]

New submission from Yury Selivanov :

The patch is intended to fix behaviour of 'return' statement in python's 
generators.

Please read this message before looking at the patch.
http://mail.python.org/pipermail/python-dev/2010-August/103297.html

--
components: Interpreter Core
files: generators_return.patch
keywords: patch
messages: 115033
nosy: Yury.Selivanov
priority: normal
severity: normal
status: open
title: Return from generators in Python 3.2
type: feature request
versions: Python 3.2
Added file: http://bugs.python.org/file18655/generators_return.patch

___
Python tracker 
<http://bugs.python.org/issue9695>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11682] PEP 380 reference implementation for 3.3

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +Yury.Selivanov

___
Python tracker 
<http://bugs.python.org/issue11682>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue10576] Add a progress callback to gcmodule

[Yury Selivanov]

Changes by Yury Selivanov :


--
nosy: +Yury.Selivanov

___
Python tracker 
<http://bugs.python.org/issue10576>
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46829] Confusing CancelError message if multiple cancellations are scheduled

[Yury Selivanov]

Yury Selivanov  added the comment:

Andrew asked me for my opinion on the matter --  I think we should get rid of 
the message. Exception messages for "signals" can be easily lost if an 
exception was re-raised. If the reason of why something is being cancelled is 
important (in my experience it never is) it should be recorded elsewhere.

--

___
Python tracker 
<https://bugs.python.org/issue46829>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46829] Confusing CancelError message if multiple cancellations are scheduled

[Yury Selivanov]

Yury Selivanov  added the comment:

IOW I think that supporting custom messages is a needless complication of our 
API. Given how complex task trees can become with TaskGroups collecting those 
messages and presenting them all to the user isn't trivial, showing just 
first/last defeats the purpose. So i'm in favor of dropping it.

--

___
Python tracker 
<https://bugs.python.org/issue46829>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue46994] Accept explicit contextvars.Context in asyncio create_task() API

[Yury Selivanov]

Yury Selivanov  added the comment:

Yeah, +1 to add a parameter. Fwiw it was on my idea list when i was working on 
the pep

--

___
Python tracker 
<https://bugs.python.org/issue46994>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45997] asyncio.Semaphore waiters deque doesn't work

[Yury Selivanov]

Yury Selivanov  added the comment:

Andrew, the same problem exists in asyncio.Queue which is also critical. Here's 
how I fixed it in edgedb code base: 
https://github.com/edgedb/edgedb/blob/08e41341024828df22a01cd690b11fcff00bca5e/edb/server/compiler_pool/queue.py#L51-L74

--

___
Python tracker 
<https://bugs.python.org/issue45997>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue47167] Allow overriding future-task compliance check in asyncio

[Yury Selivanov]

Yury Selivanov  added the comment:

Sorry, I don't like the committed change and I think it should be reverted.

Futures and Tasks have a documented `get_loop()` method which simply should be 
called by Task machinery, without the need to expose (and document) private 
methods.

--
resolution: fixed -> 
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue47167>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45099] asyncio.Task's documentation says that loop arg is removed when it's not

[Yury Selivanov]

Yury Selivanov  added the comment:

> I have a feeling that we need a low-level API section that describes 
> *supported* API that is intended to task extenders only (guys who want to 
> provide an alternative task factory for example).

I'm not sure how exposing all private methods spiraled out of this issue. We 
haven't discussed documenting the low-level methods you just exposed Andrew. 
IMO this should be reverted.

--

___
Python tracker 
<https://bugs.python.org/issue45099>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34679] asyncio.add_signal_handler call fails if not on main thread

[Yury Selivanov]

Yury Selivanov  added the comment:

Andrew, can you fix ctrl-c/windows issue? I'm basically afk until monday. 

And we're not going to do anything about add_signal_handler in 3.8.

--

___
Python tracker 
<https://bugs.python.org/issue34679>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34037] asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads

[Yury Selivanov]

Yury Selivanov  added the comment:

Here's the API I propose to solve this problem: 
https://github.com/python/cpython/pull/15735#pullrequestreview-285389412

Summary:

* Add a new loop.shutdown_threadpool() method. Just like with 
shutdown_asyncgens() -- it would be invalid to call loop.run_in_executer() 
after loop.shutdown_threadpool() is called.

* Make asyncio.run() to call the new loop.shutdown_threadpool().

--

___
Python tracker 
<https://bugs.python.org/issue34037>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34037] asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads

[Yury Selivanov]

Yury Selivanov  added the comment:

> What do you think of the "shutdown_default_executor()" name?

Yeah, I think it's a better name!

--

___
Python tracker 
<https://bugs.python.org/issue34037>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Yury Selivanov]

Yury Selivanov  added the comment:

🙏

--

___
Python tracker 
<https://bugs.python.org/issue33523>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38066] Hide internal asyncio.Stream methods

[Yury Selivanov]

Yury Selivanov  added the comment:

BTW, if we are doing this -- maybe not expose the underlying transport at all?  
I don't like that we do that currently, and there multiple ways for the user to 
screw up the Stream object (and potentially break start_tls()/sendfile() etc)

--

___
Python tracker 
<https://bugs.python.org/issue38066>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

[Yury Selivanov]

Yury Selivanov  added the comment:

We want to add TaskGroups to asyncio (a similar concept to Trio's nurseries).  
TaskGroups use the `async with` statement to clearly define where Tasks are 
created and at which point they are expected to be completed or destroyed.

asyncio.gather(), asyncio.as_completed(), and few others will be considered 
legacy APIs after we implement TaskGroups.  Implementing rate limiting on top 
of TaskGroups is easier and more reliable.

I'd really prefer to keep as_completed() and especially gather() as is, as I 
consider them a bit broken already.

--

___
Python tracker 
<https://bugs.python.org/issue30782>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

[Yury Selivanov]

Yury Selivanov  added the comment:

FWIW I've been using TaskGroups in EdgeDB codebase extensively: 
https://github.com/edgedb/edgedb/blob/master/edb/common/taskgroup.py (you can 
use the code, it's Apache 2)

The only thing that prevented us from merging them in 3.8 is that we need to 
formally define & implement ExceptionGroup (or MultiError) in CPython.  I'm 
going to work on an initial PEP for that this week.

--

___
Python tracker 
<https://bugs.python.org/issue30782>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30782] Allow limiting the number of concurrent tasks in asyncio.as_completed

[Yury Selivanov]

Yury Selivanov  added the comment:

> it will help simplify some Mock things.

Yeah, we'll need to chat about that so I can use Mock requirements in the PEP. 
:)

--

___
Python tracker 
<https://bugs.python.org/issue30782>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36373] Deprecate explicit loop parameter in all public asyncio APIs

[Yury Selivanov]

Yury Selivanov  added the comment:

(copying my GitHub message on this topic from 
https://github.com/python/cpython/pull/15889#issuecomment-530030002)


I thought we were going to be more subtle about this.

We have two scenarios:

Somebody creates an instance of asyncio.Lock outside of a coroutine, also 
manually creating the loop, etc. In this case the user has to pass the lock 
argument.

Somebody creates an instance of asyncio.Lock in a coroutine (that is, wheh 
asyncio.get_running_loop() returns a loop). In this case passing the loop 
argument is an error.

The (1) approach is how people were used to writing asyncio programs before 
asyncio.run() (that was the only way actually).

The (2) approach is how we want people to write asyncio programs.

There's a subtle difference between things like asyncio.gather() and 
asyncio.Lock. Passing loop to the former is just nonsensical. Passing loop to 
the latter can be a valid thing, it's the (1).

If we remove the loop parameter entirely from classes like asyncio.Lock we're 
making (1) impossible. I'm not sure that is what we are ready to do now.

@asvetlov thoughts?

--

___
Python tracker 
<https://bugs.python.org/issue36373>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36373] Deprecate explicit loop parameter in all public asyncio APIs

[Yury Selivanov]

Change by Yury Selivanov :


--
Removed message: https://bugs.python.org/msg351728

___
Python tracker 
<https://bugs.python.org/issue36373>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36373] Deprecate explicit loop parameter in all public asyncio APIs

[Yury Selivanov]

Yury Selivanov  added the comment:

(copying my GitHub message on this topic from 
https://github.com/python/cpython/pull/15889#issuecomment-530030002)


I thought we were going to be more subtle about this.

We have two scenarios:

1. Somebody creates an instance of asyncio.Lock outside of a coroutine, also 
manually creating the loop, etc. In this case the user has to pass the lock 
argument.

2. Somebody creates an instance of asyncio.Lock in a coroutine (that is, wheh 
asyncio.get_running_loop() returns a loop). In this case passing the loop 
argument is an error.

The (1) approach is how people were used to writing asyncio programs before 
asyncio.run() (that was the only way actually).

The (2) approach is how we want people to write asyncio programs.

There's a subtle difference between things like asyncio.gather() and 
asyncio.Lock. Passing loop to the former is just nonsensical. Passing loop to 
the latter can be a valid thing, it's the (1).

If we remove the loop parameter entirely from classes like asyncio.Lock we're 
making (1) impossible. I'm not sure that is what we are ready to do now.

@asvetlov thoughts?

--

___
Python tracker 
<https://bugs.python.org/issue36373>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38164] poloshing asyncio Streams API

[Yury Selivanov]

New submission from Yury Selivanov :

Summary of discussion at https://github.com/python-trio/trio/issues/1208:

* `asyncio.Stream.write()` will start throwing a `DeprecationWarning` asking 
people to add an `await` if they didn't;

* `asyncio.Stream.close()` will start throwing a `DeprecationWarning` asking 
people to add an `await` if they didn't;

* `asyncio.Stream.drain()` & `asyncio.Stream.wait_closed()` will start throwing 
a `DeprecationWarning` telling about a scheduled removal (in Python 3.12) when 
used on `Process.std*` streams;

* `asyncio.Stream.drain()` & `asyncio.Stream.wait_closed()` will not work at 
all on Streams created via new 3.8 APIs: `connect()` & `StreamServer`.

--
assignee: asvetlov
components: asyncio
messages: 352372
nosy: asvetlov, lukasz.langa, yselivanov
priority: release blocker
severity: normal
stage: needs patch
status: open
title: poloshing asyncio Streams API
type: behavior
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue38164>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38164] polishing asyncio Streams API

[Yury Selivanov]

Change by Yury Selivanov :


--
title: poloshing asyncio Streams API -> polishing asyncio Streams API

___
Python tracker 
<https://bugs.python.org/issue38164>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38225] iscoroutinefunction broken with cython - allow tagging of functions as async?

[Yury Selivanov]

Yury Selivanov  added the comment:

> If a python piece of code imports cython code with async defs, 
> `asyncio.iscoroutinefunction` cannot determine that the code is async.


Well, this seems to be a regression.  IIRC Cython used to emulate a Python code 
object (__code__) for functions it compiled along with the appropriate 
CO_COROUTINE flag set.  Wasn't that the case?

--

___
Python tracker 
<https://bugs.python.org/issue38225>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

New submission from Yury Selivanov :

== Context

1. Andrew and I implemented a new streaming API in asyncio 3.8.  The key idea 
is that there's a single Stream object, combining old StreamReader & 
StreamWriter APIs.  On top of that, `Stream.write()` and `Stream.close()` 
became coroutines.  The new API is significantly easier to use, but it required 
us to:

(a) Make backwards compatible changes to subprocess APIs;
(b) Add two new APIs: `asyncio.connect() -> Stream` and `asyncio.StreamServer`
(c) Soft-deprecate `asyncio.open_connection()` and `asyncio.start_serving()`.


2. The Trio project considers implementing new Streams API (see [1]).  The key 
idea is to make the core Stream object very simple and then enable building 
complex protocol pipelines using composition.  Want SSL?  Add an SSL layer.  
Have a framed binary protocol?  Push a reusable framing layer to the stack and 
parse the protocol.  On top of that, Trio wants us to standardize Streams, so 
that it's possible to write framework agnostic protocol code using async/await 
and to even reuse things like SSL implementation.


== Problem

I really like how Trio approaches this.

The asyncio.Stream class we have now is overloaded with functionality.  It's 
not composable.  It's internal buffer and APIs are designed to parsing text 
protocols (i.e. parsing a complex binary protocol requires an entirely 
different buffer implementation).

Long story short, I think we should revert the new streaming API from the 3.8 
branch and see if Trio & asyncio can design a better Streaming API.  Otherwise 
we end up in a weird situation where we added a bunch of new APIs to 3.8 which 
can be deprecated in 3.9.

Worst case scenario we'll just ship our current versions of Streams in 3.9 (not 
in 3.8).

Thoughts?

[1] https://github.com/python-trio/trio/issues/1208

--
components: asyncio
messages: 352908
nosy: asvetlov, gvanrossum, lukasz.langa, njs, yselivanov
priority: release blocker
severity: normal
status: open
title: Revert the new asyncio Streams API
type: behavior
versions: Python 3.8

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:

> Task group is another very required thing. If I choose between groups and 
> steams I probably invest my time into the former.

We should get them in 3.9.  I'm going to be working on the ExceptionGroup PEP 
today.

--

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38248] inconsistency in asyncio.Task between cancellation while running vs. cancellation immediately after it finishes

[Yury Selivanov]

Yury Selivanov  added the comment:

> The obvious fix would be to modify the 'except StopIteration' branch to 
> handle this case by calling super().cancel() instead of 
> super().set_exception(...).

Yeah, I think this is the solution we should do in 3.8.

--

___
Python tracker 
<https://bugs.python.org/issue38248>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38248] inconsistency in asyncio.Task between cancellation while running vs. cancellation immediately after it finishes

[Yury Selivanov]

Change by Yury Selivanov :


--
keywords: +patch
pull_requests: +15907
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/16330

___
Python tracker 
<https://bugs.python.org/issue38248>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38260] asyncio.run documentation does not mention its return value

[Yury Selivanov]

Change by Yury Selivanov :


--
nosy: +aeros167

___
Python tracker 
<https://bugs.python.org/issue38260>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:

Nathaniel, thanks for the summary!


A few comments to address some points:


> So Yury and I are tentatively thinking we'll make some kind of PEP for the 
> 3.9 timescale, probably just for the core ABCs

Yes!  At the very least for things like asynchronous version of "closable", 
e.g. an object with an "aclose()" coroutine method.  I'm sure there are some 
other straightforward design patterns that we can codify.  Maybe we can do that 
for streams too -- see some thoughts below.


> First decision point: will asyncio.Stream implement the ABCs directly, or 
> will you need some kind of adapter?

I'd love asyncio.Stream to implement the ABCs directly.  The only problem is 
that Trio isn't yet settled on the design of those ABCs and we need to make 
some decisions for asyncio *now*.

I hope that the Trio project can minimize the number of methods they want to 
add to those ABCs so that we don't need to duplicate a lot of functionality in 
asyncio.Stream.  E.g. in the new asyncio.Stream there's a Stream.write() 
coroutine; in Trio it's Stream.send_all().  I'm not entirely convinced that 
"send_all()" is a good name, for example, even though I now understand the 
motivation.  We can discuss that later in a relevant issue though.

Another important point to consider: if the new Trio Stream ABCs are 
*significantly different* from asyncio.Stream and would require us to alias too 
many methods or to do heavy refactoring and deprecations, then Trio will have 
to show some real world usage and traction of its APIs first.


> If we completely separated the old StreamReader/StreamWriter functions from 
> the new Stream functions, then it would also have another advantage: we could 
> clean up several issues with Stream that are only needed for compatibility 
> with the old APIs. In particular, we could get rid of the optional-await 
> hacks on 'write' and 'close', and turn them into regular coroutines.

We'd like to avoid that and have one asyncio.Stream class in asyncio.  Using 
legacy StreamReader/StreamWriter functions for subprocesses alone (long term) 
isn't a solution for us, since there're real problems with .write() and 
.close() not being awaitables.  Sticking to having a new asyncio.Stream API and 
old StreamReader/StreamWriter for subprocesses isn't an acceptable solution 
either.  We'd like to minimize the API surface that asyncio users have to deal 
with.


> The obvious change would be to leave out __aiter__ from asyncio.Stream in 3.8.

> Option 1: Keep the Stream code as-is, and accept that using asyncio.Stream 
> with future ABC-based code will require some compromises

Nathaniel, I think it's important to emphasize that those compromises should be 
mutual.  I'm willing to support changing "Stream.close()" to "Stream.aclose()" 
and to perhaps alias some methods.  We can also implement "Stream.chunks()".  
But changing the semantics of "__aiter__" is, unfortunately, not on the table, 
at least for me.

If Trio doesn't want to change the __aiter__ semantics of its Stream ABC (which 
is only a *proposal* right now!), then:

- Fragmenting asyncio APIs by letting subprocesses use old 
StreamReader/StreamWriter while we have new asyncio.Stream isn't an option.

- Asking us to implement new subprocess APIs just for the sake of having 
different Stream implementation for Process.std* channels isn't an option 
either.  

Adding new APIs and deprecating old ones is a huge burden on asyncio 
maintainers and users.  So the "obvious change" for *me* would be using 
"Stream.chunks()" iterator in Trio.  For Trio it's a question of whether the 
new API is pretty; for asyncio it's a question of how many APIs we need to 
deprecate/change.  I hope you understand my position and why I am strong -1 on 
the "Option 2".


> Right now we don't have any path to fixing 'write'/'close' at all;

Andrew and I discussed that this morning.  Here's the plan:

1. We add "aclose()" and "write()" coroutines to the new "asyncio.Stream()".  
It won't have "wait_closed()" or "drain()" or "close()".

2. We add a _LegacyStream class derived from the new asyncio.Stream.  We will 
use it for subprocesses. Its "write()" method will return an 
"OptionallyAwaitable" wrapper that will nudge users to add an await in front of 
"stdin.write()".  _LegacyStream will be completely backwards compatible.

This path enables us to add a decent new streaming API while retaining 
consistency and backwards compatibility.


> BTW Andrew, while writing that I realized that there's also overlap between 
> your new Server classes an

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:

> So like... both of these approaches are definitely possible, but to me it 
> seems like if you look at it holistically, your approach is actually making 
> the documentation and deprecations *more* complicated, not less.

I think Nathaniel might have a point here.  Andrew, Guido, what do you think?

--

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:

I slept on this and discussed this issue privately with a few non-involved 
people who use asyncio daily.

My final opinion on this issue: we must revert the new streaming API from 
asyncio and take our time to design it properly.  I don't like what we have in 
the 3.8 branch right now.  I don't feel comfortable rushing decisions and doing 
last minute API changes.

Andrew, do you want me to submit a PR or you can do it?

--

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37141] Allow multiple separators in Stream.readuntil

[Yury Selivanov]

Yury Selivanov  added the comment:

I already mentioned that in the PR, but we'll have to hit a pause on this.  We 
decided to revert the latest streams implementation from 3.8, see  
https://bugs.python.org/issue38242.  The upshot is that what we have in 3.9 
should be more amenable for things like this one.

--

___
Python tracker 
<https://bugs.python.org/issue37141>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37141] Allow multiple separators in Stream.readuntil

[Yury Selivanov]

Change by Yury Selivanov :


--
resolution:  -> later
status: open -> pending

___
Python tracker 
<https://bugs.python.org/issue37141>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:

> Since this has been elevated to a release blocker, I wouldn't mind helping to 
> revert this ASAP. I can open a PR to fix it today.

Sure, by all means, any help would be hugely appreciated.  Thank you, Kyle.

You'll need to be careful to only revert the new functions & the asyncio.Stream 
class.  Also the new docs.  Due to proximity to the deadline, please be 
prepared that we might need to abandon your pull request if it's not ready by 
Sunday morning.  In which case Andrew or I will do this ourselves.

--

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +16066
pull_request: https://github.com/python/cpython/pull/16482

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36889] Merge StreamWriter and StreamReader into just asyncio.Stream

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +16067
pull_request: https://github.com/python/cpython/pull/16482

___
Python tracker 
<https://bugs.python.org/issue36889>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36889] Merge StreamWriter and StreamReader into just asyncio.Stream

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 6758e6e12a71ef5530146161881f88df1fa43382 by Yury Selivanov in 
branch 'master':
bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482)
https://github.com/python/cpython/commit/6758e6e12a71ef5530146161881f88df1fa43382


--

___
Python tracker 
<https://bugs.python.org/issue36889>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 6758e6e12a71ef5530146161881f88df1fa43382 by Yury Selivanov in 
branch 'master':
bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482)
https://github.com/python/cpython/commit/6758e6e12a71ef5530146161881f88df1fa43382


--

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +16070
pull_request: https://github.com/python/cpython/pull/16485

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36889] Merge StreamWriter and StreamReader into just asyncio.Stream

[Yury Selivanov]

Change by Yury Selivanov :


--
pull_requests: +16071
pull_request: https://github.com/python/cpython/pull/16485

___
Python tracker 
<https://bugs.python.org/issue36889>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36889] Merge StreamWriter and StreamReader into just asyncio.Stream

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 1c19d656a79a00f58361ceb61c0a6d1faf90c686 by Yury Selivanov in 
branch '3.8':
bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482) 
(#16485)
https://github.com/python/cpython/commit/1c19d656a79a00f58361ceb61c0a6d1faf90c686


--

___
Python tracker 
<https://bugs.python.org/issue36889>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:

I've reverted the code. Andrew, would really appreciate if you could quickly do 
a post commit review.

--
stage: patch review -> commit review

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38242] Revert the new asyncio Streams API

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 1c19d656a79a00f58361ceb61c0a6d1faf90c686 by Yury Selivanov in 
branch '3.8':
bpo-38242: Revert "bpo-36889: Merge asyncio streams (GH-13251)" (#16482) 
(#16485)
https://github.com/python/cpython/commit/1c19d656a79a00f58361ceb61c0a6d1faf90c686


--

___
Python tracker 
<https://bugs.python.org/issue38242>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30773] async generator receives wrong value when shared between coroutines

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset fc4a044a3c54ce21e9ed150f7d769fb479d34c49 by Yury Selivanov in 
branch 'master':
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel 
(#7468)
https://github.com/python/cpython/commit/fc4a044a3c54ce21e9ed150f7d769fb479d34c49


--

___
Python tracker 
<https://bugs.python.org/issue30773>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30773] async generator receives wrong value when shared between coroutines

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 2f87a7dc5a1ad7f37787f0adee242c931643f878 by Yury Selivanov (Miss 
Islington (bot)) in branch '3.8':
bpo-30773: Fix ag_running; prohibit running athrow/asend/aclose in parallel 
(GH-7468) (#16486)
https://github.com/python/cpython/commit/2f87a7dc5a1ad7f37787f0adee242c931643f878


--

___
Python tracker 
<https://bugs.python.org/issue30773>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30773] async generator receives wrong value when shared between coroutines

[Yury Selivanov]

Change by Yury Selivanov :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue30773>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37146] opcode cache for LOAD_GLOBAL emits false alarm in memory leak hunting

[Yury Selivanov]

Yury Selivanov  added the comment:

> It seems like nobody came up with a solution for the debug mode since June. I 
> close the issue. Reopen it if you would like to propose a solution.

I think the only solution is to have a flag to disable optimizations, 
inlcluding this one.

--

___
Python tracker 
<https://bugs.python.org/issue37146>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38248] inconsistency in asyncio.Task between cancellation while running vs. cancellation immediately after it finishes

[Yury Selivanov]

Change by Yury Selivanov :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue38248>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34344] Fix the docstring for AbstractEventLoopPolicy.get_event_loop

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset b23a8423a923077e4f83d3f328bb7542b4c940ed by Yury Selivanov 
(idomic) in branch 'master':
bpo-34344 Fix AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)
https://github.com/python/cpython/commit/b23a8423a923077e4f83d3f328bb7542b4c940ed


--

___
Python tracker 
<https://bugs.python.org/issue34344>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36356] Failure to build with address sanitizer

[Yury Selivanov]

Yury Selivanov  added the comment:


New changeset 321def805abc5b7c92c7e90ca90cb2434fdab855 by Yury Selivanov (Ben 
Harper) in branch 'master':
bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)
https://github.com/python/cpython/commit/321def805abc5b7c92c7e90ca90cb2434fdab855


--
nosy: +yselivanov

___
Python tracker 
<https://bugs.python.org/issue36356>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33803] contextvars: hamt_alloc() must initialize h_root and h_count fields

[Yury Selivanov]

Yury Selivanov  added the comment:

> In bpo-38392, I modified PyObject_GC_Track() in debug mode to detect this bug.

Awesome!!!

--

___
Python tracker 
<https://bugs.python.org/issue33803>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36356] Failure to build with address sanitizer

[Yury Selivanov]

Change by Yury Selivanov :


--
nosy:  -yselivanov

___
Python tracker 
<https://bugs.python.org/issue36356>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38423] Event loop implementation docs advertise set_event_loop which doesn't work with asyncio.run

[Yury Selivanov]

Yury Selivanov  added the comment:

Yes, docs updates sound good.

--

___
Python tracker 
<https://bugs.python.org/issue38423>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32856] Optimize the `for y in [x]` idiom in comprehensions

[Yury Selivanov]

Change by Yury Selivanov :


--
nosy:  -yselivanov

___
Python tracker 
<https://bugs.python.org/issue32856>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38559] async generators aclose() behavior in 3.8

[Yury Selivanov]

New submission from Yury Selivanov :

I believe I might have discovered a problem with asynchronous generators in 3.8.


# Prelude

In Python prior to 3.8 it was possible to overlap running of "asend()" and 
"athrow()" methods for the same asynchronous generator.

In plain English, it was possible to await on "agen.asend()" while some other 
coroutine is already awaiting on "agen.asend()".  That created all kinds of 
problems with asynchronous generators when people used them to implement 
channels by trying to push values into the same asynchronous generator from 
different coroutines.  Regular code that used asynchronous generators in plain 
and non-sophisticated way did not care.

For classic synchronous generators we have a check for preventing overlapping 
use of "send()" and "throw()" -- the "gi_running" flag.  If the flag is set, 
both methods raise a RuntimeError saying that "generator already executing".


# Python 3.8

As was discussed in issues #30773 and #32526 we decided to replicate the same 
behavior for asynchronous generators, mainly:

* add an "ag_running" flag;

* set "ag_running" when "anext()" or "athrow()" begin to rung, and set it off 
when they are finished executing;

* if the flag is set when we are about to run "anext()" or "athrow()" it means 
that another coroutine is reusing the same generator object in parallel and so 
we raise a RuntimeError.


# Problem

Closing a generator involves throwing a GeneratorExit exception into it.  
Throwing the exception is done via calling "throw()" for sync generators, and 
"athrow()" for async generators.

As shown in https://gist.github.com/1st1/d9860cbf6fe2e5d243e695809aea674c, it's 
an error to close a synchronous generator while it is being iterated.  This is 
how async generators *started to behave* in 3.8.

The problem is that asyncio's "loop.shutdown_asyncgens()" method tries to 
shutdown orphan asynchronous generators by calling "aclose()" on them.  The 
method is public API and is called by "asyncio.run()" automatically.

Prior to 3.8, calling "aclose()" worked (maybe not in the most clean way). A 
GeneratorExit was thrown into an asynchronous generator regardless of whether 
it was running or not, aborting the execution.

In 3.8, calling "aclose()" can crash with a RuntimeError.  It's no longer 
possible to *reliably cancel* a running asynchrounous generator.


# Dilemma

Replicating the behavior of synchronous generators in asynchronous generators 
seems like the right thing.  But it seems that the requirements for 
asynchronous generators are different, and 3.8 breaks backwards compat.


# Proposed Solution

We keep "asend()" and "athrow()" as is in 3.8.  They will continue to raise 
RuntimeError if used in parallel on the same async generator.

We modify "aclose()" to allow it being called in parallel with "asend()" or 
"athrow()".  This will restore the <3.8 behavior and fix the 
"loop.shutdown_asyncgens()" method.


Thoughts?

--
components: Interpreter Core
messages: 355158
nosy: asvetlov, gvanrossum, lukasz.langa, ncoghlan, njs, yselivanov
priority: normal
severity: normal
status: open
title: async generators aclose() behavior in 3.8
type: behavior
versions: Python 3.8, Python 3.9

___
Python tracker 
<https://bugs.python.org/issue38559>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38559] async generators aclose() behavior in 3.8

[Yury Selivanov]

Yury Selivanov  added the comment:

Just discussed this issue off-list with Nathaniel.  Maybe this isn't as severe 
as I thought it is.

If we cancel all asyncio tasks before calling `loop.shutdown_asyncgens()` this 
*probably* becomes a non-issue.  And "asyncio.run()" does just that [1].

Any asynchronous generator ultimately needs a task to iterate it, so if we 
cancel tasks, we should cancel all asynchronous generators too.  Calling 
`loop.shutdown_asyncgens()` then ensures that there are no orphan async 
generators.  The only way (at least known to me) that that could be the case if 
some tasks ignores cancellation.  

Nathaniel, what do you think about this in the context of Trio?

If anything, we can update asyncio docs explaining the new behavior of async 
generators and how to use "loop.shutdown_asyncgens()".

[1] 
https://github.com/python/cpython/blob/91528f40c30717563a478920861c81d18da5bf63/Lib/asyncio/runners.py#L46

--

___
Python tracker 
<https://bugs.python.org/issue38559>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32526] Closing async generator while it is running does not raise an exception

[Yury Selivanov]

Yury Selivanov  added the comment:

Was fixed as part of #30773.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue32526>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38559] async generators aclose() behavior in 3.8

[Yury Selivanov]

Yury Selivanov  added the comment:

> So I think asyncio.run is actually totally fine with the 3.8.0 behavior.

Yes.  Looks like it.

> It's only explicit calls to shutdown_asyncgens that might run into this, and 
> I think that's probably OK?

Yes.  Calling shutdown_asyncgens after all tasks are cancelled is still useful 
to cleanup asynchronous generators that were created but not yet GCed or 
iterated.

> And if we did allow aclose() to run at any time, then I worry that that could 
> cause serious problems. Users can call aclose() at will, and it's generally 
> good practice to always call aclose() on your async generators explicitly. So 
> it's entirely possible that users will accidentally call aclose() themselves 
> while they have another task is blocked in __anext__. And in that case 
> what do we do?

I agree.  Avoiding that kind of mess was the main motivation to fix ag_running.

> So... this is super subtle and confusing, but I *think* the conclusion is 
> that yeah, 3.8.0 is fine and there's no urgent action needed.

Let's see if anyone else thinks this might be a problem, but yeah, it seems 
that the only programs that can suffer from this change are those that don't 
use 'asyncio.run'.  And if so they should be fixed to explicitly cancel all 
tasks before calling shutdown_asyncgens.

--

___
Python tracker 
<https://bugs.python.org/issue38559>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34776] Postponed annotations break inspection of dataclasses

[Yury Selivanov]

Yury Selivanov  added the comment:

FWIW we discussed my patch with Eric at PyCon 2019 and I think he had no issues 
with it.  Eric, can I merge it?  Should we backport to 3.8?

--

___
Python tracker 
<https://bugs.python.org/issue34776>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34776] Postponed annotations break inspection of dataclasses

[Yury Selivanov]

Yury Selivanov  added the comment:

> Yury: I'm okay with merging. If I recall, you were going to add a comment or 
> two about the approach (a closure, if I remember correctly).

Sure, I'll rebase and add a comment in a couple of days.  I'll elevate the 
status so that I don't forget before 3.8.1

--
assignee: eric.smith -> yselivanov
priority:  -> deferred blocker

___
Python tracker 
<https://bugs.python.org/issue34776>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30064] BaseSelectorEventLoop.sock_{recv, sendall}() don't remove their callbacks when canceled

[Yury Selivanov]

Yury Selivanov  added the comment:

I think this broke asyncio a bit.  These two uvloop sock_* API tests no longer 
pass on asyncio 3.8:

* https://github.com/MagicStack/uvloop/blob/master/tests/test_sockets.py#L192

* https://github.com/MagicStack/uvloop/blob/master/tests/test_sockets.py#L238

Andrew, since this was your patch, please take a look.

--
resolution: fixed -> 
stage: resolved -> 
status: closed -> open

___
Python tracker 
<https://bugs.python.org/issue30064>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue30064] BaseSelectorEventLoop.sock_{recv, sendall}() don't remove their callbacks when canceled

[Yury Selivanov]

Yury Selivanov  added the comment:

I'll elevate the status so that we don't forget before 3.8.1 is too close

--
priority: normal -> deferred blocker

___
Python tracker 
<https://bugs.python.org/issue30064>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38559] async generators aclose() behavior in 3.8

[Yury Selivanov]

Yury Selivanov  added the comment:

The more I think about this the more I like new 3.8 behavior. I'm going to 
close this issue; if anything comes up I'll reopen it later. Sorry for the 
noise.

--
resolution:  -> not a bug
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue38559>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue20443] __code__. co_filename should always be an absolute path

[Yury Selivanov]

Change by Yury Selivanov :


--
nosy:  -yselivanov

___
Python tracker 
<https://bugs.python.org/issue20443>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecating MultiLoopChildWatcher

[Yury Selivanov]

Yury Selivanov  added the comment:

Didn't we just add MultiLoopChildWatcher in 3.8?

--
status: pending -> open

___
Python tracker 
<https://bugs.python.org/issue38591>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecating MultiLoopChildWatcher

[Yury Selivanov]

Yury Selivanov  added the comment:

> but perhaps if we go through with that we should remove MultiLoopChildWatcher 
> from 3.8 (if it's not too late to do so) instead of deprecating it.

I'll leave that up to Andrew to decide, but I'd be +1 to drop it asap, 
especially if we want to eventually deprecate watchers.  

Speaking of watchers -- big +1 from me to drop them all at some point. I would 
start as early as 3.9.

Linux has pidfd now, freebsd/macos has kqueue, windows has its own apis for 
watching processes. Threads can be the backup method for OSes that lack proper 
APIs for watching multiple processes (without using SIGCHLD etc).

--

___
Python tracker 
<https://bugs.python.org/issue38591>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38591] Deprecating MultiLoopChildWatcher

[Yury Selivanov]

Yury Selivanov  added the comment:

Kyle, why are you resetting the status to "Pending"?

> I think it will be necessary to fix the issues in the near future if we want 
> to keep MultiLoopChildWatcher in 3.8 (as they apply to 3.8 and 3.9). I would 
> certainly not be ecstatic about the idea of removing something that was just 
> recently added (similar to how we had to revert the new streaming changes due 
> to API design), but I'm even less in favor of keeping something around that's 
> not stable in a final release. 

Your opinion on this is duly noted.  

It would be great to hear what Andrew and Victor think about this.

--
status: pending -> open

___
Python tracker 
<https://bugs.python.org/issue38591>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34790] Deprecate passing coroutine objects to asyncio.wait()

[Yury Selivanov]

Yury Selivanov  added the comment:

I'd add `and will be removed in 3.11.` now.

--

___
Python tracker 
<https://bugs.python.org/issue34790>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue38599] Deprecate creation of asyncio object when the loop is not running

[Yury Selivanov]

Yury Selivanov  added the comment:

Yes.

As a remedy for this particular problem we can add checks here and there 
comparing `asyncio.get_running_loop()` and `self._loop`.  Performance will 
suffer a bit but the usability will greatly improve.

--

___
Python tracker 
<https://bugs.python.org/issue38599>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com