Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 13 June 2015 at 04:13, Guido van Rossum wrote: > IOW I don't think that the problem here is that you haven't sufficiently > motivated your use case -- you are asking for information that just isn't > available. (Which is actually where you started the thread -- you can get to > the frame of the coroutine but there's nowhere to go from that frame.) If I'm understanding Ben's request correctly, it isn't really the stack trace that he's interested in (as you say, that specific phrasing doesn't match the way coroutine suspension works), but rather having visibility into the chain of control flow delegation for currently suspended frames: what operation is the outermost frame ultimately blocked *on*, and how did it get to the point of waiting for that operation? At the moment, all of the coroutine and generator-iterator resumption information is implicit in the frame state, so we can't externally introspect the delegation of control flow in a case like Ben's original example (for coroutines) or like this one for generators: def g1(): yield 42 def g2(): yield from g1() g = g2() next(g) # We can tell here that g is suspended # We can't tell that it delegated flow to a g1() instance I wonder if in 3.6 it might be possible to *add* some bookkeeping to "await" and "yield from" expressions that provides external visibility into the underlying iterable or coroutine that the generator-iterator or coroutine has delegated flow control to. As an initial assessment, the runtime cost would be: * an additional pointer added to generator/coroutine objects to track control flow delegation * setting that when suspending in "await" and "yield from" expressions * clearing it when resuming in "await" and "yield from" expressions (This would be a read-only borrowed reference from a Python level perspective, so it shouldn't be necessary to alter the reference count - we'd just be aliasing the existing reference from the frame's internal stack state) Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 13 June 2015 at 17:22, Nick Coghlan wrote:
> On 13 June 2015 at 04:13, Guido van Rossum wrote:
>> IOW I don't think that the problem here is that you haven't sufficiently
>> motivated your use case -- you are asking for information that just isn't
>> available. (Which is actually where you started the thread -- you can get to
>> the frame of the coroutine but there's nowhere to go from that frame.)
>
> If I'm understanding Ben's request correctly, it isn't really the
> stack trace that he's interested in (as you say, that specific
> phrasing doesn't match the way coroutine suspension works), but rather
> having visibility into the chain of control flow delegation for
> currently suspended frames: what operation is the outermost frame
> ultimately blocked *on*, and how did it get to the point of waiting
> for that operation?
>
> At the moment, all of the coroutine and generator-iterator resumption
> information is implicit in the frame state, so we can't externally
> introspect the delegation of control flow in a case like Ben's
> original example (for coroutines) or like this one for generators:
>
> def g1():
> yield 42
>
> def g2():
> yield from g1()
>
> g = g2()
> next(g)
> # We can tell here that g is suspended
> # We can't tell that it delegated flow to a g1() instance
>
> I wonder if in 3.6 it might be possible to *add* some bookkeeping to
> "await" and "yield from" expressions that provides external visibility
> into the underlying iterable or coroutine that the generator-iterator
> or coroutine has delegated flow control to. As an initial assessment,
> the runtime cost would be:
>
> * an additional pointer added to generator/coroutine objects to track
> control flow delegation
> * setting that when suspending in "await" and "yield from" expressions
> * clearing it when resuming in "await" and "yield from" expressions
Thanks Nick for rephrasing with the appropriate terminology. I had tried
to get it right but with a background of implementing OS kernels, I have
a strong habit of existing terminology to break.
I agree with your suggestion that explicitly having the pointers is much
nicer than my opcode hack implementation.
Without side-tracking this discussion I do just want to say that the
hypothetical code is something that actually works if frame objects
expose the stack, which is possibly as easy as:
+static PyObject *
+frame_getstack(PyFrameObject *f, void *closure)
+{
+PyObject **p;
+PyObject *list = PyList_New(0);
+
+if (list == NULL)
+return NULL;
+
+if (f->f_stacktop != NULL) {
+for (p = f->f_valuestack; p < f->f_stacktop; p++) {
+/* FIXME: is this the correct error handling condition? */
+if (PyList_Append(list, *p)) {
+Py_DECREF(list);
+return NULL;
+}
+}
+}
+
+return list;
+}
I have implemented and tested this and it worked well (although I really
don't know CPython internals well enough to know if the above code doesn't
have some serious issues with it).
Is there any reason an f_stack attribute is not exposed for frames? Many of the
other PyFrameObject values are exposed. I'm guessing that there probably
aren't too many places where you can get hold of a frame that doesn't have an
empty stack in normal operation, so it probably isn't necessary.
Anyway, I'm not suggesting that adding f_stack is better than explicitly
adding pointers, but it does seem a more general thing that can be exposed
and enable this use case without requiring extra book-keeping data structures.
Cheers,
Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On Sat, Jun 13, 2015 at 12:22 AM, Nick Coghlan wrote: > On 13 June 2015 at 04:13, Guido van Rossum wrote: > > IOW I don't think that the problem here is that you haven't sufficiently > > motivated your use case -- you are asking for information that just isn't > > available. (Which is actually where you started the thread -- you can > get to > > the frame of the coroutine but there's nowhere to go from that frame.) > > If I'm understanding Ben's request correctly, it isn't really the > stack trace that he's interested in (as you say, that specific > phrasing doesn't match the way coroutine suspension works), but rather > having visibility into the chain of control flow delegation for > currently suspended frames: what operation is the outermost frame > ultimately blocked *on*, and how did it get to the point of waiting > for that operation? > > At the moment, all of the coroutine and generator-iterator resumption > information is implicit in the frame state, so we can't externally > introspect the delegation of control flow in a case like Ben's > original example (for coroutines) or like this one for generators: > > def g1(): > yield 42 > > def g2(): > yield from g1() > > g = g2() > next(g) > # We can tell here that g is suspended > # We can't tell that it delegated flow to a g1() instance > > I wonder if in 3.6 it might be possible to *add* some bookkeeping to > "await" and "yield from" expressions that provides external visibility > into the underlying iterable or coroutine that the generator-iterator > or coroutine has delegated flow control to. As an initial assessment, > the runtime cost would be: > > * an additional pointer added to generator/coroutine objects to track > control flow delegation > * setting that when suspending in "await" and "yield from" expressions > * clearing it when resuming in "await" and "yield from" expressions > > (This would be a read-only borrowed reference from a Python level > perspective, so it shouldn't be necessary to alter the reference count > - we'd just be aliasing the existing reference from the frame's > internal stack state) > Ah, this makes sense. I think the object you're after is 'reciever' [sic] in the YIELD_FROM opcode implementation, right? -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 13 June 2015 at 19:03, Guido van Rossum wrote: > On Sat, Jun 13, 2015 at 12:22 AM, Nick Coghlan wrote: >> >> On 13 June 2015 at 04:13, Guido van Rossum wrote: >> > IOW I don't think that the problem here is that you haven't sufficiently >> > motivated your use case -- you are asking for information that just >> > isn't >> > available. (Which is actually where you started the thread -- you can >> > get to >> > the frame of the coroutine but there's nowhere to go from that frame.) >> >> If I'm understanding Ben's request correctly, it isn't really the >> stack trace that he's interested in (as you say, that specific >> phrasing doesn't match the way coroutine suspension works), but rather >> having visibility into the chain of control flow delegation for >> currently suspended frames: what operation is the outermost frame >> ultimately blocked *on*, and how did it get to the point of waiting >> for that operation? >> >> At the moment, all of the coroutine and generator-iterator resumption >> information is implicit in the frame state, so we can't externally >> introspect the delegation of control flow in a case like Ben's >> original example (for coroutines) or like this one for generators: >> >> def g1(): >> yield 42 >> >> def g2(): >> yield from g1() >> >> g = g2() >> next(g) >> # We can tell here that g is suspended >> # We can't tell that it delegated flow to a g1() instance >> >> I wonder if in 3.6 it might be possible to *add* some bookkeeping to >> "await" and "yield from" expressions that provides external visibility >> into the underlying iterable or coroutine that the generator-iterator >> or coroutine has delegated flow control to. As an initial assessment, >> the runtime cost would be: >> >> * an additional pointer added to generator/coroutine objects to track >> control flow delegation >> * setting that when suspending in "await" and "yield from" expressions >> * clearing it when resuming in "await" and "yield from" expressions >> >> (This would be a read-only borrowed reference from a Python level >> perspective, so it shouldn't be necessary to alter the reference count >> - we'd just be aliasing the existing reference from the frame's >> internal stack state) > > > Ah, this makes sense. I think the object you're after is 'reciever' [sic] in > the YIELD_FROM opcode implementation, right? Right this is the exact book-keeping that I was originally referring to in my previous emails (sorry for not making that more explicit earlier). The 'reciever' is actually on the stack part of the co-routine's frame all the time (i.e.: pointed to via f->f_stacktop). So from my point of view the book-keeping is there (albeit somewhat obscurely!), but the objects on the frame's stack aren't exposed via the Python wrapping of PyFrameObject, (although could, I think, easily be exposed). Once you get at the receiver object (which is another co-routine) you can traverse down to the point the co-routine 'switched'. Cheers, Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
Hi ,
I had a Question,i hope i'll find the solution here.
Say i have a Queue.
>>> h = Queue.Queue(maxsize=0)
>>> h.put(1)
>>> h.put(2)
>>> h.empty()
False
>>> h.join()
>>> h.empty()
False
>>> h.get()
1
>>> h.get()
2
>>> h.get()
Blocked...
My Question is :
In a single threaded environment why does the get() gets blocked , instead
of raising an exception.On interpreter i have no way to resume working.
And my second question is :
Why doe we have to explicitly call task_done after get(). why doesn't get()
implicitly call task_done().
as for put() entry for unfinished_task is automatically added .why not get
deleted in get() then.
Thanks.
On Fri, May 29, 2015 at 10:16 AM, Ben Leslie wrote:
> Hi all,
>
> Apologies in advance; I'm not a regular, and this may have been
> handled already (but I couldn't find it when searching).
>
> I've been using the new async/await functionality (congrats again to
> Yury on getting that through!), and I'd like to get a stack trace
> between the place at which blocking occurs and the outer co-routine.
>
> For example, consider this code:
>
> """
> async def a():
> await b()
>
> async def b():
> await switch()
>
> @types.coroutine
> def switch():
> yield
>
> coro_a = a()
> coro_a.send(None)
> """
>
> At this point I'd really like to be able to somehow get a stack trace
> similar to:
>
> test.py:2
> test.py:4
> test.py:9
>
> Using the gi_frame attribute of coro_a, I can get the line number of
> the outer frame (e.g.: line 2), but from there there is no way to
> descend the stack to reach the actual yield point.
>
> I thought that perhaps the switch() co-routine could yield the frame
> object returned from inspect.currentframe(), however once that
> function yields that frame object has f_back changed to None.
>
> A hypothetical approach would be to work the way down form the
> outer-frame, but that requires getting access to the co-routine object
> that the outer-frame is currently await-ing. Some hypothetical code
> could be:
>
> """
> def show(coro):
> print("{}:{}".format(coro.gi_frame.f_code.co_filename,
> coro.gi_frame.f_lineno))
> if dis.opname[coro.gi_code.co_code[coro.gi_frame.f_lasti + 1]] ==
> 'YIELD_FROM':
> show(coro.gi_frame.f_stack[0])
> """
>
> This relies on the fact that an await-ing co-routine will be executing
> a YIELD_FROM instruction. The above code uses a completely
> hypothetical 'f_stack' property of frame objects to pull the
> co-routine object which a co-routine is currently await-ing from the
> stack. I've implemented a proof-of-concept f_stack property in the
> frameobject.c just to test out the above code, and it seems to work.
>
> With all that, some questions:
>
> 1) Does anyone else see value in trying to get the stack-trace down to
> the actual yield point?
> 2) Is there a different way of doing it that doesn't require changes
> to Python internals?
> 3) Assuming no to #2 is there a better way of getting the information
> compared to the pretty hacking byte-code/stack inspection?
>
> Thanks,
>
> Ben
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/jaivishkothari10104733%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 13 June 2015 at 20:25, Ben Leslie wrote: > Is there any reason an f_stack attribute is not exposed for frames? Many of > the > other PyFrameObject values are exposed. I'm guessing that there probably > aren't too many places where you can get hold of a frame that doesn't have an > empty stack in normal operation, so it probably isn't necessary. There's also the fact that anything we do in CPython that assumes a value stack based interpreter implementation might not work on other Python implementations, so we generally try to keep that level of detail hidden. > Anyway, I'm not suggesting that adding f_stack is better than explicitly > adding pointers, but it does seem a more general thing that can be exposed > and enable this use case without requiring extra book-keeping data structures. Emulating CPython frames on other implementations is already hard enough, without exposing the value stack directly. Compared to "emulate CPython's value stack", "keep track of delegation to subiterators and subcoroutines" is a far more reasonable request to make of developers of other implementations. >From a learnability perspective, there's also nothing about an "f_stack" attribute that says "you can use this to find out where a generator or coroutine has delegated control", while attributes like "gi_delegate" or "cr_delegate" would be more self-explanatory. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On Sat, Jun 13, 2015 at 9:21 AM, Nick Coghlan wrote: > On 13 June 2015 at 20:25, Ben Leslie wrote: > > Is there any reason an f_stack attribute is not exposed for frames? Many > of the > > other PyFrameObject values are exposed. I'm guessing that there probably > > aren't too many places where you can get hold of a frame that doesn't > have an > > empty stack in normal operation, so it probably isn't necessary. > > There's also the fact that anything we do in CPython that assumes a > value stack based interpreter implementation might not work on other > Python implementations, so we generally try to keep that level of > detail hidden. > > > Anyway, I'm not suggesting that adding f_stack is better than explicitly > > adding pointers, but it does seem a more general thing that can be > exposed > > and enable this use case without requiring extra book-keeping data > structures. > > Emulating CPython frames on other implementations is already hard > enough, without exposing the value stack directly. > I'm not sure how strong this argument is. We also expose bytecode, which is about as unportable as anything. (Though arguably we can't keep bytecode a secret, because it's written to .pyc files.) I find Ben's implementation pretty straightforward, and because it makes a copy, I don't think there's anything one could do with the exposed stack that could violate any of the interpreter's invariants (I might change it to return a tuple to emphasize this point to the caller though). But I agree it isn't a solution to the question about the suspension "stack". > Compared to "emulate CPython's value stack", "keep track of delegation > to subiterators and subcoroutines" is a far more reasonable request to > make of developers of other implementations. > > From a learnability perspective, there's also nothing about an > "f_stack" attribute that says "you can use this to find out where a > generator or coroutine has delegated control", while attributes like > "gi_delegate" or "cr_delegate" would be more self-explanatory. > Stack frame objects are kind of expensive and I would hate to add an extra pointer to every frame just to support this functionality. Perhaps we could have a flag though that says whether the top of the stack is in fact the generator object on which we're waiting in a yield-from? This flag could perhaps sit next to f_executing (also note that this new flag is mutually exclusive with f_executing). We could then easily provide a new method or property on the frame object that returns the desired generator if the flag is set or None if the flag is not set -- other Python implementations could choose to implement this differently. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 2015-06-13 11:38, jaivish kothari wrote: Hi , I had a Question,i hope i'll find the solution here. Say i have a Queue. >>> h = Queue.Queue(maxsize=0) >>> h.put(1) >>> h.put(2) >>> h.empty() False >>> h.join() >>> h.empty() False >>> h.get() 1 >>> h.get() 2 >>> h.get() Blocked... My Question is : In a single threaded environment why does the get() gets blocked , instead of raising an exception.On interpreter i have no way to resume working. And my second question is : Why doe we have to explicitly call task_done after get(). why doesn't get() implicitly call task_done(). as for put() entry for unfinished_task is automatically added .why not get deleted in get() then. The way it's used is to get a item, process it, and then signal that it has finished with it. There's going to be a period of time when an item is not in the queue, but is still being processed. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
Nick Coghlan wrote: I wonder if in 3.6 it might be possible to *add* some bookkeeping to "await" and "yield from" expressions that provides external visibility into the underlying iterable or coroutine that the generator-iterator or coroutine has delegated flow control to. In my original implementation of yield-from, stack frames had an f_yieldfrom slot referring to the object being yielded from. I gather that slot no longer exists at the C level, but it ought to be possible to provide a Python attribute or method returning the same information. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 14 June 2015 at 09:20, Greg Ewing wrote:
> Nick Coghlan wrote:
>>
>> I wonder if in 3.6 it might be possible to *add* some bookkeeping to
>> "await" and "yield from" expressions that provides external visibility
>> into the underlying iterable or coroutine that the generator-iterator
>> or coroutine has delegated flow control to.
>
>
> In my original implementation of yield-from, stack frames
> had an f_yieldfrom slot referring to the object being
> yielded from.
>
> I gather that slot no longer exists at the C level, but
> it ought to be possible to provide a Python attribute or
> method returning the same information.
OK, this is really easy to implement actually. I implemented it as gi_yieldfrom
on the generator object rather than f_yieldfrom on the frame object,
primarily as most of the related code is already in the genobject.c module.
Something like this:
diff --git a/Objects/genobject.c b/Objects/genobject.c
index 3c32e7b..bc42fe5 100644
--- a/Objects/genobject.c
+++ b/Objects/genobject.c
@@ -553,11 +553,22 @@ gen_set_qualname(PyGenObject *op, PyObject *value)
return 0;
}
+static PyObject *
+gen_getyieldfrom(PyGenObject *gen)
+{
+PyObject *yf = gen_yf(gen);
+if (yf == NULL)
+Py_RETURN_NONE;
+
+return yf;
+}
+
static PyGetSetDef gen_getsetlist[] = {
{"__name__", (getter)gen_get_name, (setter)gen_set_name,
PyDoc_STR("name of the generator")},
{"__qualname__", (getter)gen_get_qualname, (setter)gen_set_qualname,
PyDoc_STR("qualified name of the generator")},
+{"gi_yieldfrom", (getter)gen_getyieldfrom, NULL, NULL},
{NULL} /* Sentinel */
};
(Note: Above probably doesn't exactly follow correct coding conventions, and
probably need to add an appropriate doc string.)
This greatly simplifies my very original 'show' routine to:
def show(coro):
print("{}:{} ({})".format(coro.gi_frame.f_code.co_filename,
coro.gi_frame.f_lineno, coro))
if coro.gi_yieldfrom:
show(coro.gi_yieldfrom)
I think this would give the widest flexibility for non-CPython
implementations to implement
the same property in the most appropriate manner.
If this seems like a good approach I'll try and work it in to a
suitable patch for contribution.
Cheers,
Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 14 Jun 2015 03:35, "Guido van Rossum" wrote: > > On Sat, Jun 13, 2015 at 9:21 AM, Nick Coghlan wrote: >> From a learnability perspective, there's also nothing about an >> "f_stack" attribute that says "you can use this to find out where a >> generator or coroutine has delegated control", while attributes like >> "gi_delegate" or "cr_delegate" would be more self-explanatory. > > Stack frame objects are kind of expensive and I would hate to add an extra pointer to every frame just to support this functionality. Perhaps we could have a flag though that says whether the top of the stack is in fact the generator object on which we're waiting in a yield-from? This flag could perhaps sit next to f_executing (also note that this new flag is mutually exclusive with f_executing). We could then easily provide a new method or property on the frame object that returns the desired generator if the flag is set or None if the flag is not set -- other Python implementations could choose to implement this differently. Fortunately, we can expose this control flow delegation info through the generator-iterator and coroutine object APIs, rather than needing to do it directly on the frame. I'd missed that it could be done without *any* new C level state though - I now think Ben's right that we should be able to just expose the delegation lookup from the resumption logic itself as a calculated property. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Obtaining stack-frames from co-routine objects
On 14 Jun 2015 10:01, "Ben Leslie" wrote: > > If this seems like a good approach I'll try and work it in to a > suitable patch for contribution. I think it's a good approach, and worth opening an enhancement issue for. I expect any patch would need some adjustments after Yury has finished revising the async/await implementation to address some beta compatibility issues with functools.singledispatch and Tornado. Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
