[Python-Dev] Why are captured parameters also listed in co_varnames?
I'm trying to understand how CPython implements closure variable capture and there is one minor point I can't understand. When a local is captured it gets allocated in co_cellvars and is accessed with (LOAD|STORE)_DEREF, and this is clear. However when a local is coming from a parameter it gets ALSO allocated in co_varnames even if the local slot apparently is not accesible because *_FAST opcodes are not generated. Is there a technical reason for this? It happens in CPython 2, 3 and even in PyPy... Andrea Griffini ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why are captured parameters also listed in co_varnames?
2013/7/2 Andrea Griffini > I'm trying to understand how CPython implements closure variable capture > and there is one minor point I can't understand. > > When a local is captured it gets allocated in co_cellvars and is accessed > with (LOAD|STORE)_DEREF, and this is clear. > However when a local is coming from a parameter it gets ALSO allocated in > co_varnames even if the local slot apparently is not accesible because > *_FAST opcodes are not generated. > > Is there a technical reason for this? It happens in CPython 2, 3 and even > in PyPy... > > co_varnames is also used in error messages, for example in the following code: >>> def f(): ... def g(): ... x ... print x ... x = 1 ... return g ... >>> f() UnboundLocalError: local variable 'x' referenced before assignment This is also needed when x is a parameter of f(), for inspect.signature of course, but also because in python3 you can "del x". -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Hooking into super() attribute resolution
Hi, Below is a very preliminary draft PEP for adding a special method that can be used to hook into the attribute resolution process of the super object. The primary usecase for using this special method are classes that perform custom logic in their __getattribute__ method, where the default behavior of super (peekin the the class __dict__) is not appropriate. The primary reason I wrote this proposal is PyObjC: it dynamicly looks up methods in its __getattribute__ and caches the result in the class __dict__, because of this super() will often not work correctly and therefore I'm currently shipping a custom subclass of super() that basicly contains an in-line implementation of the hook that would be used by PyObjC. I have a partial implementation of the hook system in issue 18181 and a PyObjC patch that uses it. The implementation currently does not contain tests, and I'm sure that I'll find edge cases that I haven't thought about yet when I add tests. Ronald PEP: TODO Title: Hooking into super attribute resolution Version: $Revision$ Last-Modified: $Date$ Author: Ronald Oussoren Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 12-Jun-2013 Post-History: 2-Jul-2013 Abstract In current python releases the attribute resolution of the `super class`_ peeks in the ``__dict__`` attribute of classes on the MRO to look for attributes. This PEP introduces a hook that classes can use to override that behavior for specific classes. Rationale = Peeking in the class ``__dict__`` works for regular classes, but can cause problems when a class dynamicly looks up attributes in a ``__getattribute__`` method. The new hook makes it possible to introduce the same customization for attribute lookup through the `super class`_. The superclass attribute lookup hook In C code - A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. The ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, otherwise it will peek in the class ``tp_dict``. The slot has the following prototype:: PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, PyObject* name); The function should perform attribute lookup for *name*, but only looking in type *tp* (which will be one of the types on the MRO for *self*) and without looking in the instance *__dict__*. The function returns ``NULL`` when the attribute cannot be found, and raises and exception. Exception other than ``AttributeError`` will cause failure of super's attribute resolution. In Python code -- A Python class can contain a definition for a method ``__getattribute_super__`` with the following prototype:: def __getattribute_super__(self, cls, name): pass The method should perform attribute lookup for *name* on instance *self* while only looking at *cls* (it should not look in super classes or the instance *__dict__* Alternative proposals - Reuse ``tp_getattro`` . It would be nice to avoid adding a new slot, thus keeping the API simpler and easier to understand. A comment on `Issue 18181`_ asked about reusing the ``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all methods along the MRO. AFAIK that won't work because ``tp_getattro`` will look in the instance ``__dict__`` before it tries to resolve attributes using classes in the MRO. This would mean that using ``tp_getattro`` instead of peeking the class dictionaries changes the semantics of the `super class`_. Open Issues === * The names of the new slot and magic method are far from settled. * I'm not too happy with the prototype for the new hook. * Should ``__getattribute_super__`` be a class method instead? References == * `Issue 18181`_ contains a prototype implementation The prototype uses different names than this proposal. Copyright = This document has been placed in the public domain. .. _`Issue 18181`: http://bugs.python.org/issue18181 .. _`super class`: http://docs.python.org/3/library/functions.html?highlight=super#super ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Issue 18312 "fix" broke buildbots
I'm not sure how my change broke the buildbots, but apparently it did. I need to run to a meeting now, but I can roll this back in the next few hours. If someone else wants to roll it back before I get to it, feel free. Sorry about the problem. I tested it locally, I'm not sure how the buildbots are affected. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 18312 "fix" broke buildbots
On Tue, 02 Jul 2013 09:52:56 -0400, "Eric V. Smith" wrote: > I'm not sure how my change broke the buildbots, but apparently it did. I > need to run to a meeting now, but I can roll this back in the next few > hours. > > If someone else wants to roll it back before I get to it, feel free. > > Sorry about the problem. I tested it locally, I'm not sure how the > buildbots are affected. I'm no longer sure it was your patch. Anyone who wants to look at the buildbots please do: http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030 http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742 http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522 http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/699 http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2125 http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278 http://buildbot.python.org/all/builders/x86%20XP-4%203.3/builds/832 I haven't looked at all of these. The blamelist cites Richard Oudkerk, but at least one of the logs I looked at was about @test files, in a number of different tests. I don't really have time to look at it firther right now either, I've got a client with an issue that needs solved asap... --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue 18312 "fix" broke buildbots
On 07/02/2013 10:33 AM, R. David Murray wrote: > On Tue, 02 Jul 2013 09:52:56 -0400, "Eric V. Smith" > wrote: >> I'm not sure how my change broke the buildbots, but apparently it did. I >> need to run to a meeting now, but I can roll this back in the next few >> hours. >> >> If someone else wants to roll it back before I get to it, feel free. >> >> Sorry about the problem. I tested it locally, I'm not sure how the >> buildbots are affected. > > I'm no longer sure it was your patch. Anyone who wants to look > at the buildbots please do: > > http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030 > http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742 > http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522 > http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%20dtrace%2Bclang%203.3/builds/699 > http://buildbot.python.org/all/builders/x86%20Windows7%202.7/builds/2125 > http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278 > http://buildbot.python.org/all/builders/x86%20XP-4%203.3/builds/832 > > I haven't looked at all of these. The blamelist cites Richard Oudkerk, > but at least one of the logs I looked at was about @test files, in > a number of different tests. > > I don't really have time to look at it firther right now either, I've > got a client with an issue that needs solved asap... I've looked at a few of these, and they don't seem related to my change. I'll review it some more this afternoon. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Solving the import-deadlock case
Hello everyone, I'd like to bring your attention to this issue, since it touches the fundamentals of python's import workflow: http://bugs.python.org/issue17716 /I've tried to post it on the python-import ML for weeks, but it must still be blocked somewhere in a moderation queue, so here I come ^^/ TLDR version: because of the way import current works, if importing a package "temporarily" fails whereas importing one of its children succeeded, we reach an unusable state, all subsequent attempts at importing that package will fail if a "from...import" is used somewhere. Typically, it makes a web worker broken, even though the typical behaviour of such process woudl be to retry loading, again and again, the failing view. I agree that a module loading should be, as much as possible, "side effects free", and thus shouldn't have temporary errors. But well, in practice, module loading is typically the time where process-wide initialization are done (modifying sys.path, os.environ, instantiating connection or thread pools, registering atexit handler, starting maintenance threads...), so that case has chances to happen at a moment or another, especially if accesses to filesystem or network (SQL...) are done at module loading, due to the lack of initialization system at upper levels. That's why I propose modifying the behaviour of module import, so that submodules are deleted as well when a parent module import fails. True, it means they will be reloaded as well when importing the parent will start again, but anyway we already have a "double execution" problem with the reloading of the parent module, so it shouldn't make a big difference. The only other solution I'd see would be to SYSTEMATICALLY perform name (re)binding when processing a from...import statement, to recover from the previously failed initialization. Dunno if it's a good idea. On a (separate but related) topic, to be safer on module reimports or reloadings, it could be interesting to add some "idempotency" to common initialization tasks ; for example the "atexit" registration system, wouldn't it be worth adding a boolean flag to explicitely skip registration if a callable with same fully qualified name is already registered. Do you have opinions on these subjects ? thanks, regards, Pascal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Solving the import-deadlock case
On Tue, 02 Jul 2013 20:31:48 +0200, Pascal Chambon wrote: > I agree that a module loading should be, as much as possible, "side > effects free", and thus shouldn't have temporary errors. But well, in > practice, module loading is typically the time where process-wide > initialization are done (modifying sys.path, os.environ, instantiating > connection or thread pools, registering atexit handler, starting > maintenance threads...), so that case has chances to happen at a moment > or another, especially if accesses to filesystem or network (SQL...) are > done at module loading, due to the lack of initialization system at > upper levels. There may well be a bug that could be/should be fixed here, but...it seems to me that other than the sys.path modifications, doing any of that at module import time has a strong code smell. --David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Solving the import-deadlock case
On 3 Jul 2013 05:44, "R. David Murray" wrote: > > On Tue, 02 Jul 2013 20:31:48 +0200, Pascal Chambon wrote: > > I agree that a module loading should be, as much as possible, "side > > effects free", and thus shouldn't have temporary errors. But well, in > > practice, module loading is typically the time where process-wide > > initialization are done (modifying sys.path, os.environ, instantiating > > connection or thread pools, registering atexit handler, starting > > maintenance threads...), so that case has chances to happen at a moment > > or another, especially if accesses to filesystem or network (SQL...) are > > done at module loading, due to the lack of initialization system at > > upper levels. > > There may well be a bug that could be/should be fixed here, but...it > seems to me that other than the sys.path modifications, doing any of that > at module import time has a strong code smell. Unfortunately it's one of those "Your code is dubious, but so many people do it anyway we should handle it better than we do" cases. We could also be a lot more emphatic about "import side effects are what marks the boundary between a library and a framework. To stay on the library side of that fence provide a 'start' or 'configure' function instead of doing things implicitly on import". Heck, even *defining* library, framework and application would be a good thing. OTOH, it's hard to find motivation to work on improving the handling of things you think people shouldn't be doing in the first place (that's one of the reasons circular import handling has never been made more consistent). (That's not to dismiss the work Pascal's already done - just pointing out why it may sometimes feel like it's difficult to get interest and feedback on things like this). Cheers, Nick. > > --David > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Solving the import-deadlock case
On 3 Jul 2013 04:34, "Pascal Chambon" wrote: > > Hello everyone, > > I'd like to bring your attention to this issue, since it touches the fundamentals of python's import workflow: > http://bugs.python.org/issue17716 > > I've tried to post it on the python-import ML for weeks, but it must still be blocked somewhere in a moderation queue, so here I come ^^ > > TLDR version: because of the way import current works, if importing a package "temporarily" fails whereas importing one of its children succeeded, we reach an unusable state, all subsequent attempts at importing that package will fail if a "from...import" is used somewhere. Typically, it makes a web worker broken, even though the typical behaviour of such process woudl be to retry loading, again and again, the failing view. > > I agree that a module loading should be, as much as possible, "side effects free", and thus shouldn't have temporary errors. But well, in practice, module loading is typically the time where process-wide initialization are done (modifying sys.path, os.environ, instantiating connection or thread pools, registering atexit handler, starting maintenance threads...), so that case has chances to happen at a moment or another, especially if accesses to filesystem or network (SQL...) are done at module loading, due to the lack of initialization system at upper levels. > > That's why I propose modifying the behaviour of module import, so that submodules are deleted as well when a parent module import fails. True, it means they will be reloaded as well when importing the parent will start again, but anyway we already have a "double execution" problem with the reloading of the parent module, so it shouldn't make a big difference. > The only other solution I'd see would be to SYSTEMATICALLY perform name (re)binding when processing a from...import statement, to recover from the previously failed initialization. Dunno if it's a good idea. > > On a (separate but related) topic, to be safer on module reimports or reloadings, it could be interesting to add some "idempotency" to common initialization tasks ; for example the "atexit" registration system, wouldn't it be worth adding a boolean flag to explicitely skip registration if a callable with same fully qualified name is already registered. > > Do you have opinions on these subjects ? Back on topic... As I stated on the issue, I think purging the whole subtree when a package implicitly imports child modules is the least bad of the available options, and better than leaving the child modules in place in violation of the "all parent packages can be assumed to be in sys.modules" invariant (which is what we do now). Cheers, Nick. > > thanks, > regards, > Pascal > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hooking into super() attribute resolution
No time to read the PEP in detail but the motivation sound reasonable. --Guido van Rossum (sent from Android phone) On Jul 2, 2013 4:53 AM, "Ronald Oussoren" wrote: > Hi, > > Below is a very preliminary draft PEP for adding a special method that can > be used to hook into the attribute resolution process of the super object. > > The primary usecase for using this special method are classes that perform > custom logic in their __getattribute__ method, where the default behavior > of super (peekin the the class __dict__) is not appropriate. The primary > reason I wrote this proposal is PyObjC: it dynamicly looks up methods in > its __getattribute__ and caches the result in the class __dict__, because > of this super() will often not work correctly and therefore I'm currently > shipping a custom subclass of super() that basicly contains an in-line > implementation of the hook that would be used by PyObjC. > > I have a partial implementation of the hook system in issue 18181 and a > PyObjC patch that uses it. The implementation currently does not contain > tests, and I'm sure that I'll find edge cases that I haven't thought about > yet when I add tests. > > Ronald > > > > > PEP: TODO > Title: Hooking into super attribute resolution > Version: $Revision$ > Last-Modified: $Date$ > Author: Ronald Oussoren > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 12-Jun-2013 > Post-History: 2-Jul-2013 > > > Abstract > > > In current python releases the attribute resolution of the `super class`_ > peeks in the ``__dict__`` attribute of classes on the MRO to look > for attributes. This PEP introduces a hook that classes can use > to override that behavior for specific classes. > > > Rationale > = > > Peeking in the class ``__dict__`` works for regular classes, but can > cause problems when a class dynamicly looks up attributes in a > ``__getattribute__`` method. > > The new hook makes it possible to introduce the same customization for > attribute lookup through the `super class`_. > > > The superclass attribute lookup hook > > > In C code > - > > A new slot ``tp_getattro_super`` is added to the ``PyTypeObject`` struct. > The > ``tp_getattro`` slot for super will call this slot when it is not ``NULL``, > otherwise it will peek in the class ``tp_dict``. > > The slot has the following prototype:: > > PyObject* (*getattrosuperfunc)(PyTypeObject* tp, PyObject* self, > PyObject* name); > > The function should perform attribute lookup for *name*, but only looking > in > type *tp* (which will be one of the types on the MRO for *self*) and > without looking > in the instance *__dict__*. > > The function returns ``NULL`` when the attribute cannot be found, and > raises and > exception. Exception other than ``AttributeError`` will cause failure of > super's > attribute resolution. > > > In Python code > -- > > A Python class can contain a definition for a method > ``__getattribute_super__`` with > the following prototype:: > >def __getattribute_super__(self, cls, name): pass > > The method should perform attribute lookup for *name* on instance *self* > while only > looking at *cls* (it should not look in super classes or the instance > *__dict__* > > > Alternative proposals > - > > Reuse ``tp_getattro`` > . > > It would be nice to avoid adding a new slot, thus keeping the API simpler > and easier > to understand. A comment on `Issue 18181`_ asked about reusing the > ``tp_getattro`` slot, > that is super could call the ``tp_getattro`` slot of all methods along the > MRO. > > AFAIK that won't work because ``tp_getattro`` will look in the instance > ``__dict__`` before > it tries to resolve attributes using classes in the MRO. This would mean > that using > ``tp_getattro`` instead of peeking the class dictionaries changes the > semantics of > the `super class`_. > > > Open Issues > === > > * The names of the new slot and magic method are far from settled. > > * I'm not too happy with the prototype for the new hook. > > * Should ``__getattribute_super__`` be a class method instead? > > > References > == > > * `Issue 18181`_ contains a prototype implementation > > The prototype uses different names than this proposal. > > > Copyright > = > > This document has been placed in the public domain. > > .. _`Issue 18181`: http://bugs.python.org/issue18181 > > .. _`super class`: > http://docs.python.org/3/library/functions.html?highlight=super#super > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/ma
Re: [Python-Dev] Issue 18312 "fix" broke buildbots
"Eric V. Smith" writes: >> http://buildbot.python.org/all/builders/x86%20Tiger%202.7/builds/2030 >> http://buildbot.python.org/all/builders/x86%20Tiger%203.3/builds/742 >> http://buildbot.python.org/all/builders/x86%20Tiger%203.x/builds/6522 >> http://buildbot.python.org/all/builders/bolen-dmg-3.x/builds/278 (...) > I've looked at a few of these, and they don't seem related to my change. > I'll review it some more this afternoon. The above set (which are all on the x86 Tiger buildbot) appear due to some sort of filesystem corruption on the buildbot machine, so it appears that the timing of the failures and particular changesets is coincidental. I've repaired the filesystem and the disk passes a surface scan, so hopefully it'll be cleaned up. I've also re-queued the most recent builds in each branch. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
