[Python-Dev] cProfile with generator throwing
Hi. It seems that cProfile does not support throwing exceptions into
generators properly, when an external timer routine is used.
The problem is that _lsprof.c: ptrace_enter_call assumes that there
are no exceptions set when it is called, which is not true when the
generator frame is being gen_send_ex'd to send an exception into it
(Maybe you could say that only CallExternalTimer assumes this, but I
am not sure). This assumption causes its eventual call to
CallExternalTimer to discover that an error is set and assume that it
was caused by its own work (which it wasn't).
I am not sure what the right way to fix this is, so I cannot send a patch.
Here is a minimalist example to reproduce the bug:
>>> import cProfile
>>> import time
>>> p=cProfile.Profile(time.clock)
>>> def f():
... yield 1
...
>>> p.run("f().throw(Exception())")
Exception exceptions.Exception: Exception() in ignored
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/cProfile.py", line 135, in run
return self.runctx(cmd, dict, dict)
File "/usr/lib/python2.5/cProfile.py", line 140, in runctx
exec cmd in globals, locals
File "", line 1, in
File "", line 1, in f
SystemError: error return without exception set
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Instance variable access and descriptors
Hi. I was surprised to find in my profiling that instance variable access was pretty slow. I looked through the CPython code involved, and discovered something that really surprises me. Python, probably through the valid assumption that most attribute lookups go to the class, tries to look for the attribute in the class first, and in the instance, second. What Python currently does is quite peculiar! Here's a short description o PyObject_GenericGetAttr: A. Python looks for a descriptor in the _entire_ mro hierarchy (len(mro) class/type check and dict lookups). B. If Python found a descriptor and it has both get and set functions - it uses it to get the value and returns, skipping the next stage. C. If Python either did not find a descriptor, or found one that has no setter, it will try a lookup in the instance dict. D. If Python failed to find it in the instance, it will use the descriptor's getter, and if it has no getter it will use the descriptor itself. I believe the average costs of A are much higher than of C. Because there is just 1 instance dict to look through, and it is also typically smaller than the class dicts (in rare cases of worse-case timings of hash lookups), while there are len(mro) dicts to look for a descriptor in. This means that for simple instance variable lookups, Python is paying the full mro lookup price! I believe that this should be changed, so that Python first looks for the attribute in the instance's dict and only then through the dict's mro. This will have the following effects: A. It will break code that uses instance.__dict__['var'] directly, when 'var' exists as a property with a __set__ in the class. I believe this is not significant. B. It will simplify getattr's semantics. Python should _always_ give precedence to instance attributes over class ones, rather than have very weird special-cases (such as a property with a __set__). C. It will greatly speed up instance variable access, especially when the class has a large mro. I think obviously the code breakage is the worst problem. This could probably be addressed by a transition version in which Python warns about any instance attributes that existed in the mro as descriptors as well. What do you think? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Frame zombies
I was just looking through the code that handles frames (as part of my current effort to determine how to improve on CPython's performance), when I noticed the freelist/zombie mechanism for frame allocation handling. While the zombie mechanism seems like a nice optimization, I believe there can be a couple of improvements. Currently, each code object has a zombie frame that last executed it. This zombie is reused when that code object is re-executed in a frame. When a frame is released, it is reassigned as the zombie of the code, and iff the code object already has a zombie assigned to it, it places the frame in the freelist. If I understand correctly, this means, that the "freelist" is actually only ever used for recursive-call frames that were released. It also means that these released frames will be reassigned to other code objects in the future - in which case they will be reallocated, perhaps unnecessarily. "freelist" is just temporary storage for released recursive calls. A program with no recursions will always have an empty freelist. What is bounding the memory consumption of this mechanism is a limit on the number of frames in the freelist (and the fact that there is a limited number of code objects, each of which may have an additional zombie frame). I believe a better way to implement this mechanism: A. Replace the co_zombie frame with a co_zombie_freelist. B. Remove the global freelist. In other words, have a free list for each code object, rather than one-per-code-object and a freelist. This can be memory-bound by limiting the freelist size in each code object. This can be use a bit more memory if a recursion is called just once - and then discarded (waste for example 10 frames instead of 1), but can save a lot of realloc calls when there is more than one recursion used in the same program. It is also possible to substantially increase the number of frames stored per code-object, and then use some kind of more sophisticated aging mechanism on the zombie freelists to periodically get rid of unused freelists. That kind of mechanism would mean that even in the case of recursive calls, virtually all frame allocs are available from the freelist. I also believe this to be somewhat simpler, as there is only one concept (a zombie freelist) rather than 2 (a zombie code object and a freelist for recursive calls), and frames are never realloc'd, but only allocated. Should I make a patch? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Instance variable access and descriptors
I must be missing something, as I really see no reason to keep the existing semantics other than backwards compatibility (which can be achieved by introducing a __fastattr__ or such). Can you explain under which situations or find any example situation where the existing semantics are desirable? As for the mro cache - thanks for pointing it out - I think it can serve as a platform for another idea that in conjunction with psyco, can possibly speed up CPython very significantly (will create a thread about this soon). Please note that speeding up the mro-lookup solves only half of the problem (if it was solved - which it seems not to have been), the more important half of the problem remains, allow me to emphasize: ALL instance attribute accesses look up in both instance and class dicts, when it could look just in the instance dict. This is made worse by the fact that the class dict lookup is more expensive (with or without the mro cache). Some code that accesses a lot of instance attributes in an inner loop can easily be sped up by a factor of 2 or more (depending on the size of the mro). Eyal On 6/10/07, Kevin Jacobs <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> wrote: > I agree with Phillip with regard to the semantics. They are semantically > desirable. However, there is a patch to add a mro cache to speed up these > sorts of cases on the Python tracker, originally submitted by Armin Rigo. > He saw ~20% speedups, others see less. It is currently just sitting there > with no apparent activity. So if the overhead of mro lookups is that > bothersome, it may be well worth your time to review the patch. > > URL: > http://sourceforge.net/tracker/index.php?func=detail&aid=1700288&group_id=5470&atid=305470 > > -Kevin > > > > On 6/9/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > > > > At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote: > > >A. It will break code that uses instance.__dict__['var'] directly, > > >when 'var' exists as a property with a __set__ in the class. I believe > > >this is not significant. > > >B. It will simplify getattr's semantics. Python should _always_ give > > >precedence to instance attributes over class ones, rather than have > > >very weird special-cases (such as a property with a __set__). > > > > Actually, these are features that are both used and desirable; I've > > been using them both since Python 2.2 (i.e., for many years > > now). I'm -1 on removing these features from any version of Python, even > 3.0. > > > > > > >C. It will greatly speed up instance variable access, especially when > > >the class has a large mro. > > > > ...at the cost of slowing down access to properties and __slots__, by > > adding an *extra* dictionary lookup there. > > > > Note, by the way, that if you want to change attribute lookup > > semantics, you can always override __getattribute__ and make it work > > whatever way you like, without forcing everybody else to change *their* > code. > > > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jacobs%40bioinformed.com > > > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: Instance variable access and descriptors
On 6/10/07, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 12:23 AM 6/10/2007 +0300, Eyal Lotem wrote: > >A. It will break code that uses instance.__dict__['var'] directly, > >when 'var' exists as a property with a __set__ in the class. I believe > >this is not significant. > >B. It will simplify getattr's semantics. Python should _always_ give > >precedence to instance attributes over class ones, rather than have > >very weird special-cases (such as a property with a __set__). > > Actually, these are features that are both used and desirable; I've > been using them both since Python 2.2 (i.e., for many years > now). I'm -1 on removing these features from any version of Python, even 3.0. It is the same feature, actually, two sides of the same coin. Why do you use self.__dict__['propertyname'] when you can use self._propertyname? Why even call the first form, which is both longer and causes performance problems "a feature"? > >C. It will greatly speed up instance variable access, especially when > >the class has a large mro. > > ...at the cost of slowing down access to properties and __slots__, by > adding an *extra* dictionary lookup there. It will slow down access to properties - but that slowdown is insignificant: A. The vast majority of lookups are *NOT* of properties. They are the rare case and should not be the focus of optimization. B. Property access involves calling Python functions - which is heavier than a single dict lookup. C. The dict lookup to find the property in the __mro__ can involve many dicts (so in those cases adding a single dict lookup is not heavy). > Note, by the way, that if you want to change attribute lookup > semantics, you can always override __getattribute__ and make it work > whatever way you like, without forcing everybody else to change *their* code. If I write my own __getattribute__ I lose the performance benefit that I am after. I do agree that code shouldn't be broken, that's why a transitional that requires using __fastlookup__ can be used (Unfortunately, from __future__ cannot be used as it is not local to a module, but to a class hierarchy - unless one imports a feature from __future__ into a class). ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Frame zombies
The freelist currently serves as a good optimization of a special case of a recurring recursion. If the same code object (or one of the same size) is used for recursive calls repeatedly, the freelist will realloc-to-same-size (which probably has no serious cost) and thus the cost of allocating/deallocating frames was averted. I think that in general, the complexity of a sophisticated and efficient aging mechanism is not worth it just to optimize recursive calls. The only question is whether it is truly a memory problem, if using, say, up-to 50 frames per code object? Note that _only_ recursions will have more than 1 frame attached. How many recursions are used and then discarded? How slow is it to constantly malloc/free frames in a recursion? My proposal will accelerate the following example: def f(x, y): if 0 == x: return f(x-1, y) def g(x): if 0 == x: return g(x-1) while True: f(100, 100) g(100) The current implementation will work well with the following: while True: f(100, 100) But removing freelist altogether will not work well with any type of recursion. Eyal On 6/10/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > Should I make a patch? > > -1. This could consume quite a lot of memory, and I doubt > that the speed improvement would be significant. Instead, > I would check whether performance could be improved by > just dropping the freelist. Looking at the code, I see > that it performs a realloc (!) of the frame object if > the one it found is too small. That surely must be > expensive, and should be replaced with a free/malloc pair > instead. > > I'd be curious to see whether malloc on today's systems > is still so slow as to justify a free list. If it is, > I would propose to create multiple free lists per size > classes, e.g. for frames with 10, 20, 30, etc. variables, > rather than having free lists per code object. > > Regards, > Martin > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/eyal.lotem%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Frame zombies
On 6/10/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > > Note that _only_ recursions will have more than 1 frame attached. > > That's not true; in the presence of threads, the same method > may also be invoked more than one time simultaneously. Yes, I have missed that, and realized that I missed that myself a bit later. I guess I can rationalize that with the fact that I myself tend to avoid threads. > > But removing freelist altogether will not work well with any type of > > recursion. > > How do you know that? Did you measure the time? On what system? > What were the results? > > Performance optimization without measuring is just unacceptable. I agree, I may have used the wrong tone above. Removing the freelist will probably either not have a significant effect (at worst, its adding very little work of maintaining it), or improve recursions and functions that tend to be running simultaniously in multiple threads (as in those cases the realloc will not actually resize the frame, and mallocs/free will indeed be saved). But do note my corrected tone (I said "probably" :-) - and anyone is welcome to try removing it and see if they get a performance benefit. The fact threading also causes the same code object to be used in multiple frames makes everything a little less predictable and may mean that having a larger-than-1 number of frames associated with each code object may indeed yield a performance benefit. I am not sure how to benchmark such modifications. Is there any benchmark that includes threaded use of the same functions in typical use cases? > Regards, > Martin > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/eyal.lotem%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Question about dictobject.c:lookdict_string
My question is specifically regarding the transition back from lookdict_string (the initial value) to the general lookdict. Currently, when a string-only dict is trying to look up any non-string, it reverts back to a general lookdict. Wouldn't it be better (especially in the more important case of a string-key-only dict), to revert to the generic lookdict when a non-string is inserted to the dict, rather than when one is being searched? This seems to me as it would shift this (admittedly very slight) performance cost of a type ptr comparison from the read-access, to write-access on all dicts (which means insertions of new keys in non-string-only dicts may pay for another check, or that the lookdict funcptr will be replaced by two funcptrs so that a different insertion func on string-only dicts is used too [was tempted to say vtable here, but that would add another dereference to lookups]). It would also have the slight benefit of speeding up non-string lookups in string-only dicts. This does not seem like a significant issue, but as I know a lot of effort went into optimizing dicts, I was wondering if I am missing something here. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Question about dictobject.c:lookdict_string
On 6/11/07, Carl Friedrich Bolz <[EMAIL PROTECTED]> wrote: > Eyal Lotem wrote: > > My question is specifically regarding the transition back from > > lookdict_string (the initial value) to the general lookdict. > > > > Currently, when a string-only dict is trying to look up any > > non-string, it reverts back to a general lookdict. > > > > Wouldn't it be better (especially in the more important case of a > > string-key-only dict), to revert to the generic lookdict when a > > non-string is inserted to the dict, rather than when one is being > > searched? > [...] > > This does not seem like a significant issue, but as I know a lot of > > effort went into optimizing dicts, I was wondering if I am missing > > something here. > > Yes, you are: when doing a lookup with a non-string-key, that key could > be an instance of a class that has __hash__ and __eq__ implementations > that make the key compare equal to some string that is in the > dictionary. So you need to change to lookdict, otherwise that lookup > might fail. Ah, thanks for clarification. But doesn't it make sense to only revert that single lookup, and not modify the function ptr until the dict contains a non-string? Eyal ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Cycle collection enhancement idea
I see why a cycle that has multiple objects with a __del__ method is a problem. Once you call __del__ on one of the objects, its no longer usable by the others, and its not clear which order is correct. My question regards the case where a cycle of objects only has 1 object which has a __del__. I think a correct strategy to collect the entire cycle is the same one used on a single object. On a single object Python uses: 1. Temporarily revive object 2. Call __del__ 3. Unrevive object (if(--refcount == 0) then we're done), otherwise it was resurrected). We can apply this to the whole cycle: 1. Temporarily revive entire cycle (each of its objects) 2. Call __del__ 3. Unrevive the objects of the entire cycle (each of its objects). Step 1 will allow __del__ to run safely. Since there is only one __del__ in the cycle, it is not dangerous that its references will disappear from "under its feet". (Some code restructuring will probably be necessary, because of assumptions that are hard-coded into slot_tp_del and subtype_dealloc). I believe this enhancement is important, because: A. When using existing code -- you do not control whether its objects have a __del__. In my experience, a majority of these cases only have a single __del__-containing object in their cycles. B. Python's exit cleanup calls __del__ in the wrong order, and Python's runtime is full of cycles (Each global is a cycle, including the class objects themselves: class->dict->function->func_globals)). These cycles very often have only 1 __del__ method. Some examples of the problem posed by B: http://www.google.com/search?q=ignored+%22%27NoneType%27+object+has+no+attribute%22+%22__del__+of%22&btnG=Search Ugly workarounds exist even in the standard library [subprocess]: "def __del__(self, sys=sys):"). Example: import os class RunningFile(object): filename = '/tmp/running' def __init__(self): open(self.filename, 'wb') def __del__(self): os.unlink(self.filename) running_file = RunningFile() The deller object is in a cycle as described above [as well as the Deller class itself]. When Python exits, it could call deller.__del__() and then collect the cycle. But Python does the wrong thing here, and gets rid of the globals before calling __del__: Exception exceptions.AttributeError: "'NoneType' object has no attribute 'unlink'" in > ignored I believe applying the above enhancement would solve these problems. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cycle collection enhancement idea
On Sun, Jun 29, 2008 at 9:00 PM, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: >> As I explained above, it *is* part of a cycle: """including >> the class objects themselves: class->dict->function->func_globals""". > > Ah, right. I must have missed that explanation. > >> I know. I assumed Python does not rely on cyclic garbage collectioin >> for shutdown, because it wouldn't work, as _all_ globals that have >> *any* instance method will be part of a cycle, and any of them which >> have a __del__ will not be collected. > > No. The mechanism for cleaning modules at shutdown time predates cyclic > GC, and was not removed because "it wouldn't work". > > This specific issue certainly contributes to the fact that it doesn't > work, but there might be other problems as well (such as extension > modules holding onto objects participating in cycles). > >> I *mentioned* this workaround. What I propose is not a workaround but >> a solution. You wouldn't need to clean up module globals ad-hoc'ishly, >> because the cyclic collection would collect your object, even with its >> __del__. > > I don't think it solves the problem. You seem to be assuming that any > such cycle will contain only a single global object with an __del__. > However, as modules refer to each other, I very much doubt that is the > case. > >> Please read my entire mail before replying to it. Thanks! > > I really, really tried. I read it three times before replying. > > However, I found it really, really difficult to follow your writing, > as it was mixing problem statement and solution, so that I couldn't > tell what paragraph was about what. English is not my native language, > complicating communication further. Please accept my apologies. I apologize for my tone as well, it just seemed frustrating that all the replies seemed to completely ignore the core point I was trying to make and claim that the problem I was trying to solve was not a problem at all, but a behavior I was unaware of... Mixing another quote from another mail: > > That would be no worse than what happens now - but its still not > > perfect (__del__ ordering issues). Also, you would need to temporarily > > revive the cycles as mentioned above (to avoid accessibility of > > partially destructed objects). > > I don't quite understand what you mean by "revive cycles". There is > not need to revive any object in a cycle, as all objects are still alive > (or else they wouldn't be garbage). By "revive cycles", I mean make sure that they are referenced by an independent referrer (one that won't go away as part of the __del__ calling process). This is similar to how the tp_dealloc code increases the refcount (actually sets it to 1, because it was certainly 0 when entering the destructor) before calling the __del__ slot. Without reviving the object before calling its __del__ in the destructor, and without reviving the objects of a cycle before calling its __del__'s, the __del__ Pythonic code may be exposed to "dead objects" (refcount==0). Consider the cycle: a.x = b b.x = a Lets suppose the a object has a __del__. Lets assume each object in the cycle has a refcount of 1 (and the cycle should die). Now lets say this is a's __del__ code: def __del__(self): self.x = None Running it will set 'b's refcount to 0 and call its destructor, which will set 'a's refcount to 0 and also call its destructor. But its __del__ is currently running - so "self" must not have a refcount of 0. If you only incref on 'a' before calling __del__, then you are probably alright, as long as there is only one __del__. > Can you please elaborate? What would such __del__ ordering issues be? class A(object): def __del__(self): print self.x.attribute class B(object): def __del__(self): print "B is going down!" del self.attribute a = A() b = B() a.x = b b.attribute = 1 If you call b's __del__ first then a's __del__ will fail. If you call a's __del__ first, then all is well. Ofcourse you can create true cyclic dependencies that no order will work, and its pretty clear there is no way to deduce the right order anyway. This is what I mean by "ordering issues". > There could be a global barricade for calling __del__: you first call > all __del__s of existing objects, then set the barricade, and then > start breaking cycles. > This could even be done with the current approach to module clearing. Note that the __del__'s themselves may be breaking cycles and refcounts will go to 0 - unless you temporarily revive (incref) the entire cycle first. > I still don't understand what "revive the cycle" means. You will need to > incref the object for which you call __del__, that's all. Unless there are multiple __del__'s in the cycle. > > Regards, > Martin > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Security capabilities in Python
I would like to experiment with security based on Python references as security capabilities. Unfortunatly, there are several problems that make Python references invalid as capabilities: * There is no way to create secure proxies because there are no private attributes. * Lots of Python objects are reachable unnecessarily breaking the principle of least privelege (i.e: object.__subclasses__() etc.) I was wondering if any such effort has already begun or if there are other considerations making Python unusable as a capability platform? (Please cc the reply to my email) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Security capabilities in Python
It may be really hard to get it right, unless we are overlooking some simple solution.I disagree that we should "just use OS protections".The
reason I am interested in Pythonic protection is because it is so much
more powerful than OS protections. The capability model is
much more powerful than the ACL model used by all OS's these days, and
allows for interesting security concepts.What about implementing the facet in C? This could avoid the class of problems you have just mentioned.On Apr 9, 2005 2:02 PM, James Y Knight <[EMAIL PROTECTED]> wrote:> On Apr 9, 2005, at 5:37 PM, Ka-Ping Yee wrote:> > Let me know if you figure out how to defeat that.> > You can protect against this, too, but it does show that it's *really*> hard to get restricting code right...I'm of the opinion that it's not> really worth it -- you should just use OS protections.> > untrusted_module.py:> > class foostr(str):> def __eq__(self, other):>return True> > def have_at_it(immutable_facet, readonly_facet):>getattr(immutable_facet, foostr('append'))(5)>print immutable_facet> > James___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Early PEP draft (For Python 3000?)
I would like to re-suggest a suggestion I have made in the past, but with a mild difference, and a narrower scope. Name: Attribute access for all namespaces Rationale: globals() access is conceptually the same as setting the module's attributes but uses a different idiom (access of the dict directly). Also, locals() returns a dict, which implies it can affect the local scope, but quietly ignores changes. Using attribute access to access the local/global namespaces just as that is used in instance namespaces and other modules' namespaces, could reduce the mental footprint of Python. Method: All namespace accesses are attribute accesses, and not direct __dict__ accesses. Thus globals() is replaced by a "module" keyword (or "magic variable"?) that evaluates to the module object. Thus, reading/writing globals in module X, uses getattr/setattr on the module object, just like doing so in module Y would be. locals() would return be replaced by a function that returns the frame object (or a weaker equivalent of a frame object) of the currently running function. This object will represent the local namespace and will allow attribute getting/setting to read/write attributes. Or it can disallow attribute setting. Examples: global x ; x = 1 Replaced by: module.x = 1 or: globals()[x] = 1 Replaced by: setattr(module, x, 1) locals()['x'] = 1 # Quietly fails! Replaced by: frame.x = 1 # Raises error x = locals()[varname] Replaced by: x = getattr(frame, varname) Advantages: - Python becomes more consistent w.r.t namespacing and scopes. Disadvantages: - "module" is already possible by importing one's own module, but that is: * Confusing and unnecessarily requires naming one's self redundantly (Making renaming of the module a bit more difficult). * Not easily possible in a __main__/importable module. * No equivalent for locals() - Automatic script conversion may be difficult in some use cases of globals()/locals() ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Assignment to __class__ of module? (Autoloading? (Making Queue.Queue easier to use))
Why not lazily import modules by importing them when they are needed (i.e inside functions), and not in the top-level module scope? On 10/13/05, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 04:02 PM 10/13/2005 +0100, Michael Hudson wrote: > >Greg Ewing <[EMAIL PROTECTED]> writes: > > > > > Phillip J. Eby wrote: > > >> At 01:47 PM 10/13/2005 +1300, Greg Ewing wrote: > > >> > > >>> I'm trying to change the __class__ of a newly-imported > > >>> module to a subclass of types.ModuleType > > >> > > >> It happened in Python 2.3, actually. > > > > > > Is there a discussion anywhere about the reason this was > > > done? It would be useful if this capability could be > > > regained somehow without breaking things. > > > >Well, I think it's undesirable that you be able to do this to, e.g., > >strings. Modules are something of a greyer area, I guess. > > Actually, it's desirable to be *able* to do it for anything. But certainly > > for otherwise-immutable objects it can lead to aliasing issues. > > For mutable objects, it's *very* desirable, and I think the rules added in > 2.3 might have been overly strict, as they disallow you changing any > built-in type to a non built-in type, even if the allocator is the > same. It seems to me the safety check could perhaps be reduced to just > checking whether the old and new classes have the same tp_free. (Apart > from the layout and other inheritance-related checks, I mean.) > > (By the way, for an example use case other than modules, note that somebody > > wrote an "observables" package that could detect mutation of lists and > dictionaries in Python 2.2 using __class__ changes, which then became > useless as of Python 2.3.) > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/eyal.lotem%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Class decorators vs metaclasses
I have a few claims, some unrelated, and some built on top of each other. I would like to hear your responses as to which are convincing, which arne't, and why. I think that if these claims are true, Python 3000 should change quite a bit. A. Metaclass code is black magic and few understand how it works, while decorator code is mostly understandable, even by non-gurus. B. One of Decorators' most powerful features is that they can mixed-and-matched, which makes them very powerful for many purposes, while metaclasses are exclusive, and only one can be used. This is especially problematic as some classes may assume their subclasses must use their respective metaclasses. This means classdecorators are strictly more powerful than metaclasses, without cumbersome convertions between metaclass mechanisms and decorator mechanisms. C. Interesting uses of classdecorators are allowing super-calling without redundantly specifying the name of your class, or your superclass. D. Python seems to be incrementally adding power to the core language with these features, which is great, but it also causes significant overlapping of language features, which I believe is something to avoid when possible. If metaclasses are replaced with class decorators, then suddenly inheritence becomes a redundant feature. E. If inheritence is a redundant feature, it can be removed and an "inherit" class decorator can be used. This could also reduce all the __mro__ clutter from the language along with other complexities, into alternate implementations of the inherit classdecorator. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Class decorators vs metaclasses
On 11/5/05, Alex Martelli <[EMAIL PROTECTED]> wrote:
> On 11/4/05, Eyal Lotem <[EMAIL PROTECTED]> wrote:
> > I have a few claims, some unrelated, and some built on top of each
> > other. I would like to hear your responses as to which are
> > convincing, which arne't, and why. I think that if these claims are
> > true, Python 3000 should change quite a bit.
> >
> > A. Metaclass code is black magic and few understand how it works,
> > while decorator code is mostly understandable, even by non-gurus.
>
> I disagree. I've held many presentations and classes on both
> subjects, and while people may INITIALLY feel like metaclasses are
> black magic, as soon as I've explained it the fear dissipates. It all
> boils down do understanding that:
>
> class Name(Ba,Ses): <>
>
> means
>
> Name = suitable_metaclass('Name', (Ba,Ses), <>)
>
> which isn't any harder than understanding that
>
> @foo(bar)
> def baz(args): ...
>
> means
>
> def baz(args): ...
> baz = foo(bar)(baz)
I disagree again. My experience is that metaclass code is very hard to
understand. Especially when it starts doing non-trivial things, such
as using a base metaclass class that is parametrized by metaclass
attributes in its subclasses. Lookups of attributes in the base
metaclass methods is mind boggling (is it searching them in the base
metaclass, the subclass, the instance [which is the class]?). The
same code would be much easier to understand with class decorators.
> > B. One of Decorators' most powerful features is that they can
> > mixed-and-matched, which makes them very powerful for many purposes,
> > while metaclasses are exclusive, and only one can be used. This is
>
> Wrong. You can mix as many metaclasses as you wish, as long as
> they're properly coded for multiple inheritance (using super, etc) --
> just inherit from them all. This is reasonably easy to automate (see
> the last recipe in the 2nd ed of the Python Cookbook), too.
Multiple inheritence is an awful way to mix class fucntionalities
though. Lets take a simpler example. Most UT frameworks use a
TestCase base class they inherit from to implement setup, tearDown,
and then inherit from it again to implement the test itself. I argue
this is a weak approach, because then mixing/matching setups is
difficult. You would argue this is not the case, because of the
ability to multiply-inherit from test cases, but how easy is the
equivalent of:
@with_setup('blah')
@with_other_setup('bleh')
def my_test():
# the blah setup and bleh other setup are up and usable here,
# and will be "torn down" at the end of this test
The equivalent of this requires a lot more work and violating DRY.
Creating a specific function to multiply inherit from TestCases is a
possible solution, but it is much more conceptually complex, and needs
to be reimplemented in the next scenario (Metaclasses for example).
> > especially problematic as some classes may assume their subclasses
> > must use their respective metaclasses. This means classdecorators are
> > strictly more powerful than metaclasses, without cumbersome
> > convertions between metaclass mechanisms and decorator mechanisms.
>
> The assertion that classdecorators are strictly more powerful than
> custom metaclasses is simply false. How would you design
> classdecorator XXX so that
>
> @XXX
> class Foo: ...
>
> allows 'print Foo' to emit 'this is beautiful class Foo', for example?
> the str(Foo) implicit in print calls type(Foo).__str__(Foo), so you
> do need a custom type(Foo) -- which is all that is meant by "a custom
> metaclass"... a custom type whose instances are classes, that's all.
I would argue that this is not such a useful feature, as in that case
you can simply use a factory object instead of a class. If this
feature remains, that's fine, but the fact it allows for a weak form
of "decoration" of classes should not kill the concept of class
decorators.
The only reason of using metaclasses rather than factory objects, in
my experience, was that references to class objects are considered
different than references to factories (by pickle and deepcopy, and
maybe others) and that can be a useful feature. This feature can be
implemented in more readable means though.
> > C. Interesting uses of classdecorators are allowing super-calling
> > without redundantly specifying the name of your class, or your
> > superclass.
>
> Can you give an example?
@anotherclassdecorator
@supercallerclass
class MyClass(object):
@supercaller
def my_method(self, supcaller, x, y, z):
...
result = supcaller.my_method(x, y, z)
...
Could be nice to
