[Python-Dev] About dictionary lookup caching
I'm experimenting with a patch for dictionary lookup caching, the main idea being avoiding the lookup of a constant (co_names) in a dictionary if the dictionary didn't change since last lookup. Currently the cache is held in a structure that is parallel to co_names (the LOAD_GLOBAL opcode uses oparg to access the cache slot) and this is wasteful if there are co_names entries that are not used for global lookups. Probably I could act at the code object creation time to sort names so that ones used by LOAD_GLOBAL are at the beginning of co_names but this would require inspecting and hacking the already generated opcode. Another case that would IMO be worth optimizing is the LOAD_ATTR lookup when it's done after a LOAD_GLOBAL (I suspect that in many cases the element being searched will be a module so caching would be simple to implement; for general objects things are way more complex and I see no simple solution for caching dictionary lookups). My opinion is that it would be by far better to do this ordering of co_names at compile time but I've no idea where to look for trying to make such a change. Can someone please point me in the right direction ? Andrea ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] About dictionary lookup caching
On 12/19/06, Andrea Griffini <[EMAIL PROTECTED]> wrote: I'm experimenting with a patch for dictionary lookup caching, the main idea being avoiding the lookup of a constant (co_names) in a dictionary if the dictionary didn't change since last lookup. Currently the cache is held in a structure that is parallel to co_names (the LOAD_GLOBAL opcode uses oparg to access the cache slot) and this is wasteful if there are co_names entries that are not used for global lookups. Probably I could act at the code object creation time to sort names so that ones used by LOAD_GLOBAL are at the beginning of co_names but this would require inspecting and hacking the already generated opcode. Another case that would IMO be worth optimizing is the LOAD_ATTR lookup when it's done after a LOAD_GLOBAL (I suspect that in many cases the element being searched will be a module so caching would be simple to implement; for general objects things are way more complex and I see no simple solution for caching dictionary lookups). My opinion is that it would be by far better to do this ordering of co_names at compile time but I've no idea where to look for trying to make such a change. Can someone please point me in the right direction ? For guidance on how the compiler works, see PEP 339 ( http://www.python.org/dev/peps/pep-0339). -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] About dictionary lookup caching
Andrea Griffini schrieb: > My opinion is that it would be by far better to do this ordering > of co_names at compile time but I've no idea where to look > for trying to make such a change. > > Can someone please point me in the right direction ? It's all in Python/compiler.c. The names list is created in dict_keys_inorder, where the dictionary keys are supposed to be tuples (name, str), and the values are the variable number. The dictionaries are filled through compiler_add_o, which looks up whether there is already a name in the table (and if so returns its number), or else adds a new entry, using len(dict) as the new number. As you can see, this inherently adds the names in the order in which they are encountered. I can see a number of choices: 1. make a separate list for names loaded through LOAD_GLOBAL. This would be an API change (and affect reflection and whatnot), but it would likely be the algorithmically easiest way 2. make a pass through the AST before compilation starts, to already add all globals to the names dictionary (while you are at it, you might as well fill all symbol tables in that pass). You can then renumber them as you please, and only invoke byte code generation afterwards. 3. try fitting this into the peephole optimizer. You'd make a peephole pass which reindexes co_names, and then run through the byte code to update all opargs. Be prepared to have this pass fail, in case you'd need an extended arg, but don't have space for it. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] classes and cell variables question
to my understanding of the object model, the code of snippet 1
and snippet 2 should be equivalent. a class is just a "special function"
that returns its locals automatically and passes them to the metaclass
constructor:
--- snippet 1 ---
class foo(object):
x = 5
def f(self):
print "f", x
--- snippet 2 ---
def bar():
x = 5
def g(self):
print "g", x
return locals()
barcls = type("barcls", (object,), bar())
but there's one big difference. classes don't create cell variables
to hold bound external variables. the "bar" version works, because
"x" is a bound cell variable, but the "foo" version fails, as it attempts
to access "x" as a global.
.>>> barcls().g()
g 5
.>>> foo().f()
f
Traceback (most recent call last):
File "", line 1, in
File "", line 4, in f
NameError: global name 'x' is not defined
for reference, i attached the code of all four functions below.
my question is, how come classes don't create cell variables, like
normal functions? was this done on purpose? does it have to
do with inheritance? if so, what's wrong with my "bar" version?
[1]
# code of class foo
# 2 0 LOAD_NAME0 (__name__)
# 3 STORE_NAME 1 (__module__)
# 3 6 LOAD_CONST 0 (5)
# 9 STORE_NAME 2 (x)
#
# 4 12 LOAD_CONST 1 (", line 4>)
# 15 MAKE_FUNCTION0
# 18 STORE_NAME 3 (f)
# 21 LOAD_LOCALS
# 22 RETURN_VALUE
[2]
# code of foo.f:
# 5 0 LOAD_CONST 1 ('f')
# 3 PRINT_ITEM
# 4 LOAD_GLOBAL 0 (x)
# 7 PRINT_ITEM
# 8 PRINT_NEWLINE
# 9 LOAD_CONST 0 (None)
# 12 RETURN_VALUE
[3]
# code of bar:
# 2 0 LOAD_CONST 1 (5)
# 3 STORE_DEREF 0 (x)
#
# 3 6 LOAD_CLOSURE 0 (x)
# 9 BUILD_TUPLE 1
# 12 LOAD_CONST 2 (", line 3>)
# 15 MAKE_CLOSURE 0
# 18 STORE_FAST 0 (g)
#
# 5 21 LOAD_GLOBAL 0 (locals)
# 24 CALL_FUNCTION0
# 27 RETURN_VALUE
[4]
# code of bar.g:
# 4 0 LOAD_CONST 1 ('g')
# 3 PRINT_ITEM
# 4 LOAD_DEREF 0 (x)
# 7 PRINT_ITEM
# 8 PRINT_NEWLINE
# 9 LOAD_CONST 0 (None)
# 12 RETURN_VALUE
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] About dictionary lookup caching
[Andrea Griffini] > I'm experimenting with a patch for dictionary lookup caching, the > main idea being avoiding the lookup of a constant (co_names) in > a dictionary if the dictionary didn't change since last lookup. . . . > My opinion is that it would be by far better to do this ordering > of co_names at compile time It would be nice if you could avoid introducing a new invariant that would be have to be maintained by other code generation tools or bytecode hacks. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] About dictionary lookup caching
Raymond Hettinger schrieb: > If you can find some other design that doesn't depend on the ordering of > co_names, that would be nice; otherwise, we're adding a permanent > complication to the compiler and establishing a new invariant that would > have to be maintained by everyone hoping to generate or hack bytecodes. It wouldn't be a strict invariant, but instead, breaking it would mean that the memory consumption goes up somewhat. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] classes and cell variables question
On 12/19/06, tomer filiba <[EMAIL PROTECTED]> wrote:
> to my understanding of the object model, the code of snippet 1
> and snippet 2 should be equivalent. a class is just a "special function"
> that returns its locals automatically and passes them to the metaclass
> constructor:
>
> --- snippet 1 ---
> class foo(object):
> x = 5
> def f(self):
> print "f", x
>
> --- snippet 2 ---
> def bar():
> x = 5
> def g(self):
> print "g", x
> return locals()
> barcls = type("barcls", (object,), bar())
>
> but there's one big difference. classes don't create cell variables
> to hold bound external variables. the "bar" version works, because
> "x" is a bound cell variable, but the "foo" version fails, as it attempts
> to access "x" as a global.
>
> .>>> barcls().g()
> g 5
>
> .>>> foo().f()
> f
> Traceback (most recent call last):
> File "", line 1, in
> File "", line 4, in f
> NameError: global name 'x' is not defined
>
> for reference, i attached the code of all four functions below.
>
> my question is, how come classes don't create cell variables, like
> normal functions? was this done on purpose? does it have to
> do with inheritance? if so, what's wrong with my "bar" version?
I did this very much on purpose, although I have to think a bit to
remember the reason. I think I didn't want "accidental Java code"
where one would define a class variable and reference it from a method
without prefixing it with self or the class name and it would
accidentally work, but it would be very unpythonic. So I rigged it so
that it wouldn't work.
> [1]
> # code of class foo
>
> # 2 0 LOAD_NAME0 (__name__)
> # 3 STORE_NAME 1 (__module__)
> # 3 6 LOAD_CONST 0 (5)
> # 9 STORE_NAME 2 (x)
> #
> # 4 12 LOAD_CONST 1 ( 009E5608, file "", line 4>)
> # 15 MAKE_FUNCTION0
> # 18 STORE_NAME 3 (f)
> # 21 LOAD_LOCALS
> # 22 RETURN_VALUE
>
> [2]
> # code of foo.f:
>
> # 5 0 LOAD_CONST 1 ('f')
> # 3 PRINT_ITEM
> # 4 LOAD_GLOBAL 0 (x)
> # 7 PRINT_ITEM
> # 8 PRINT_NEWLINE
> # 9 LOAD_CONST 0 (None)
> # 12 RETURN_VALUE
>
> [3]
> # code of bar:
>
> # 2 0 LOAD_CONST 1 (5)
> # 3 STORE_DEREF 0 (x)
> #
> # 3 6 LOAD_CLOSURE 0 (x)
> # 9 BUILD_TUPLE 1
> # 12 LOAD_CONST 2 ( 009F6698, file "", line 3>)
> # 15 MAKE_CLOSURE 0
> # 18 STORE_FAST 0 (g)
> #
> # 5 21 LOAD_GLOBAL 0 (locals)
> # 24 CALL_FUNCTION0
> # 27 RETURN_VALUE
>
> [4]
> # code of bar.g:
>
> # 4 0 LOAD_CONST 1 ('g')
> # 3 PRINT_ITEM
> # 4 LOAD_DEREF 0 (x)
> # 7 PRINT_ITEM
> # 8 PRINT_NEWLINE
> # 9 LOAD_CONST 0 (None)
> # 12 RETURN_VALUE
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] classes and cell variables question
tomer filiba schrieb: > my question is, how come classes don't create cell variables, like > normal functions? Not sure what you mean by "how come"? Why is the implementation reacting as it is? Because the body of class is compiled as a global code fragment, not as a nested one. Or why is the implementation the way it is? For several reasons, one being that classes predate nested functions. > was this done on purpose? Yes. Attributes defined inside a class are assumed to be accessed through attribute access *only*. So you write self.foo() to invoke a method, instead of invoking foo() Python treats methods and data attributes uniformly, so the same applies to data variables. > does it have to > do with inheritance? if so, what's wrong with my "bar" version? It also has to do with inheritance. If you do self.foo, it looks 1. in the object itself 2. in the class of the object 3. in the bases of the class of the object (recursively) It would be counter-intuitive if some things (i.e. things defined in the class itself) could be accessed directly, whereas other things (ie. attributes of the object itself, and things in the base classes) would need to be accessed through qualification. It would also be counter-intuitive if you find methods in an unqualified manner, but then can't call them because you didn't give a self argument. If you don't follow this reasoning, please write a counter-proposal so that people have something to shoot down. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] classes and cell variables question
> If you don't follow this reasoning, please write a counter-proposal > so that people have something to shoot down. ? i just wanted to be sure it was done on purpose, and what were the reasons for that. -tomer On 12/20/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > tomer filiba schrieb: > > my question is, how come classes don't create cell variables, like > > normal functions? > > Not sure what you mean by "how come"? Why is the implementation > reacting as it is? Because the body of class is compiled as > a global code fragment, not as a nested one. > Or why is the implementation the way it is? For several reasons, > one being that classes predate nested functions. > > > was this done on purpose? > > Yes. Attributes defined inside a class are assumed to be accessed > through attribute access *only*. So you write > > self.foo() > > to invoke a method, instead of invoking > > foo() > > Python treats methods and data attributes uniformly, so the > same applies to data variables. > > > does it have to > > do with inheritance? if so, what's wrong with my "bar" version? > > It also has to do with inheritance. If you do self.foo, it looks > 1. in the object itself > 2. in the class of the object > 3. in the bases of the class of the object (recursively) > > It would be counter-intuitive if some things (i.e. things defined > in the class itself) could be accessed directly, whereas other things > (ie. attributes of the object itself, and things in the base classes) > would need to be accessed through qualification. It would also be > counter-intuitive if you find methods in an unqualified manner, > but then can't call them because you didn't give a self argument. > > If you don't follow this reasoning, please write a counter-proposal > so that people have something to shoot down. > > Regards, > Martin > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
