Re: [Cython] PEP 3135 -- New Super
Vitja Makarov, 05.07.2011 08:21:
I was thinking about implementing new super() with no arguments.
Please do :)
If you start working on it, please assign the ticket to you:
http://trac.cython.org/cython_trac/ticket/696
The problem is where to store __class__, I see two options here:
1. Add func_class member to CyFunction, this way __class__ will be
private and not visible for inner functions:
2. Put it into closure
The second option has the advantage of requiring the field only when
super() is used, whereas the first impacts all functions.
I would expect that programs commonly have a lot more functions than
specifically methods that use a no-argument call to super(), so this may
make a difference.
OTOH, not all methods have a closure, so creating one just to store the
"__class__" field is very wasteful, in terms of both runtime and memory
overhead. A lot more wasteful than paying 8 bytes of memory for each
function, with no additional time overhead.
And I don't think that __class__ should be use somewhere outside super()
Agreed. CPython simply uses a compile time heuristic ("is there a function
call to something global named 'super'?") when creating this field, so it's
strictly reserved for this use case.
BTW, I like the irony in the fact that CPython essentially gives Cython
semantics to the "super" builtin here, by (partially) evaluating it at
compile time.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PEP 3135 -- New Super
2011/7/5 Stefan Behnel :
> Vitja Makarov, 05.07.2011 08:21:
>>
>> I was thinking about implementing new super() with no arguments.
>
> Please do :)
>
> If you start working on it, please assign the ticket to you:
>
> http://trac.cython.org/cython_trac/ticket/696
>
Ok, I'll do this if I start.
>
>> The problem is where to store __class__, I see two options here:
>>
>> 1. Add func_class member to CyFunction, this way __class__ will be
>> private and not visible for inner functions:
>> 2. Put it into closure
>
> The second option has the advantage of requiring the field only when super()
> is used, whereas the first impacts all functions.
>
> I would expect that programs commonly have a lot more functions than
> specifically methods that use a no-argument call to super(), so this may
> make a difference.
>
So, now classes are created the following way:
class_dict = {}
class_dict.foo = foo_func
class = CreateClass(class_dict)
So after class is created I should check its dict for CyFunction
members (maybe only ones that actually require __class__)
and set __class__:
for value in class.__dict__.itervalues():
if isinstance(value, CyFunction) and value.func_class is WantClass:
value.func_class = class
> OTOH, not all methods have a closure, so creating one just to store the
> "__class__" field is very wasteful, in terms of both runtime and memory
> overhead. A lot more wasteful than paying 8 bytes of memory for each
> function, with no additional time overhead.
>
Going this way it only requires to initialize closure:
closure.func_class = class
Btw, first way requires cyfunction signature change, it would accept
cyfunction object as first argument.
This also could help to solve default args problem.
>
>> And I don't think that __class__ should be use somewhere outside super()
>
> Agreed. CPython simply uses a compile time heuristic ("is there a function
> call to something global named 'super'?") when creating this field, so it's
> strictly reserved for this use case.
>
> BTW, I like the irony in the fact that CPython essentially gives Cython
> semantics to the "super" builtin here, by (partially) evaluating it at
> compile time.
>
Yeah, I think Cython super with no args should be a little bit faster
then classic one.
--
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PEP 3135 -- New Super
Vitja Makarov, 05.07.2011 09:17:
2011/7/5 Stefan Behnel:
Vitja Makarov, 05.07.2011 08:21:
I was thinking about implementing new super() with no arguments.
http://trac.cython.org/cython_trac/ticket/696
The problem is where to store __class__, I see two options here:
1. Add func_class member to CyFunction, this way __class__ will be
private and not visible for inner functions:
2. Put it into closure
The second option has the advantage of requiring the field only when super()
is used, whereas the first impacts all functions.
I would expect that programs commonly have a lot more functions than
specifically methods that use a no-argument call to super(), so this may
make a difference.
So, now classes are created the following way:
class_dict = {}
class_dict.foo = foo_func
class = CreateClass(class_dict)
So after class is created I should check its dict for CyFunction
members (maybe only ones that actually require __class__)
and set __class__:
for value in class.__dict__.itervalues():
if isinstance(value, CyFunction) and value.func_class is WantClass:
value.func_class = class
Remember that no-args super() can only be used in functions that are
literally written inside of a class body, so we actually know at compile
time which functions need this field. We can thus do better than a generic
loop over all fields. We even have the function object pointers directly
available in the module init function where we create the class body.
BTW, we also need a way to make this work for cdef classes. No idea how
different that would be.
OTOH, not all methods have a closure, so creating one just to store the
"__class__" field is very wasteful, in terms of both runtime and memory
overhead. A lot more wasteful than paying 8 bytes of memory for each
function, with no additional time overhead.
Going this way it only requires to initialize closure:
Yes, and that's costly.
Btw, first way requires cyfunction signature change, it would accept
cyfunction object as first argument.
We currently pass the binding (i.e. owning) object, right?
This also could help to solve default args problem.
And potentially other problems, too. Think of heap allocated modules, for
example.
http://trac.cython.org/cython_trac/ticket/173
http://trac.cython.org/cython_trac/ticket/218
Seeing this, I'm all for using a field in CyFunction.
And I don't think that __class__ should be use somewhere outside super()
Agreed. CPython simply uses a compile time heuristic ("is there a function
call to something global named 'super'?") when creating this field, so it's
strictly reserved for this use case.
BTW, I like the irony in the fact that CPython essentially gives Cython
semantics to the "super" builtin here, by (partially) evaluating it at
compile time.
Yeah, I think Cython super with no args should be a little bit faster
then classic one.
I think speed isn't really all that important here. Calling Python methods
is costly enough anyway.
IMO, the main reason for the heuristic is to prevent class objects from
being kept alive by their methods, except for the single case where super()
is used. Keeping a class alive just because one of its methods is still
used somewhere can be very costly, depending on the content of the class
dict. It also creates a reference cycle, which is another costly thing in
CPython as it requires a GC run over the whole class dict to get cleaned up.
The situation for modules is at least slightly different, as modules do not
tend to get unloaded, so there's always a reference to them from
sys.modules. If that ever gets removed, it's really ok if it takes time to
clean things up.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PEP 3135 -- New Super
2011/7/5 Stefan Behnel :
> Vitja Makarov, 05.07.2011 09:17:
>>
>> 2011/7/5 Stefan Behnel:
>>>
>>> Vitja Makarov, 05.07.2011 08:21:
I was thinking about implementing new super() with no arguments.
>>>
>>> http://trac.cython.org/cython_trac/ticket/696
>>>
The problem is where to store __class__, I see two options here:
1. Add func_class member to CyFunction, this way __class__ will be
private and not visible for inner functions:
2. Put it into closure
>>>
>>> The second option has the advantage of requiring the field only when
>>> super()
>>> is used, whereas the first impacts all functions.
>>>
>>> I would expect that programs commonly have a lot more functions than
>>> specifically methods that use a no-argument call to super(), so this may
>>> make a difference.
>>>
>>
>> So, now classes are created the following way:
>>
>> class_dict = {}
>> class_dict.foo = foo_func
>> class = CreateClass(class_dict)
>>
>> So after class is created I should check its dict for CyFunction
>> members (maybe only ones that actually require __class__)
>> and set __class__:
>>
>> for value in class.__dict__.itervalues():
>> if isinstance(value, CyFunction) and value.func_class is WantClass:
>> value.func_class = class
>
> Remember that no-args super() can only be used in functions that are
> literally written inside of a class body, so we actually know at compile
> time which functions need this field. We can thus do better than a generic
> loop over all fields. We even have the function object pointers directly
> available in the module init function where we create the class body.
>
Yes, but now cyfunction references are lost as they were in temps.
> BTW, we also need a way to make this work for cdef classes. No idea how
> different that would be.
>
I think super() for cdef classes should be much easy to implement as
we already know cname for a class at compile time. That would be
simple transform.
>
>>> OTOH, not all methods have a closure, so creating one just to store the
>>> "__class__" field is very wasteful, in terms of both runtime and memory
>>> overhead. A lot more wasteful than paying 8 bytes of memory for each
>>> function, with no additional time overhead.
>>
>> Going this way it only requires to initialize closure:
>
> Yes, and that's costly.
>
>
>> Btw, first way requires cyfunction signature change, it would accept
>> cyfunction object as first argument.
>
> We currently pass the binding (i.e. owning) object, right?
>
Right. When we pass CyFunction closure should be moved into its members.
>
>> This also could help to solve default args problem.
>
> And potentially other problems, too. Think of heap allocated modules, for
> example.
>
> http://trac.cython.org/cython_trac/ticket/173
> http://trac.cython.org/cython_trac/ticket/218
>
> Seeing this, I'm all for using a field in CyFunction.
>
>
>> Yeah, I think Cython super with no args should be a little bit faster
>> then classic one.
>
> I think speed isn't really all that important here. Calling Python methods
> is costly enough anyway.
>
> IMO, the main reason for the heuristic is to prevent class objects from
> being kept alive by their methods, except for the single case where super()
> is used. Keeping a class alive just because one of its methods is still used
> somewhere can be very costly, depending on the content of the class dict. It
> also creates a reference cycle, which is another costly thing in CPython as
> it requires a GC run over the whole class dict to get cleaned up.
>
> The situation for modules is at least slightly different, as modules do not
> tend to get unloaded, so there's always a reference to them from
> sys.modules. If that ever gets removed, it's really ok if it takes time to
> clean things up.
>
Yes, but I hope that classes are rarely deleted.
And I'm afraid that we can't avoid cycles when implementing __class__
for pure-python classes.
--
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PEP 3135 -- New Super
Vitja Makarov, 05.07.2011 10:37: 2011/7/5 Stefan Behnel: IMO, the main reason for the heuristic is to prevent class objects from being kept alive by their methods, except for the single case where super() is used. Keeping a class alive just because one of its methods is still used somewhere can be very costly, depending on the content of the class dict. It also creates a reference cycle, which is another costly thing in CPython as it requires a GC run over the whole class dict to get cleaned up. The situation for modules is at least slightly different, as modules do not tend to get unloaded, so there's always a reference to them from sys.modules. If that ever gets removed, it's really ok if it takes time to clean things up. Yes, but I hope that classes are rarely deleted. And I'm afraid that we can't avoid cycles when implementing __class__ for pure-python classes. As I said, it's fine if super() is used, but not otherwise. Basically, no-args super() is just a shortcut for "super(TheClass, self)", with the difference that the long form looks up the name at runtime, whereas the short form keeps a reference to the type at module setup time. So, both need the class at runtime, but have different characteristics otherwise. (but if the code is not using super(), there is no need to have the methods own a reference to their class) Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Hudson is down?
On Mon, Jul 4, 2011 at 9:12 PM, Stefan Behnel wrote: > Vitja Makarov, 04.07.2011 20:43: >> >> It seems that hudson server is down. > > It's back now. Looks like the sage.math machine was restarted. Yep, sage.math was restarted. Thanks for getting this back up and running. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] PEP 3135 -- New Super
Stefan Behnel, 05.07.2011 10:04:
Vitja Makarov, 05.07.2011 09:17:
2011/7/5 Stefan Behnel:
Vitja Makarov, 05.07.2011 08:21:
I was thinking about implementing new super() with no arguments.
http://trac.cython.org/cython_trac/ticket/696
The problem is where to store __class__, I see two options here:
1. Add func_class member to CyFunction, this way __class__ will be
private and not visible for inner functions:
2. Put it into closure
The second option has the advantage of requiring the field only when
super()
is used, whereas the first impacts all functions.
I would expect that programs commonly have a lot more functions than
specifically methods that use a no-argument call to super(), so this may
make a difference.
So, now classes are created the following way:
class_dict = {}
class_dict.foo = foo_func
class = CreateClass(class_dict)
So after class is created I should check its dict for CyFunction
members (maybe only ones that actually require __class__)
and set __class__:
for value in class.__dict__.itervalues():
if isinstance(value, CyFunction) and value.func_class is WantClass:
value.func_class = class
Btw, first way requires cyfunction signature change, it would accept
cyfunction object as first argument.
We currently pass the binding (i.e. owning) object, right?
So, how would this work for methods? We need to pass the 'self' object
there, which the CyFunction doesn't know. If anything, it only knows the
class it was defined in, which doesn't help here.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Cython backend aiming PyPy Status
Hi I created a blog post summarizing what I've done the last few weeks on the Cython backend aiming PyPy. It's located at this URL : http://rguillebert.blogspot.com/2011/07/cython-backend-aiming-pypy-status.html Cheers Romain ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cython backend aiming PyPy Status
Romain Guillebert, 06.07.2011 05:10: I created a blog post summarizing what I've done the last few weeks on the Cython backend aiming PyPy. It's located at this URL : http://rguillebert.blogspot.com/2011/07/cython-backend-aiming-pypy-status.html Congrats, that sounds pretty usable already. Instead of stepping directly into writing dedicated tests for your existing code, you should first try to get Cython's normal test suite running. That will give you a huge set of tests right away. One remark for this list: given that you simply reuse the type analysis steps in the ctypes pipeline (which is totally the right thing to do!), Cython should be careful what it does during these steps. Traditionally (and especially prehistorically), the type analysis step has been a point where the tree is changed in ways that didn't quite fit anywhere else. From now on, we need to take care not to break the ctypes backend by applying C-only tree changes here. Another important reason for getting the test suite to run, so that we can hand the regression testing over to Jenkins. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
