[Python-Dev] recursive closures - reference leak

[Kristján Valur Jónsson]

Hello there.

Consider this code:

def factorial(n):
def helper(n):
   if n:
   return n*helper(n-1)
   else:
   return 1
return helper(n)

Ok, this is a bit contrived, but there are other valid reasons for calling a 
closure recursively.
The problem with this is that once you have called factorial() once, you end up 
with a recursive cycle.  "factorial" has become a cell object, referencing the 
"helper" function, which again refers to the outer cell object.  This requires 
"gc" to clean up.  Also, it is entirely non-obvious.  the problem becomes worse 
if the inner function also refers to some large, temporary variable, since it 
will get caught up in the reference loop.

A workaround is this.  Replace the final line with:
try:
   return helper(n)
finally:
helper = None


This clears the cell before function exit.  Note that it is not possible to 
"del" it, you get a syntax error for that.

Now, the reason I'm posting this here, and not to python-lang, is this:  Is 
there a way to implement  this differently so that this pattern doesn't result 
in a cycle?
Possible ideas would be:

1)  clear "helper" on function exit.  Not good if the internal function is 
exported, though.

2)  Use have 'helper' store a weakref to the cell object

Cheers,
Kristján
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference leak

[Hrvoje Niksic]

Kristján Valur Jónsson wrote:
The problem with this is that once you have called factorial() once, you 
end up with a recursive cycle.  „factorial“ has become a cell object, 
referencing the „helper“ function, which again refers to the outer cell 
object.  This requires „gc“ to clean up.  Also, it is entirely 
non-obvious.  the problem becomes worse if the inner function also 
refers to some large, temporary variable, since it will get caught up in 
the reference loop.


What problem are you referring to?  Python has a gc exactly to deal with 
situations like this one.  Surely you are aware that the cycle collector 
is invoked automatically without requiring user intervention.  What 
specific issue are you trying to work around?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference leak

[Kristján Valur Jónsson]

> -Original Message-
> From: Hrvoje Niksic [mailto:hrvoje.nik...@avl.com]
> Sent: 8. desember 2009 13:52
> To: Kristján Valur Jónsson
> Cc: python-dev@python.org
> Subject: Re: [Python-Dev] recursive closures - reference leak
> What problem are you referring to?  Python has a gc exactly to deal
> with
> situations like this one.  Surely you are aware that the cycle
> collector
> is invoked automatically without requiring user intervention.  What
> specific issue are you trying to work around?

Ah, yes.  In my particular case, I'm running a cluster of hundreds of nodes, 
supporting 50.000 players in a real-time space simulation.  We disable GC 
because of its unpredictable performance impact and are careful to avoid 
reference cycles.  We use gc from time to time to _find_ those cases that our 
programmers have missed, and fix them.  This is how I stumbled upon this 
particular reference cycle that even a high level programmer would not have 
expected to have created.  This is, IMHO the best use you can make of "gc":  
Help you code well, not let you cope with sloppy code :)

K

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference cycle

[Antoine Pitrou]

Kristján Valur Jónsson  ccpgames.com> writes:
> 
> The problem with this is that once you have called
> factorial() once, you end up with a recursive cycle.

You don't need a closure to exhibit a reference cycle. A global function is 
enough:

>>> def helper(n):
...  if n:
...   return n*helper(n-1)
...  else:
...   return 1
... 
>>> helper.func_globals['helper'] is helper
True


If you really want to avoid this you can prevent the function from depending
on its outside environment:

>>> from functools import partial
>>> def helper2(rec, n):
...  if n:
...   return n*rec(rec, n-1)
...  else:
...   return 1
... 
>>> factorial = partial(helper2, helper2)
>>> "helper2" in factorial.func.func_globals
True
>>> del helper2
>>> "helper2" in factorial.func.func_globals
False
>>> factorial(3)
6


Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference leak

[Maciej Fijalkowski]

>
> Ah, yes.  In my particular case, I'm running a cluster of hundreds of nodes, 
> supporting 50.000 players in a real-time space simulation.  We disable GC 
> because of its unpredictable performance impact and are careful to avoid 
> reference cycles.  We use gc from time to time to _find_ those cases that our 
> programmers have missed, and fix them.  This is how I stumbled upon this 
> particular reference cycle that even a high level programmer would not have 
> expected to have created.  This is, IMHO the best use you can make of "gc":  
> Help you code well, not let you cope with sloppy code :)
>
> K
>

Then it is a bit your fault. There is nothing particularly wrong with
creating reference cycles (ie you can't avoid having a gc running in
Java or Jython or anything else basically). Note that disabling gc
does not mean that you will not have unpredictable pauses. Consider
for example that if you loose a reference to a very long chain of
objects, you can have arbitrarily many frees being called before
anything else can happen.

Cheers,
fijal
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference leak

[Daniel Stutzbach]

2009/12/8 Maciej Fijalkowski 

> Note that disabling gc
> does not mean that you will not have unpredictable pauses. Consider
> for example that if you loose a reference to a very long chain of
> objects, you can have arbitrarily many frees being called before
> anything else can happen.
>

That strikes me as a *predictable* long pause.

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference leak

[Kristján Valur Jónsson]

Yes.  Just –in-time releaseing of objects is much preferable to delayed release.
They tend to be in the cache, the individual runs of releases are smaller and 
have less of an individual impact.
a gc.collect() cycle visits a large amount of objects that it won‘t release 
causing cache thrashing.
There is a reason we disabled ‚gc‘, and it is simply because we get lower cpu 
and smoother execution.
K

From: daniel.stutzb...@gmail.com [mailto:daniel.stutzb...@gmail.com] On Behalf 
Of Daniel Stutzbach
Sent: 8. desember 2009 15:04
To: Maciej Fijalkowski
Cc: Kristján Valur Jónsson; python-dev@python.org
Subject: Re: [Python-Dev] recursive closures - reference leak

2009/12/8 Maciej Fijalkowski mailto:fij...@gmail.com>>
Note that disabling gc
does not mean that you will not have unpredictable pauses. Consider
for example that if you loose a reference to a very long chain of
objects, you can have arbitrarily many frees being called before
anything else can happen.

That strikes me as a *predictable* long pause.
--

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference cycle

[Kristján Valur Jónsson]

> -Original Message-
> From: python-dev-bounces+kristjan=ccpgames@python.org
> [mailto:python-dev-bounces+kristjan=ccpgames@python.org] On Behalf
> Of Antoine Pitrou
> Sent: 8. desember 2009 14:55
> To: python-dev@python.org
> Subject: Re: [Python-Dev] recursive closures - reference cycle
> 
> Kristján Valur Jónsson  ccpgames.com> writes:
> >
> > The problem with this is that once you have called
> > factorial() once, you end up with a recursive cycle.
> 
> You don't need a closure to exhibit a reference cycle. A global
> function is
> enough:
> 
> >>> def helper(n):
> ...  if n:
> ...   return n*helper(n-1)
> ...  else:
> ...   return 1
> ...
> >>> helper.func_globals['helper'] is helper
> True
> 
yes, because:
func_globals is globals() == True

You don't need recursion for this to be true.  And as soon as you delete 
"helper" from globals() it goes away.
w = wearref.weakref(helper)
del helper
w() == False.

> 
> If you really want to avoid this you can prevent the function from
> depending
> on its outside environment:
> 
> >>> from functools import partial
> >>> def helper2(rec, n):
> ...  if n:
> ...   return n*rec(rec, n-1)
> ...  else:
> ...   return 1
> ...
> >>> factorial = partial(helper2, helper2)
> >>> "helper2" in factorial.func.func_globals
> True
> >>> del helper2
> >>> "helper2" in factorial.func.func_globals
> False
> >>> factorial(3)
> 6

Interesting, pass itself in as an argument.
Yes, there are ways around this, I know.  But you have to agree that it is 
unexpected, no?
Somethign for the "reference cycle FAQ."

Anyway, I´ll return to my lair now, thanks for your time python-dev :)

K
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Proposing PEP 386 for addition

[Tarek Ziadé]

Hello,

On behalf of the Distutils-SIG, I would like to propose PEP 386 for
inclusion in the sdtlib, and have a final discussion here on
Python-dev.

http://www.python.org/dev/peps/pep-0386

This PEP has been discussed for some time in Distutils-SIG, and we
agreed there that it's important to have it accepted for the future of
packaging because:

1/ it will offer interoperability for all Python package managers out
there. (namely: Distutils, Distribute, Setuptools, Pip, PyPM)
2/ it will provide a reference for OS packagers, so they can translate
automatically Python distributions versions in their own schemes.

Choosing a schema was quite controversial because every development
team have their own version scheme. But notice that it is not in the
scope of this PEP to come up with an universal versioning schema,
intended to support many or all existing versioning schemas. There
will always be competing grammars, either mandated by distro or
project policies or by historical reasons and we cannot expect that to
change.

The goal of this PEP is rather to provide a standard reference schema
that is able to express the usual versioning semantics, so it's
possible to parse any alternative versioning schema and transform it
into a compliant one. This is how OS packagers usually deal with the
existing version schemas and is a preferable alternative than
supporting an arbitrary set of versioning schemas.

I believe that the current situation is as close to consensus as we
will get on distutils-sig, and in the interests of avoiding months of
further discussion which won't take things any further, I propose to
allow final comments from python-dev and then look for a final
decision.

Regards
Tarek

-- 
Tarek Ziadé | http://ziade.org | オープンソースはすごい! | 开源传万世，因有你参与
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposing PEP 386 for addition

[Terry Reedy]

Tarek Ziadé wrote:

Hello,

On behalf of the Distutils-SIG, I would like to propose PEP 386 for
inclusion in the sdtlib, and have a final discussion here on
Python-dev.

http://www.python.org/dev/peps/pep-0386


Some English copy editor comments:

"and it will optionally allow to use that field"
I believe you mean
"and it will optionally allow use of that field"

"requires zope.interface, as long as its version is superior to 3.5.0."
"requires zope.interface with a version greater than 3.5.0."

"It is not in the scope of this PEP to come with an universal versioning 
schema,intended to support..."

I believe you mean
"It is not in the scope of this PEP to provide a universal versioning 
schema intended to support..."



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposing PEP 386 for addition

[MRAB]

Terry Reedy wrote:

Tarek Ziadé wrote:

Hello,

On behalf of the Distutils-SIG, I would like to propose PEP 386 for
inclusion in the sdtlib, and have a final discussion here on
Python-dev.

http://www.python.org/dev/peps/pep-0386


Some English copy editor comments:

"and it will optionally allow to use that field"
I believe you mean
"and it will optionally allow use of that field"

"requires zope.interface, as long as its version is superior to 3.5.0."
"requires zope.interface with a version greater than 3.5.0."


[snip]

I assume this does mean > 3.5.0. If instead it means >= 3.5.0 then it
would be clearer as "3.5.0 or greater/above/later".

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposing PEP 386 for addition

[Russell E. Owen]

In article 
<94bdd2610912080916s2dbb79d0ub8a77295bba92...@mail.gmail.com>,
 Tarek Ziad?  wrote:

> http://www.python.org/dev/peps/pep-0386

It looks great to me. Very complete and easy to understand.

-- Russell

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] recursive closures - reference cycle

[Greg Ewing]

You could use a class:

  class factorial():

def fac(self, n):
  if n == 0:
return 1
  else:
return n * self.fac(n - 1)

def __call__(self, n):
  return self.fac(n)

  factorial = factorial()

  print factorial(5)

--
Greg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposing PEP 386 for addition

[Tarek Ziadé]

On Tue, Dec 8, 2009 at 8:45 PM, Terry Reedy  wrote:
> Tarek Ziadé wrote:
>>
>> Hello,
>>
>> On behalf of the Distutils-SIG, I would like to propose PEP 386 for
>> inclusion in the sdtlib, and have a final discussion here on
>> Python-dev.
>>
>> http://www.python.org/dev/peps/pep-0386
>
> Some English copy editor comments:
>
> "and it will optionally allow to use that field"
> I believe you mean
> "and it will optionally allow use of that field"
>
> "requires zope.interface, as long as its version is superior to 3.5.0."
> "requires zope.interface with a version greater than 3.5.0."
>
> "It is not in the scope of this PEP to come with an universal versioning
> schema,intended to support..."
> I believe you mean
> "It is not in the scope of this PEP to provide a universal versioning schema
> intended to support..."

Applied, thank you
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com