[Python-Dev] new unbounded memory leak in exception handling?

[Greg Hewgill]

I've constructed an example program that does not leak memory in Python
2.x, but causes unbounded memory allocation in Python 3.1. Here is the
code:

import gc
import sys

class E(Exception):
def __init__(self, fn):
self.fn = fn
def call(self):
self.fn()

def f():
raise E(f)

a = E(f)
while True:
print(len(gc.get_objects()))
try:
a.call()
except E:
# get exception value in a python2/3 portable way
a = sys.exc_info()[1]

Every time through the loop, the length of gc.get_objects() increases
by 7 under Python 3.1. I believe this is a regression error, since
Python 2.x does not exhibit the same behaviour.

Can anybody confirm this?

Greg Hewgill
http://hewgill.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Add an optional timeout to lock operations

[Antoine Pitrou]

Hello,

I've submitted a patch (*) to add an optional timeout to locking operations
(Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to
know if there was any good reason for not doing it.

(*) http://bugs.python.org/issue7316

Thank you

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] new unbounded memory leak in exception handling?

[R. David Murray]

On Tue, 17 Nov 2009 at 10:31, Greg Hewgill wrote:

I've constructed an example program that does not leak memory in Python
2.x, but causes unbounded memory allocation in Python 3.1. Here is the
code:

import gc
import sys

class E(Exception):
   def __init__(self, fn):
   self.fn = fn
   def call(self):
   self.fn()

def f():
   raise E(f)

a = E(f)
while True:
   print(len(gc.get_objects()))
   try:
   a.call()
   except E:
   # get exception value in a python2/3 portable way
   a = sys.exc_info()[1]

Every time through the loop, the length of gc.get_objects() increases
by 7 under Python 3.1. I believe this is a regression error, since
Python 2.x does not exhibit the same behaviour.

Can anybody confirm this?


I think you want to take a look at PEP 3134.  And then please file a doc
bug to have someone update the documentation of sys.exc_info, since the
advice in the warning box is no longer valid in Python 3.

--David (RDM)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[Guido van Rossum]

I think I can answer the "why" question: thread.c is *very* old code,
in fact it predates the posix threads standard. When we (actually
Sjoerd Mullender) wrote it, we had a number of OS-specific locking
APIs to work with and the API was designed to fit all of them. I don't
even recall the initial set, but I think it included SGI Irix and and
old version of SunOS. So then over time new platforms were added, but
adding a new method or parameter to the API was nearly impossible
because someone would have to find out how to implement the new
feature on all supported platforms. I think the number of platforms
has dwindled to two or three (Posix, Windows, and maybe one minority
OS?), so now's the time to do it. (IOW I think the idea of the patch
is fine.)

Will locks be interruptible with ^C? That is an oft-requested feature
which also wasn't supported at that time; what's the situation
nowadays?

--Guido

On Tue, Nov 17, 2009 at 3:46 AM, Antoine Pitrou  wrote:
>
> Hello,
>
> I've submitted a patch (*) to add an optional timeout to locking operations
> (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like 
> to
> know if there was any good reason for not doing it.
>
> (*) http://bugs.python.org/issue7316
>
> Thank you
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add an optional timeout to lock operations

[Antoine Pitrou]

Guido van Rossum  python.org> writes:
> 
> I think the number of platforms
> has dwindled to two or three (Posix, Windows, and maybe one minority
> OS?), so now's the time to do it. (IOW I think the idea of the patch
> is fine.)

Thanks. (the minority OS would be OS/2, I think)

> Will locks be interruptible with ^C? That is an oft-requested feature
> which also wasn't supported at that time; what's the situation
> nowadays?

They still aren't interruptible. From what I can read it may be possible to make
them interruptible in the POSIX semaphore-based implementation, not in the POSIX
condition variable-based implementation (which is used as a fallback when POSIX
semaphores are not available, but I don't know whether this fallback is still
useful). 
As for Windows I have absolutely no idea.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] new unbounded memory leak in exception handling?

[Greg Hewgill]

On Tue, Nov 17, 2009 at 08:40:37AM -0500, R. David Murray wrote:
> I think you want to take a look at PEP 3134.  And then please file a doc
> bug to have someone update the documentation of sys.exc_info, since the
> advice in the warning box is no longer valid in Python 3.

Thanks for the pointer, I understand the cause now. I had read the
warning in the description of sys.exc_info, but I hadn't realised that
Python was (now) implicitly saving the traceback for me. I was able to
solve this memory leak by explicitly clearing __traceback__:

except E:
# get exception value in a python2/3 portable way
a = sys.exc_info()[1]
a.__traceback__ = None

I have added a doc bug issue for this: http://bugs.python.org/issue7340

Greg Hewgill
http://hewgill.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Static analysis of CPython using coccinelle/spatch

[Brett Cannon]

On Mon, Nov 16, 2009 at 12:27, David Malcolm  wrote:
> Has anyone else looked at using Coccinelle/spatch[1] on CPython source
> code?

Not that has been mentioned on the list before.

>
> It's a GPL-licensed tool for matching semantic patterns in C source
> code. It's been used on the Linux kernel for detecting and fixing
> problems, and for autogenerating patches when refactoring
> (http://coccinelle.lip6.fr/impact_linux.php).  Although it's implemented
> in OCaml, it is scriptable using Python.
>
> I've been experimenting with using it on CPython code, both on the core
> implementation, and on C extension modules.
>
> As a test, I've written a validator for the mini-language used by
> PyArg_ParseTuple and its variants.  My code examines the types of the
> variables passed as varargs, and attempts to check that they are
> correct, according to the rules here
> http://docs.python.org/c-api/arg.html (and in Python/getargs.c)
>
> It can detect this old error (fixed in svn r34931):
> buggy.c:12:socket_htons:Mismatching type of argument 1 in ""i:htons"":
> expected "int *" but got "unsigned long *"
>
> Similarly, it finds the deliberate error in xxmodule.c:
> xxmodule.c:207:xx_roj:unknown format char in "O#:roj": '#'
>
> (Unfortunately, when run on the full source tree, I see numerous
> messages, and as far as I can tell, the others are false positives)
>
> You can see the code here:
> http://fedorapeople.org/gitweb?p=dmalcolm/public_git/check-cpython.git;a=tree
> and download using anonymous git in this manner:
> git clone 
> git://fedorapeople.org/home/fedora/dmalcolm/public_git/check-cpython.git
>
> The .cocci file detects invocations of PyArg_ParseTuple and determines
> the types of the arguments.  At each matching call site it invokes
> python code, passing the type information to validate.py's
> validate_types.
>
> (I suspect it's possible to use spatch to detect reference counting
> antipatterns; I've also attempted 2to3 refactoring of c code using
> semantic patches, but so far macros tend to get in the way).
>
> Alternatively, are there any other non-proprietary static analysis tools
> for CPython?

Specific to CPython? No. But I had a chance to run practically every
major commercial static analysis tool over the code base back on 2006.
We also occasionally run valgrind over the code. But thanks to have we
have structured the code and taken performance shortcuts static
analysis tools easily get tripped up by CPython (as you have
discovered).

>
> Thoughts?

Running the tool over the code base and reporting the found bugs would
be appreciated.

-Brett


> Dave
>
> [1] http://coccinelle.lip6.fr/
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Static analysis of CPython using coccinelle/spatch

[A.M. Kuchling]

On Mon, Nov 16, 2009 at 03:27:53PM -0500, David Malcolm wrote:
> Has anyone else looked at using Coccinelle/spatch[1] on CPython source
> code?

For an excellent explanation of Coccinelle, see
.

--amk
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 and 3.2 release schedules

[Georg Brandl]

Benjamin Peterson schrieb:
> After more thought, I think that separating the 2.7 and 3.2 releases
> is not as big of an issue as I once thought. Therefore, I'd like to
> adopt the schedule I posted a few weeks back for 2.7 only.
> 
> This only means some other lucky victi... I mean volunteer can do 3.2. :)

If no one else wants to try and ruin Python 3, I'll do it .

In the regular 18-month schedule, it is due around Jan 2011, when I will
have finished my diploma and have enough time while looking for PhD
opportunities.

cheers,
Georg

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 and 3.2 release schedules

[Barry Warsaw]

On Nov 17, 2009, at 4:55 PM, Georg Brandl wrote:


Benjamin Peterson schrieb:

After more thought, I think that separating the 2.7 and 3.2 releases
is not as big of an issue as I once thought. Therefore, I'd like to
adopt the schedule I posted a few weeks back for 2.7 only.

This only means some other lucky victi... I mean volunteer can do  
3.2. :)


If no one else wants to try and ruin Python 3, I'll do it .


Ha ha ha^H^H^H^H^H^H^H^H^HGreat!  Benjamin's the expert now, but I'm  
here to help if needed.


-Barry



PGP.sig
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 and 3.2 release schedules

[Antoine Pitrou]

Barry Warsaw  python.org> writes:
> 
> > If no one else wants to try and ruin Python 3, I'll do it .
> 
> Ha ha ha^H^H^H^H^H^H^H^H^HGreat!  Benjamin's the expert now, but I'm  
> here to help if needed.

Well Georg isn't a novice when it comes to ruining things, especially
documentation and commit statistics. Although I'm sure guidance from a veteran
can be useful if we want to reach that ultimate degree of perfection which helps
attract an endless stream of anonymous function syntax proposals.

Regards

Antoine, who is improving as well.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Static analysis of CPython using coccinelle/spatch

[Terry Reedy]

A.M. Kuchling wrote:

On Mon, Nov 16, 2009 at 03:27:53PM -0500, David Malcolm wrote:

Has anyone else looked at using Coccinelle/spatch[1] on CPython source
code?


For an excellent explanation of Coccinelle, see
.


For those who have not looked, Coccinelle means ladybug (a bug-eating 
bug ;-) in French. Its principle use to to take C code and a SmPl file 
of high-level patch descriptions (fixers, in 2to3 talk) and produce a 
standard diff file. I wonder if this could be used to help people 
migrate C extensions to 3.1, by developing a SmPl file with the needed 
changes dictated by API changes. This is similar to its motivating 
application to Linux. From


http://coccinelle.lip6.fr/

"Coccinelle is a program matching and transformation engine which 
provides the language SmPL (Semantic Patch Language) for specifying 
desired matches and transformations in C code. Coccinelle was initially 
targeted towards performing collateral evolutions in Linux. Such 
evolutions comprise the changes that are needed in client code in 
response to evolutions in library APIs, and may include modifications 
such as renaming a function, adding a function argument whose value is 
somehow context-dependent, and reorganizing a data structure. "


As I understand it, the problem with C extensions and 3.1 is the current 
lack of a "collateral evolution" tool like 2to3 for Python code.


Terry Jan Reedy




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com