[Python-Dev] Reference counting when entering and exiting scopes

[Matthew F. Barnes]

Someone on python-help suggested that I forward this question to
python-dev.

I've been studying Python's core compiler and bytecode interpreter as a
model for my own interpreted language, and I've come across what appears
to be a reference counting problem in the `symtable_exit_scope' function
in .

At this point I assume that I'm just misunderstanding what's going on.  So
I was hoping to contact one of the core developers before I go filing what
could very well be a spurious bug report against Python's core.

Here's the function copied from CVS HEAD:

static int
symtable_exit_scope(struct symtable *st)
{
int end;

if (st->st_pass == 1)
symtable_update_free_vars(st);
Py_DECREF(st->st_cur);
end = PyList_GET_SIZE(st->st_stack) - 1;
st->st_cur = (PySymtableEntryObject *)PyList_GET_ITEM(st->st_stack,
  end);
if (PySequence_DelItem(st->st_stack, end) < 0)
return -1;
return 0;
}

My issue is with the use of PyList_GET_ITEM to fetch a new value for the
current scope.  As I understand it, PyList_GET_ITEM does not increment the
reference count for the returned value.  So in effect we're borrowing the
reference to the symtable entry object from the tail of the scope stack. 
But then we turn around and delete the object from the tail of the scope
stack, which DOES decrement the reference count.

So `symtable_exit_scope' has a net effect of decrementing the reference
count of the new current symtable entry object, when it seems to me like
it should stay the same.  Shouldn't the reference count be incremented
when we assign to "st->st_cur" (either explicitly or by fetching the
object using the PySequence API instead of PyList)?

Can someone explain the rationale here?

---

As an addendum to the previous question, further study of the
 code has made me believe that there's a reference
counting problem in the `symtable_enter_scope' function as well (pasted
below from CVS HEAD).  Namely, that `prev' should be Py_XDECREF'd at some
point in the function (at the end of the first IF block, perhaps?).

static void
symtable_enter_scope(struct symtable *st, char *name, int type,
 int lineno)
{
PySymtableEntryObject *prev = NULL;

if (st->st_cur) {
prev = st->st_cur;
if (PyList_Append(st->st_stack, (PyObject *)st->st_cur) < 0) {
st->st_errors++;
return;
}
}
st->st_cur = (PySymtableEntryObject *)
PySymtableEntry_New(st, name, type, lineno);
if (st->st_cur == NULL) {
st->st_errors++;
return;
}
if (strcmp(name, TOP) == 0)
st->st_global = st->st_cur->ste_symbols;
if (prev && st->st_pass == 1) {
if (PyList_Append(prev->ste_children,
  (PyObject *)st->st_cur) < 0)
st->st_errors++;
}
}

Thanks,
Matthew Barnes


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reference counting when entering and exiting scopes

[Matthew F. Barnes]

On Wed, 2005-04-20 at 18:59 -0700, Brett C. wrote:
> So no leak.  Yes, there should be more explicit refcounting to be proper, but
> the compiler cheats in a couple of places for various reasons.  But basically
> everything is fine since st->st_cur and st->st_stack are only played with
> refcount-wise by either symtable_enter_scope() and symtable_exit_scope() and
> they are always called in pairs in the end.

... except for the "global" scope, for which symtable_exit_scope() is
never called.  But the last reference to *that* scope (st->st_cur) gets
cleaned up in PySymtable_Free().  Correct?

So the two things I thought were glitches are actually cancelling each
other out.  Very good.  Thanks for your help.

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Lexical analysis and NEWLINE tokens

[Matthew F. Barnes]

I posted this question to python-help, but I think I have a better chance
of getting the answer here.

I'm looking for clarification on when NEWLINE tokens are generated during
lexical analysis of Python source code.  In particular, I'm confused about
some of the top-level components in Python's grammar (file_input,
interactive_input, and eval_input).

Section 2.1.7 of the reference manual states that blank lines (lines
consisting only of whitespace and possibly a comment) do not generate
NEWLINE tokens.  This is supported by the definition of a suite, which
does not allow for standalone or consecutive NEWLINE tokens.

suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT

Yet the grammar for top-level components seems to suggest that a parsable
input may consist entirely of a single NEWLINE token, or include
consecutive NEWLINE tokens.

file_input ::= (NEWLINE | statement)*
interactive_input ::= [stmt_list] NEWLINE | compound_stmt NEWLINE
eval_input ::= expression_list NEWLINE*

To me this seems to contradict section 2.1.7 in so far as I don't see how
it's possible to generate such a sequence of tokens.

What kind of input would generate NEWLINE tokens in the top-level
components of the grammar?

Matthew Barnes
[EMAIL PROTECTED]
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

Hello,

I've not had much luck in searching for a discussion on this in the
Python-Dev archives, so bear with me.

I had an idea this morning for a simple extension to Python's iterator
protocol that would allow the user to force an iterator to raise
StopIteration on the next call to next().  My thought was to add a new
method to iterators called stop().

In my situation it would be useful as a control-flow mechanism, but I
imagine there are many other use cases for it:


generator = some_generator_function()

for x in generator:
... deeply ...
... nested ...
... control-flow ...

if satisfaction_condition:
# Terminates the for-loop, but
# finishes the current iteration
generator.stop()

... more stuff ...


I'm curious if anything like this has been proposed in the past.  If so,
could someone kindly point me to any relevant mailing list threads?

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

On Wed, 2005-11-30 at 10:57 -0800, Alex Martelli wrote:
> PEP 342, already accepted and found at
> http://python.org/peps/pep-0342.html , covers related functionality
> (as well as many other points).

Thanks Alex, I'll take another look at that PEP.  The first time I tried
to read it my brain started to sizzle.

I happened to use a generator-iterator in my example, but my thought was
that the extension could be applied to iterators in general, including
sequence-iterators.

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

On Wed, 2005-11-30 at 14:36 -0500, Edward Loper wrote:
> There's no need to change the iterator protocol for your example use 
> case; you could just define a simple iterator-wrapper:

Good point.  Perhaps it would be a useful addition to the itertools
module then?

itertools.interruptable(iterable)

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Short-circuiting iterators

[Matthew F. Barnes]

On Tue, 2005-12-06 at 19:58 -0500, Raymond Hettinger wrote:
> Any real-world use cases or compelling contrived examples?
> 
> ISTM, that the code calling it.stop() would already be in position to
> break-out of the iteration directly or set a termination flag.  Instead
> of:
> 
> it = itertools.interruptable(iterable):
> for x in it:
> . . .
> if cond(x):
> it.stop()
> 
> Why not write:
> 
> for x in iterable:
> . . .
> if cond(x):
> break
> 
> If needed, the for-loop can have an else-clause for any processing
> needed in the event of interruption.

The idea was motivated by a case of nested loops, similar to:

for x in iterable1:
for y in iterable2:
for z in iterable3:
. . .
if cond1(x):
iterable1.stop()
if cond2(y):
iterable2.stop()
if cond3(z):
iterable3.stop()
. . .

It seemed more convenient at the time than having to deal with multiple
termination flags, or breaks, or a combination thereof.

The ability to remotely terminate a for-loop also struck me as somewhat
interesting:

def estimate(item, iterable):
. . .
if good_enough:
iterable.stop()
return result

for x in iterable:
. . .
approx *= estimate(x, iterable)

But these are highly contrived and hardly compelling.

I was primarily interested in whether anyone recalls discussing the
ability to prematurely terminate an iterator and whether there are any
technical drawbacks other than it being redundant.

Matthew Barnes
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com