[Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Nick Coghlan]

PEP 377 is a proposal to allow context manager __enter__() methods to
skip the body of the with statement by raising a specific (new) flow
control exception.

Since there is a working reference implementation now, I thought it was
time to open it up for broader discussion.

Full PEP attached, or you can find it in the usual place at
http://www.python.org/dev/peps/pep-0377

Cheers,
Nick.

P.S. I expect a rationale for the StatementSkipped value binding is
probably going to be pretty high on the list of questions that aren't
currently covered by the PEP. I hope to write more on that some time
this week.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
PEP: 377
Title: Allow __enter__() methods to skip the statement body
Version: $Revision: 70384 $
Last-Modified: $Date: 2009-03-15 22:48:49 +1000 (Sun, 15 Mar 2009) $
Author: Nick Coghlan 
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 8-Mar-2009
Python-Version: 2.7, 3.1
Post-History: 8-Mar-2009


Abstract


This PEP proposes a backwards compatible mechanism that allows ``__enter__()``
methods to skip the body of the associated ``with`` statement. The lack of
this ability currently means the ``contextlib.contextmanager`` decorator
is unable to fulfil its specification of being able to turn arbitrary
code into a context manager by moving it into a generator function
with a yield in the appropriate location. One symptom of this is that
``contextlib.nested`` will currently raise ``RuntimeError`` in
situations where writing out the corresponding nested ``with``
statements would not [1].

The proposed change is to introduce a new flow control exception
``SkipStatement``, and skip the execution of the ``with``
statement body if ``__enter__()`` raises this exception.


Proposed Change
===

The semantics of the ``with`` statement will be changed to include a
new ``try``/``except``/``else`` block around the call to ``__enter__()``.
If ``SkipStatement`` is raised by the ``__enter__()`` method, then
the main section of the ``with`` statement (now located in the ``else``
clause) will not be executed. To avoid leaving the names in any ``as``
clause unbound in this case, a new ``StatementSkipped`` singleton
(similar to the existing ``NotImplemented`` singleton) will be
assigned to all names that appear in the ``as`` clause.

The components of the ``with`` statement remain as described in PEP 343 [2]::

with EXPR as VAR:
BLOCK

After the modification, the ``with`` statement semantics would
be as follows::

mgr = (EXPR)
exit = mgr.__exit__  # Not calling it yet
try:
value = mgr.__enter__()
except SkipStatement:
VAR = StatementSkipped
# Only if "as VAR" is present and
# VAR is a single name
# If VAR is a tuple of names, then StatementSkipped
# will be assigned to each name in the tuple
else:
exc = True
try:
try:
VAR = value  # Only if "as VAR" is present
BLOCK
except:
# The exceptional case is handled here
exc = False
if not exit(*sys.exc_info()):
raise
# The exception is swallowed if exit() returns true
finally:
# The normal and non-local-goto cases are handled here
if exc:
exit(None, None, None)

With the above change in place for the ``with`` statement semantics,
``contextlib.contextmanager()`` will then be modified to raise
``SkipStatement`` instead of ``RuntimeError`` when the underlying
generator doesn't yield.


Rationale for Change


Currently, some apparently innocuous context managers may raise
``RuntimeError`` when executed. This occurs when the context
manager's ``__enter__()`` method encounters a situation where
the written out version of the code corresponding to the
context manager would skip the code that is now the body
of the ``with`` statement. Since the ``__enter__()`` method
has no mechanism available to signal this to the interpreter,
it is instead forced to raise an exception that not only
skips the body of the ``with`` statement, but also jumps over
all code until the nearest exception handler. This goes against
one of the design goals of the ``with`` statement, which was to
be able to factor out arbitrary common exception handling code
into a single context manager by putting into a generator
function and replacing the variant part of the code with a
``yield`` statement.

Specifically, the following examples behave differently if
``cmB().__enter__()`` raises an exception which ``cmA().__exit__()``
then handles and suppresses::

  with cmA():
with cmB():
  do_stuff()
  # This will resume here without executing "do_stuff()"

  @contextlib.contextmanager
  def combined():
with cmA():
  with cmB():
yield

  with

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Brett Cannon]

Without knowing what StatementSkipped is (just some singleton? If so why not
just used SkipStatement instance that was raised?) and wondering if we are
just going to continue to adding control flow exceptions that directly
inherit from BaseException or some ControlFlowException base class, the
basic idea seems fine by me.

On Sun, Mar 15, 2009 at 05:56, Nick Coghlan  wrote:

> PEP 377 is a proposal to allow context manager __enter__() methods to
> skip the body of the with statement by raising a specific (new) flow
> control exception.
>
> Since there is a working reference implementation now, I thought it was
> time to open it up for broader discussion.
>
> Full PEP attached, or you can find it in the usual place at
> http://www.python.org/dev/peps/pep-0377
>
> Cheers,
> Nick.
>
> P.S. I expect a rationale for the StatementSkipped value binding is
> probably going to be pretty high on the list of questions that aren't
> currently covered by the PEP. I hope to write more on that some time
> this week.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ---
>
> PEP: 377
> Title: Allow __enter__() methods to skip the statement body
> Version: $Revision: 70384 $
> Last-Modified: $Date: 2009-03-15 22:48:49 +1000 (Sun, 15 Mar 2009) $
> Author: Nick Coghlan 
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 8-Mar-2009
> Python-Version: 2.7, 3.1
> Post-History: 8-Mar-2009
>
>
> Abstract
> 
>
> This PEP proposes a backwards compatible mechanism that allows
> ``__enter__()``
> methods to skip the body of the associated ``with`` statement. The lack of
> this ability currently means the ``contextlib.contextmanager`` decorator
> is unable to fulfil its specification of being able to turn arbitrary
> code into a context manager by moving it into a generator function
> with a yield in the appropriate location. One symptom of this is that
> ``contextlib.nested`` will currently raise ``RuntimeError`` in
> situations where writing out the corresponding nested ``with``
> statements would not [1].
>
> The proposed change is to introduce a new flow control exception
> ``SkipStatement``, and skip the execution of the ``with``
> statement body if ``__enter__()`` raises this exception.
>
>
> Proposed Change
> ===
>
> The semantics of the ``with`` statement will be changed to include a
> new ``try``/``except``/``else`` block around the call to ``__enter__()``.
> If ``SkipStatement`` is raised by the ``__enter__()`` method, then
> the main section of the ``with`` statement (now located in the ``else``
> clause) will not be executed. To avoid leaving the names in any ``as``
> clause unbound in this case, a new ``StatementSkipped`` singleton
> (similar to the existing ``NotImplemented`` singleton) will be
> assigned to all names that appear in the ``as`` clause.
>
> The components of the ``with`` statement remain as described in PEP 343
> [2]::
>
>with EXPR as VAR:
>BLOCK
>
> After the modification, the ``with`` statement semantics would
> be as follows::
>
>mgr = (EXPR)
>exit = mgr.__exit__  # Not calling it yet
>try:
>value = mgr.__enter__()
>except SkipStatement:
>VAR = StatementSkipped
># Only if "as VAR" is present and
># VAR is a single name
># If VAR is a tuple of names, then StatementSkipped
># will be assigned to each name in the tuple
>else:
>exc = True
>try:
>try:
>VAR = value  # Only if "as VAR" is present
>BLOCK
>except:
># The exceptional case is handled here
>exc = False
>if not exit(*sys.exc_info()):
>raise
># The exception is swallowed if exit() returns true
>finally:
># The normal and non-local-goto cases are handled here
>if exc:
>exit(None, None, None)
>
> With the above change in place for the ``with`` statement semantics,
> ``contextlib.contextmanager()`` will then be modified to raise
> ``SkipStatement`` instead of ``RuntimeError`` when the underlying
> generator doesn't yield.
>
>
> Rationale for Change
> 
>
> Currently, some apparently innocuous context managers may raise
> ``RuntimeError`` when executed. This occurs when the context
> manager's ``__enter__()`` method encounters a situation where
> the written out version of the code corresponding to the
> context manager would skip the code that is now the body
> of the ``with`` statement. Since the ``__enter__()`` method
> has no mechanism available to signal this to the interpreter,
> it is instead forced to raise an exception that not only
> skips the body of the ``with`` statement, but also jumps over
> all code until the nearest exception handler. This goes against
> one of the design goals of the ``with`` statement, which wa

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Michael Foord]

Brett Cannon wrote:
Without knowing what StatementSkipped is (just some singleton? If so 
why not just used SkipStatement instance that was raised?) and 
wondering if we are just going to continue to adding control flow 
exceptions that directly inherit from BaseException or some 
ControlFlowException base class, the basic idea seems fine by me.




Note that using exceptions for control flow can  be bad for other 
implementations of Python. For example exceptions on the .NET framework 
are very expensive. (Although there are workarounds such as not really 
raising the exception - but they're ugly).


Isn't it better practise for exceptions to be used for exceptional 
circumstances rather than for control flow?


Michael

On Sun, Mar 15, 2009 at 05:56, Nick Coghlan > wrote:


PEP 377 is a proposal to allow context manager __enter__() methods to
skip the body of the with statement by raising a specific (new) flow
control exception.

Since there is a working reference implementation now, I thought
it was
time to open it up for broader discussion.

Full PEP attached, or you can find it in the usual place at
http://www.python.org/dev/peps/pep-0377

Cheers,
Nick.

P.S. I expect a rationale for the StatementSkipped value binding is
probably going to be pretty high on the list of questions that aren't
currently covered by the PEP. I hope to write more on that some time
this week.

--
Nick Coghlan   |   ncogh...@gmail.com 
  |   Brisbane, Australia
---

PEP: 377
Title: Allow __enter__() methods to skip the statement body
Version: $Revision: 70384 $
Last-Modified: $Date: 2009-03-15 22:48:49 +1000 (Sun, 15 Mar 2009) $
Author: Nick Coghlan mailto:ncogh...@gmail.com>>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 8-Mar-2009
Python-Version: 2.7, 3.1
Post-History: 8-Mar-2009


Abstract


This PEP proposes a backwards compatible mechanism that allows
``__enter__()``
methods to skip the body of the associated ``with`` statement. The
lack of
this ability currently means the ``contextlib.contextmanager``
decorator
is unable to fulfil its specification of being able to turn arbitrary
code into a context manager by moving it into a generator function
with a yield in the appropriate location. One symptom of this is that
``contextlib.nested`` will currently raise ``RuntimeError`` in
situations where writing out the corresponding nested ``with``
statements would not [1].

The proposed change is to introduce a new flow control exception
``SkipStatement``, and skip the execution of the ``with``
statement body if ``__enter__()`` raises this exception.


Proposed Change
===

The semantics of the ``with`` statement will be changed to include a
new ``try``/``except``/``else`` block around the call to
``__enter__()``.
If ``SkipStatement`` is raised by the ``__enter__()`` method, then
the main section of the ``with`` statement (now located in the
``else``
clause) will not be executed. To avoid leaving the names in any ``as``
clause unbound in this case, a new ``StatementSkipped`` singleton
(similar to the existing ``NotImplemented`` singleton) will be
assigned to all names that appear in the ``as`` clause.

The components of the ``with`` statement remain as described in
PEP 343 [2]::

   with EXPR as VAR:
   BLOCK

After the modification, the ``with`` statement semantics would
be as follows::

   mgr = (EXPR)
   exit = mgr.__exit__  # Not calling it yet
   try:
   value = mgr.__enter__()
   except SkipStatement:
   VAR = StatementSkipped
   # Only if "as VAR" is present and
   # VAR is a single name
   # If VAR is a tuple of names, then StatementSkipped
   # will be assigned to each name in the tuple
   else:
   exc = True
   try:
   try:
   VAR = value  # Only if "as VAR" is present
   BLOCK
   except:
   # The exceptional case is handled here
   exc = False
   if not exit(*sys.exc_info()):
   raise
   # The exception is swallowed if exit() returns true
   finally:
   # The normal and non-local-goto cases are handled here
   if exc:
   exit(None, None, None)

With the above change in place for the ``with`` statement semantics,
``contextlib.contextmanager()`` will then be modified to raise
``SkipStatement`` instead of ``RuntimeError`` when the underlying
generator doesn't yield.


Rationale for Change


Currentl

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Steven Bethard]

On Sun, Mar 15, 2009 at 10:50 AM, Michael Foord
 wrote:
> Brett Cannon wrote:
>>
>> Without knowing what StatementSkipped is (just some singleton? If so why
>> not just used SkipStatement instance that was raised?) and wondering if we
>> are just going to continue to adding control flow exceptions that directly
>> inherit from BaseException or some ControlFlowException base class, the
>> basic idea seems fine by me.
>>
>
> Note that using exceptions for control flow can  be bad for other
> implementations of Python. For example exceptions on the .NET framework are
> very expensive. (Although there are workarounds such as not really raising
> the exception - but they're ugly).
>
> Isn't it better practise for exceptions to be used for exceptional
> circumstances rather than for control flow?

If my understanding is correct, the primary use case for this is when
an exception is raised by an __enter__() method and caught by an
enclosing __exit__() method. So at least in that case, you've already
incurred the cost of an exception.

It might be nice to see an example of this being used with only a
single context manager. Is that possible?

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
--- Bucky Katt, Get Fuzzy
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[glyph]

On 12:56 pm, ncogh...@gmail.com wrote:

PEP 377 is a proposal to allow context manager __enter__() methods to
skip the body of the with statement by raising a specific (new) flow
control exception.

Since there is a working reference implementation now, I thought it was
time to open it up for broader discussion.


Why not allow a context manager to implement some other method, for the 
sake of argument let's say "__start__", which was invoked with a 
callable object and could choose to evaluate or not evaluate the 
statement body by simply not calling that object (or perhaps iterable, 
in the case of a generator)?


This PEP proposes that we have two ways to deal with the body of a 
'with' statement: either the body is run or not.  I have always wanted 
to have another option: run the body later.


Passing around an object representing the body of the with statement 
would allow for this use-case, as well as removing the ugly protrusion 
of yet another control-flow exception (which, as has already been noted, 
creates difficulties for some other python implementations).

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Martin v. Löwis]

> Note that using exceptions for control flow can  be bad for other
> implementations of Python. For example exceptions on the .NET framework
> are very expensive.

Why do you say that? What specific implementation of .NET are you
referring to? What do you mean by "very"?

> Isn't it better practise for exceptions to be used for exceptional
> circumstances rather than for control flow?

This is an ongoing debate (in Python, and outside). I'm in the camp
that says that exceptions are a control flow mechanism just like
loops, conditionals, and recursion. With exceptions, you get essentially
multiple alternative outcomes of a function call, rather than just a
single result. In principle, it would be possible to eliminate the
return statement altogether, but it is useful syntactic sugar.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Jim Baker]

For Jython, this proposal would not present any problems. Exceptions are in
any event of lower cost than for CPython.

Given that we have now adopted Python bytecode for various scenarios where
we cannot compile to Java bytecode, it would be nice to track any changes in
the VM such as the proposed SETUP_WITH opcode. But I'm sure we'll continue
to diff ceval.c, etc. (Consider this request perhaps fodder for the language
summit?)

- Jim

On Sun, Mar 15, 2009 at 12:37 PM,  wrote:

>
> On 12:56 pm, ncogh...@gmail.com wrote:
>
>> PEP 377 is a proposal to allow context manager __enter__() methods to
>> skip the body of the with statement by raising a specific (new) flow
>> control exception.
>>
>> Since there is a working reference implementation now, I thought it was
>> time to open it up for broader discussion.
>>
>
> Why not allow a context manager to implement some other method, for the
> sake of argument let's say "__start__", which was invoked with a callable
> object and could choose to evaluate or not evaluate the statement body by
> simply not calling that object (or perhaps iterable, in the case of a
> generator)?
>
> This PEP proposes that we have two ways to deal with the body of a 'with'
> statement: either the body is run or not.  I have always wanted to have
> another option: run the body later.
>
> Passing around an object representing the body of the with statement would
> allow for this use-case, as well as removing the ugly protrusion of yet
> another control-flow exception (which, as has already been noted, creates
> difficulties for some other python implementations).
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com
>



-- 
Jim Baker
jba...@zyasoft.com
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Michael Foord]

Martin v. Löwis wrote:

Note that using exceptions for control flow can  be bad for other
implementations of Python. For example exceptions on the .NET framework
are very expensive.



Why do you say that? What specific implementation of .NET are you
referring to? What do you mean by "very"?
  
I'm talking about IronPython on the Microsoft .NET framework - although 
it is likely that the same is true of IronPython on Mono.


On the .NET framework the setup for exception handling is virtually free 
until an exception is raised. Once an exception is raised it takes a lot 
longer (expensive in time). This means that in IronPython exception 
handling code (try... except and try... finally blocks) are much faster 
than CPython if no exception is raised - but much more expensive if an 
exception is raised.


You can see this in a comparison of IronPython 2 and Python 2.5 running 
PyBench:


http://ironpython.codeplex.com/Wiki/View.aspx?title=IP201VsCPy25Perf

TryExcept:  26ms  888ms  -97.1%  63ms  890ms  -92.9%
TryRaiseExcept: 58234ms 1286ms +4428.6% 58474ms 
1298ms +4404.6%



Isn't it better practise for exceptions to be used for exceptional
circumstances rather than for control flow?



This is an ongoing debate (in Python, and outside). I'm in the camp
that says that exceptions are a control flow mechanism just like
loops, conditionals, and recursion. With exceptions, you get essentially
multiple alternative outcomes of a function call, rather than just a
single result. In principle, it would be possible to eliminate the
return statement altogether, but it is useful syntactic sugar.
  


Using exceptions for control flow is akin to goto. Sometimes useful but 
a dubious practise. :-)


Michael



Regards,
Martin
  



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Nick Coghlan]

Michael Foord wrote:
> Brett Cannon wrote:
>> Without knowing what StatementSkipped is (just some singleton? If so
>> why not just used SkipStatement instance that was raised?)

It does get described in the full PEP - it is indeed just a singleton
like NotImplemented. That whole aspect of the PEP is something I'm not
entirely sure of at this stage - it may make more sense to just leave
the variables unbound. If a particular program cares to tell whether or
not the statement was skipped, that would still be easy enough to do:

  x = skipped = object()
  with cm() as x:
do_stuff()

  if x is skipped:
print "CM aborted with statement!"

Actually, now that I see how easy it is to do something equivalent for
yourself, I'm definitely going to drop the StatementSkipped assignment
part of the PEP in favour of just skipping over the variable assignment
part as well.

>> and
>> wondering if we are just going to continue to adding control flow
>> exceptions that directly inherit from BaseException or some
>> ControlFlowException base class, the basic idea seems fine by me.

Given how different the control flow exceptions are from each other, I
don't think it really makes sense to ever treat them as a cohesive
group. There's also the fact that StopIteration is a control flow
exception that doesn't inherit directly from BaseException.

> Note that using exceptions for control flow can  be bad for other
> implementations of Python. For example exceptions on the .NET framework
> are very expensive. (Although there are workarounds such as not really
> raising the exception - but they're ugly).

Is it that exceptions are expensive, or setting up a try/except block is
expensive? The reason the SkipStatement idea is tenable at all (even in
CPython) is that try/except is fairly cheap when no exception is raised.

(In this specific case, my initial patch does slow things down a bit,
since one side effect of the extra try/except block is to disallow a
couple of stack based optimisations that are used in the current CPython
implementation of the with statement)

> Isn't it better practise for exceptions to be used for exceptional
> circumstances rather than for control flow?

This *is* an exceptional circumstance: a typical __enter__ method will
just return or raise some other exception. I suppose you could use some
kind of dedicated thread-local state instead of an exception to indicate
that the underlying generator didn't yield, but a new control flow
exception seemed like the most straightforward option.

I'm somewhat intrigued by Glyph's idea though - if I can figure out a
way to make it practical, it does offer very some interesting
possibilities (and would, in effect, bring reusable embedded code blocks
to Python...).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Michael Foord]

Nick Coghlan wrote:

Note that using exceptions for control flow can  be bad for other
implementations of Python. For example exceptions on the .NET framework
are very expensive. (Although there are workarounds such as not really
raising the exception - but they're ugly).



Is it that exceptions are expensive, or setting up a try/except block is
expensive? The reason the SkipStatement idea is tenable at all (even in
CPython) is that try/except is fairly cheap when no exception is raised.
  


It is the raising of the exception that is expensive.

Michael

(In this specific case, my initial patch does slow things down a bit,
since one side effect of the extra try/except block is to disallow a
couple of stack based optimisations that are used in the current CPython
implementation of the with statement)

  

Isn't it better practise for exceptions to be used for exceptional
circumstances rather than for control flow?



This *is* an exceptional circumstance: a typical __enter__ method will
just return or raise some other exception. I suppose you could use some
kind of dedicated thread-local state instead of an exception to indicate
that the underlying generator didn't yield, but a new control flow
exception seemed like the most straightforward option.

I'm somewhat intrigued by Glyph's idea though - if I can figure out a
way to make it practical, it does offer very some interesting
possibilities (and would, in effect, bring reusable embedded code blocks
to Python...).

Cheers,
Nick.

  



--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Nick Coghlan]

gl...@divmod.com wrote:
> 
> On 12:56 pm, ncogh...@gmail.com wrote:
>> PEP 377 is a proposal to allow context manager __enter__() methods to
>> skip the body of the with statement by raising a specific (new) flow
>> control exception.
>>
>> Since there is a working reference implementation now, I thought it was
>> time to open it up for broader discussion.
> 
> Why not allow a context manager to implement some other method, for the
> sake of argument let's say "__start__", which was invoked with a
> callable object and could choose to evaluate or not evaluate the
> statement body by simply not calling that object (or perhaps iterable,
> in the case of a generator)?

So the with statement would in effect create a separate code object for
the statement body that still shared the scope of the containing
function, and then pass a zero-argument callable in to the new method to
allow it to execute that code?

There are some practical hurdles to that idea (specifically, creating a
callable which uses its parent's namespace rather than having its own),
but the basic concept seems sound.

Rough spec for the concept:

Implementing __enter__/__exit__ on a CM would work as per PEP 343.

Implementing __with__ instead would give the CM complete control over
whether or not to execute the block.

The implementation of contextlib.GeneratorContextManager would then
change so that instead of providing __enter__/__exit__ as it does now it
would instead provide __with__ as follows:

  def __with__(self, exec_block):
try:
  return self.gen.next()
except StopIteration:
  pass
else:
  try:
exec_block()
  except:
exc_type, value, traceback = sys.exc_info()
try:
  self.gen.throw(type, value, traceback)
  raise RuntimeError("generator didn't stop after throw()")
except StopIteration, exc:
  # Suppress the exception *unless* it's the same exception that
  # was passed to throw().  This prevents a StopIteration
  # raised inside the "with" statement from being suppressed
  return exc is not value
except:
  # only re-raise if it's *not* the exception that was
  # passed to throw(), because __exit__() must not raise
  # an exception unless __exit__() itself failed.  But throw()
  # has to raise the exception to signal propagation, so this
  # fixes the impedance mismatch between the throw() protocol
  # and the __exit__() protocol.
  if sys.exc_info()[1] is not value:
raise
  else:
try:
  self.gen.next()
except StopIteration:
  return
else:
  raise RuntimeError("generator didn't stop")

More radical in some ways that what I was suggesting, but also cleaner
and more powerful.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Martin v. Löwis]

> I'm talking about IronPython on the Microsoft .NET framework - although
> it is likely that the same is true of IronPython on Mono.

I see. It would be interesting to find out why this is so much slower -
I cannot believe that it is inherent in the commercial .NET framework,
but rather expect that it is some issue in IronPython (*). Also, the
test case measured doesn't entirely reflect what is proposed, since it
catches the exception in the same function - for a realistic comparison,
the raise should occur in a function call (so to measure the overhead
of stack unwinding also).

Regards,
Martin

(*) My wild guess is that IronPython feels obliged to provide traceback
objects, and that this a costly operation - I just can't believe that
exceptions are themselves costly on .NET, in the Microsoft
implementation. In the specific case, it would be possible to suppress
traceback generation.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Nick Coghlan]

Michael Foord wrote:
> Nick Coghlan wrote:
>>> Note that using exceptions for control flow can  be bad for other
>>> implementations of Python. For example exceptions on the .NET framework
>>> are very expensive. (Although there are workarounds such as not really
>>> raising the exception - but they're ugly).
>>> 
>>
>> Is it that exceptions are expensive, or setting up a try/except block is
>> expensive? The reason the SkipStatement idea is tenable at all (even in
>> CPython) is that try/except is fairly cheap when no exception is raised.
>>   
> 
> It is the raising of the exception that is expensive.

Then that isn't a huge drawback in this case - the SkipStatement
exception is only used in situations which would currently probably be
handled by raising an exception anyway (e.g. the change to
contextlib.contextmanager.__enter__() in the patch is to raise
SkipStatement where it currently raises RuntimeError).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Nick Coghlan]

Nick Coghlan wrote:
> Rough spec for the concept:
> 
> Implementing __enter__/__exit__ on a CM would work as per PEP 343.
> 
> Implementing __with__ instead would give the CM complete control over
> whether or not to execute the block.
> 
> The implementation of contextlib.GeneratorContextManager would then
> change so that instead of providing __enter__/__exit__ as it does now it
> would instead provide __with__ as follows:

Expansion in the previous message wasn't quite right since it didn't
give the executed block access to the result of __enter__().

Mark II:

  def __with__(self, exec_block):
try:
  enter_result = self.gen.next()
except StopIteration:
  pass
else:
  try:
exec_block(enter_result)
  except:
exc_type, value, traceback = sys.exc_info()
try:
  self.gen.throw(type, value, traceback)
  raise RuntimeError("generator didn't stop after throw()")
except StopIteration, exc:
  # Suppress the exception *unless* it's the same exception that
  # was passed to throw().  This prevents a StopIteration
  # raised inside the "with" statement from being suppressed
  return exc is not value
except:
  # only re-raise if it's *not* the exception that was
  # passed to throw(), because __exit__() must not raise
  # an exception unless __exit__() itself failed.  But throw()
  # has to raise the exception to signal propagation, so this
  # fixes the impedance mismatch between the throw() protocol
  # and the __exit__() protocol.
  if sys.exc_info()[1] is not value:
raise
  else:
try:
  self.gen.next()
except StopIteration:
  return
else:
  raise RuntimeError("generator didn't stop")


-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Michael Foord]

Martin v. Löwis wrote:

I'm talking about IronPython on the Microsoft .NET framework - although
it is likely that the same is true of IronPython on Mono.



I see. It would be interesting to find out why this is so much slower -
I cannot believe that it is inherent in the commercial .NET framework,
but rather expect that it is some issue in IronPython (*). Also, the
test case measured doesn't entirely reflect what is proposed, since it
catches the exception in the same function - for a realistic comparison,
the raise should occur in a function call (so to measure the overhead
of stack unwinding also).

Regards,
Martin

(*) My wild guess is that IronPython feels obliged to provide traceback
objects, and that this a costly operation - I just can't believe that
exceptions are themselves costly on .NET, in the Microsoft
implementation. In the specific case, it would be possible to suppress
traceback generation.
  
I have discussed this issue with the IronPython team several times. They 
say that it is a deliberate design decision in .NET - to minimize the 
cost of exception handling code in the absence of exceptions at the 
expense of slower performance when exceptions are raised.


Googling for ".NET exceptions performance" would seem to confirm that.

Apparently this article on the managed exception model was written by 
one of the core developers:


http://blogs.msdn.com/cbrumme/archive/2003/10/01/51524.aspx

"This is light years away from returning a -1 from your function call. 
Exceptions are inherently non-local, and if there’s an obvious and 
enduring trend for today’s architectures, it’s that you must remain 
local for good performance.


Relative to straight-line local execution, exception performance will 
keep getting worse. Sure, we might dig into our current behavior and 
speed it up a little. But the trend will relentlessly make exceptions 
perform worse.


How do I reconcile the trend to worse performance with our 
recommendation that managed code should use exceptions to communicate 
errors? By ensuring that error cases are exceedingly rare. We used to 
say that exceptions should be used for exceptional cases, but folks 
pushed back on that as tautological."


Michael

--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[P.J. Eby]

At 06:28 AM 3/16/2009 +1000, Nick Coghlan wrote:

There are some practical hurdles to that idea (specifically, creating a
callable which uses its parent's namespace rather than having its own),
but the basic concept seems sound.


Actually, that bit's pretty simple -- they're just "nonlocal" 
variables.  But the cost of creating that function, and the cost of 
having cell variables in the surrounding function is potentially 
high.  (In CPython at least, function calls are optimized in certain 
ways when the function only has "fast" locals, and no "cell" locals.)


The cost of creating the function (though not the code object) could 
be held till runtime, since the interpreter could wait until it's 
sure there's a __with__ method before doing a MAKE_CLOSURE on the code object.


Of course, at that point, what's the difference between:

with foo() as bar:
baz

and...

@foo
def bar():
   baz

except for being slightly less verbose? (due to missing nonlocal 
statements, etc.)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Ext4 data loss

[Greg Ewing]

Nick Coghlan wrote:


It actually wouldn't be a bad place to put a "create a temporary file
and rename it to  when closing it" helper class.


I'm not sure it would be a good idea to make that
behaviour automatic on closing. If anything goes
wrong while writing the file, you *don't* want the
rename to happen, otherwise it defeats the purpose.

It would be okay to have an explicit close_and_rename()
method, although there wouldn't be much gained over
just calling os.rename() afterwards.

What might make more sense is a context manager,
e.g.

  with renaming_file("blarg.txt", "w") as f:
...

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Aahz]

On Sun, Mar 15, 2009, Michael Foord wrote:
>
> Note that using exceptions for control flow can  be bad for other  
> implementations of Python. For example exceptions on the .NET framework  
> are very expensive. (Although there are workarounds such as not really  
> raising the exception - but they're ugly).
>
> Isn't it better practise for exceptions to be used for exceptional  
> circumstances rather than for control flow?

It seems to me that we as a development community already made a decision
when we switched to StopIteration as the primary mechanism for halting
``for`` loops.  (Not that it was really a new decision because parts of
the Python community have always advocated using exceptions for control
flow, but the ``for`` loop enshrines it.)  I doubt that using exceptions
for control flow in ``with`` blocks will cause anywhere near so much a
performance degradation.
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Michael Foord]

Aahz wrote:

On Sun, Mar 15, 2009, Michael Foord wrote:
  
Note that using exceptions for control flow can  be bad for other  
implementations of Python. For example exceptions on the .NET framework  
are very expensive. (Although there are workarounds such as not really  
raising the exception - but they're ugly).


Isn't it better practise for exceptions to be used for exceptional  
circumstances rather than for control flow?



It seems to me that we as a development community already made a decision
when we switched to StopIteration as the primary mechanism for halting
``for`` loops.  (Not that it was really a new decision because parts of
the Python community have always advocated using exceptions for control
flow, but the ``for`` loop enshrines it.)  I doubt that using exceptions
for control flow in ``with`` blocks will cause anywhere near so much a
performance degradation.
  
Well, StopIteration is still an implementation detail that only 
occasionally bleeds through to actual programming. It says nothing about 
whether using exceptions for non-exceptional circumstances (control 
flow) is good practise. Personally I think it makes the intent of code 
less easy to understand - in effect the exceptions *are* being used as a 
goto.


Michael

--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Ext4 data loss

[Mikko Ohtamaa]

>
>
> > Ok. In that use case, however, it is completely irrelevant whether the
> > tempfile module calls fsync. After it has generated the non-conflicting
> > filename, it's done.
>
> I agree, but my comment was that it would be nice if better fsync
> support (if added) could be done in such a way that it helped not only
> file objects, but also *file-like* objects (such as the wrappers in the
> tempfile module) without making the file-like API any fatter.


fsync() might not be "the answer".

I found this blog post very entertaining to read:
http://www.advogato.org/person/mjg59/diary.html?start=195

"So, on the one hand, we're trying to use things like relatime to batch data
to reduce the amount of time a disk has to be spun up. And on the other
hand, we're moving to filesystems that require us to generate *more* io in
order to guarantee that our data hits disk, which is a guarantee we often
don't want anyway! Users will be fine with losing their most recent changes
to preferences if a machine crashes. They will not be fine with losing the
entirity of their preferences. Arguing that applications need to use *fsync*()
and are otherwise broken is ignoring the important difference between these
use cases. It's no longer going to be possible to spin down a disk when any
software is running at all, since otherwise it's probably going to write
something and then have to *fsync* it out of sheer paranoia that something
bad will happen. And then probably *fsync* the directory as well, because
what if someone writes an even more pathological filesystem. And the disks
sit there spinning gently and chitter away as they write tiny files[4] and
never spin down and the polar bears all drown in the bitter tears of
application developers who are forced to drink so much to forget that they
all die of acute liver failure by the age of 35 and where are we then oh yes
we're fucked."

-M
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Aahz]

On Sun, Mar 15, 2009, Michael Foord wrote:
> Aahz wrote:
>> On Sun, Mar 15, 2009, Michael Foord wrote:
>>>   
>>> Note that using exceptions for control flow can  be bad for other   
>>> implementations of Python. For example exceptions on the .NET 
>>> framework  are very expensive. (Although there are workarounds such 
>>> as not really  raising the exception - but they're ugly).
>>>
>>> Isn't it better practise for exceptions to be used for exceptional   
>>> circumstances rather than for control flow?
>>
>> It seems to me that we as a development community already made a decision
>> when we switched to StopIteration as the primary mechanism for halting
>> ``for`` loops.  (Not that it was really a new decision because parts of
>> the Python community have always advocated using exceptions for control
>> flow, but the ``for`` loop enshrines it.)  I doubt that using exceptions
>> for control flow in ``with`` blocks will cause anywhere near so much a
>> performance degradation.
>   
> Well, StopIteration is still an implementation detail that only
> occasionally bleeds through to actual programming. It says nothing
> about whether using exceptions for non-exceptional circumstances
> (control flow) is good practise. Personally I think it makes the
> intent of code less easy to understand - in effect the exceptions
> *are* being used as a goto.

Let me know how you'd rewrite this more clearly without a control-flow
exception:

try:
for field in curr_fields:
for item in record[field]:
item = item.lower()
for filter in excludes:
if match(item, filter):
raise Excluded
except Excluded: 
continue 

This is pretty much the canonical example showing why control-flow
exceptions are a Good Thing.  They're a *structured* goto.
-- 
Aahz (a...@pythoncraft.com)   <*> http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Michael Foord]

Aahz wrote:

On Sun, Mar 15, 2009, Michael Foord wrote:
  

Aahz wrote:


On Sun, Mar 15, 2009, Michael Foord wrote:
  
  
Note that using exceptions for control flow can  be bad for other   
implementations of Python. For example exceptions on the .NET 
framework  are very expensive. (Although there are workarounds such 
as not really  raising the exception - but they're ugly).


Isn't it better practise for exceptions to be used for exceptional   
circumstances rather than for control flow?


It seems to me that we as a development community already made a decision
when we switched to StopIteration as the primary mechanism for halting
``for`` loops.  (Not that it was really a new decision because parts of
the Python community have always advocated using exceptions for control
flow, but the ``for`` loop enshrines it.)  I doubt that using exceptions
for control flow in ``with`` blocks will cause anywhere near so much a
performance degradation.
  
  
Well, StopIteration is still an implementation detail that only

occasionally bleeds through to actual programming. It says nothing
about whether using exceptions for non-exceptional circumstances
(control flow) is good practise. Personally I think it makes the
intent of code less easy to understand - in effect the exceptions
*are* being used as a goto.



Let me know how you'd rewrite this more clearly without a control-flow
exception:

try:
for field in curr_fields:
for item in record[field]:
item = item.lower()
for filter in excludes:
if match(item, filter):
raise Excluded
except Excluded: 
continue 


This is pretty much the canonical example showing why control-flow
exceptions are a Good Thing.  They're a *structured* goto.
  


You didn't include all the code - so impossible to match the exact 
semantics. Breaking out of multiple loops with a return is a cleaner way 
to handle it IMHO.



def find_excludes():
   for field in curr_fields:
   for item in record[field]:
   item = item.lower()
   for filter in excludes:
   if match(item, filter):
   return
  
while something:

   find_excludes()

Michael

--
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Terry Reedy]

Aahz wrote:

On Sun, Mar 15, 2009, Michael Foord wrote:

It seems to me that we as a development community already made a decision
when we switched to StopIteration as the primary mechanism for halting
``for`` loops. 


If was previously IndexError that stopped for loops, so that was not new 
;-).



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Tristan Seligmann]

On Mon, Mar 16, 2009 at 1:00 AM, Michael Foord
 wrote:
> You didn't include all the code - so impossible to match the exact
> semantics. Breaking out of multiple loops with a return is a cleaner way to
> handle it IMHO.

I don't really see why this is cleaner; they're both just structured
gotos. Heck, the code even looks practically the same.
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Greg Ewing]

Aahz wrote:


This is pretty much the canonical example showing why control-flow
exceptions are a Good Thing.  They're a *structured* goto.


I'm wondering whether what we really want is something
that actually *is* a structured goto. Or something like
a very light-weight exception that doesn't carry all the
expensive baggage of tracebacks, isinstance() tests for
matching, etc.

Ruby seems to have something like this. It has a
try/rescue/raise mechanism that works like Python's
try/except/raise, but it also has try/catch/throw as
a less expensive alternative for flow control.

A Python version might look like

  Foo = object() # a token to throw

  try:
do_something()
  catch Foo:
someone_threw_a_foo()

  def do_something():
throw Foo

Nothing is instantiated -- the token value itself is
thrown -- and the catch clauses compare it by identity
with candidate values.

There is also no traceback carried by the thrown token.

(Although if something is thrown but not caught, an
exception should be raised at the point of the throw
with an appropriate traceback -- implementation of that
is left as an exercise for the reader.)

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Carl Johnson]

P.J. Eby wrote:

> Of course, at that point, what's the difference between:
>
>    with foo() as bar:
>        baz
>
> and...
>
>   �...@foo
>    def bar():
>       baz
>
> except for being slightly less verbose? (due to missing nonlocal statements,
> etc.)

That's not quite direct translation. Closer would be:

@foo()
def _(bar):
   baz
del _

since the "bar" of the with statement is injected into the namespace
of the block. (Obviously, the foo would have be implemented
differently in the two cases in order to have it operate the same
way.)

I have thought about suggesting adding control flow to the with
statement before, since I think if done properly, it might be able to
negate the desire people have for multi-line lambdas/Ruby-style
blocks, but I do wonder if it would be too confusing to have "with"
sometimes mean "abstracting error handling code" and other times mean
"abstracted control flow." But now it looks like there's no way around
using the with-statement for control flow, since sometimes the block
needs to be skipped anyway. So, I'm +.5 on the idea.

-- Carl
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body

[Martin v. Löwis]

> Well, StopIteration is still an implementation detail that only
> occasionally bleeds through to actual programming. It says nothing about
> whether using exceptions for non-exceptional circumstances (control
> flow) is good practise. Personally I think it makes the intent of code
> less easy to understand - in effect the exceptions *are* being used as a
> goto.

The same can be said about if statements and while loops - they are also
being used "as gotos". The bad thing about the goto statement is that it
allows arbitrary, unstructured control flow. This is unlike if
statements, while loops, and - yes - exceptions. They all provide for
structured control flow. raise, in particular, is no more evil than
break, continue, return, or yield.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com