Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
P.J. Eby wrote: > At 06:28 AM 3/16/2009 +1000, Nick Coghlan wrote: >> There are some practical hurdles to that idea (specifically, creating a >> callable which uses its parent's namespace rather than having its own), >> but the basic concept seems sound. > > Actually, that bit's pretty simple -- they're just "nonlocal" > variables. But the cost of creating that function, and the cost of > having cell variables in the surrounding function is potentially high. > (In CPython at least, function calls are optimized in certain ways when > the function only has "fast" locals, and no "cell" locals.) Some additional complexities occurred to me today - they go by the names "return", "break" and "continue". With the current design those are perfectly well defined inside a with statement, but they pose a pretty serious hurdle for a practical implementation of glyph's idea. So I'm going to stick with the more modest approach of a new control flow exception for PEP 377. I still find the callable block idea somewhat intriguing, but if it ever happens it should be a new construct rather than overloading the meaning of the with statement. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Michael Foord wrote: > Well, StopIteration is still an implementation detail that only > occasionally bleeds through to actual programming. It says nothing about > whether using exceptions for non-exceptional circumstances (control > flow) is good practise. Personally I think it makes the intent of code > less easy to understand - in effect the exceptions *are* being used as a > goto. Note that raising SkipStatement manually is likely to be even rarer than raising StopIteration. Catching it should almost never happen other than implicitly inside a with statement (that's the reason I made it a peer of SystemExit and GeneratorExit rather than a peer of StopIteration). It is primarily proposed as a way for contextlib.contextmanager to tell the interpreter that the underlying generator didn't yield, so the body of the with statement should be skipped completely. It just so happens that manually implemented context managers will also be free to use it if they need to for some reason. An alternative approach worth considering may be to use NotImplemented as a model instead of StopIteration. With that approach, instead of having SkipStatement be an exception, have it be a singleton that can be returned from __enter__ to indicate that the with statement body would be skipped. That has a big advantage over using an exception when it comes to execution speed. The statement semantics in that case would become: mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet value = mgr.__enter__() if value is not SkipStatement: exc = True try: try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None) (keeping in mind that I already plan to change PEP 377 to drop the idea of assigning anything to VAR when the statement body is skipped) The major drawback of that approach is that it becomes a little trickier to write a context manager like nested() correctly - it would need to check all of the __enter__() return values and start unwinding the context manager stack if it encountered SkipStatement. The fix isn't particularly complicated*, but it does contrast with the fact that having SkipStatement as an exception means that the current implementation of nested() will "just work" with the new semantics. Cheers, Nick. * For reference, to support a "SkipStatement as return value" approach the main loop in nested() would have to change from this: for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ vars.append(enter()) exits.append(exit) yield vars To this: for mgr in managers: exit = mgr.__exit__ enter = mgr.__enter__ var = enter() if var is SkipStatement: break vars.append(var) exits.append(exit) else: yield vars As mentioned above, if SkipStatement is an exception then nested() works correctly without any changes. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
Greg Ewing wrote:
> What might make more sense is a context manager,
> e.g.
>
> with renaming_file("blarg.txt", "w") as f:
> ...
As you were describing the problems with "rename on close", I actually
immediately thought of the oft-repeated db transaction commit/rollback
example from PEP 343 :)
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
I have no right to speak because I haven't read through all the details of the proposal, but reading this I am very sad that we have to introduce a whole new exception (and one with special status as well) in order to fix such a niggly corner case of the context manager protocol. Since IIUC the original context manager design was intended to have exactly one yield in the body of the context manager -- can't we just declare fewer (or more) yields an error and rase an appropriate TypeError or something? --Guido On Mon, Mar 16, 2009 at 4:43 AM, Nick Coghlan wrote: > Michael Foord wrote: >> Well, StopIteration is still an implementation detail that only >> occasionally bleeds through to actual programming. It says nothing about >> whether using exceptions for non-exceptional circumstances (control >> flow) is good practise. Personally I think it makes the intent of code >> less easy to understand - in effect the exceptions *are* being used as a >> goto. > > Note that raising SkipStatement manually is likely to be even rarer than > raising StopIteration. Catching it should almost never happen other than > implicitly inside a with statement (that's the reason I made it a peer > of SystemExit and GeneratorExit rather than a peer of StopIteration). > > It is primarily proposed as a way for contextlib.contextmanager to tell > the interpreter that the underlying generator didn't yield, so the body > of the with statement should be skipped completely. It just so happens > that manually implemented context managers will also be free to use it > if they need to for some reason. > > An alternative approach worth considering may be to use NotImplemented > as a model instead of StopIteration. With that approach, instead of > having SkipStatement be an exception, have it be a singleton that can be > returned from __enter__ to indicate that the with statement body would > be skipped. > > That has a big advantage over using an exception when it comes to > execution speed. The statement semantics in that case would become: > > mgr = (EXPR) > exit = mgr.__exit__ # Not calling it yet > value = mgr.__enter__() > if value is not SkipStatement: > exc = True > try: > try: > VAR = value # Only if "as VAR" is present > BLOCK > except: > # The exceptional case is handled here > exc = False > if not exit(*sys.exc_info()): > raise > # The exception is swallowed if exit() returns true > finally: > # The normal and non-local-goto cases are handled here > if exc: > exit(None, None, None) > > (keeping in mind that I already plan to change PEP 377 to drop the idea > of assigning anything to VAR when the statement body is skipped) > > The major drawback of that approach is that it becomes a little trickier > to write a context manager like nested() correctly - it would need to > check all of the __enter__() return values and start unwinding the > context manager stack if it encountered SkipStatement. The fix isn't > particularly complicated*, but it does contrast with the fact that > having SkipStatement as an exception means that the current > implementation of nested() will "just work" with the new semantics. > > Cheers, > Nick. > > * For reference, to support a "SkipStatement as return value" approach > the main loop in nested() would have to change from this: > > for mgr in managers: > exit = mgr.__exit__ > enter = mgr.__enter__ > vars.append(enter()) > exits.append(exit) > yield vars > > To this: > > for mgr in managers: > exit = mgr.__exit__ > enter = mgr.__enter__ > var = enter() > if var is SkipStatement: > break > vars.append(var) > exits.append(exit) > else: > yield vars > > As mentioned above, if SkipStatement is an exception then nested() works > correctly without any changes. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > --- > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Moreover, since the main use case seems to be fixing a corner case of the nested() context manager, perhaps the effort towards changing the language would be better directed towards supporting "with a, b:" as a shorthand for "with a: with b:" . On Mon, Mar 16, 2009 at 10:01 AM, Guido van Rossum wrote: > I have no right to speak because I haven't read through all the > details of the proposal, but reading this I am very sad that we have > to introduce a whole new exception (and one with special status as > well) in order to fix such a niggly corner case of the context manager > protocol. > > Since IIUC the original context manager design was intended to have > exactly one yield in the body of the context manager -- can't we just > declare fewer (or more) yields an error and rase an appropriate > TypeError or something? > > --Guido > > On Mon, Mar 16, 2009 at 4:43 AM, Nick Coghlan wrote: >> Michael Foord wrote: >>> Well, StopIteration is still an implementation detail that only >>> occasionally bleeds through to actual programming. It says nothing about >>> whether using exceptions for non-exceptional circumstances (control >>> flow) is good practise. Personally I think it makes the intent of code >>> less easy to understand - in effect the exceptions *are* being used as a >>> goto. >> >> Note that raising SkipStatement manually is likely to be even rarer than >> raising StopIteration. Catching it should almost never happen other than >> implicitly inside a with statement (that's the reason I made it a peer >> of SystemExit and GeneratorExit rather than a peer of StopIteration). >> >> It is primarily proposed as a way for contextlib.contextmanager to tell >> the interpreter that the underlying generator didn't yield, so the body >> of the with statement should be skipped completely. It just so happens >> that manually implemented context managers will also be free to use it >> if they need to for some reason. >> >> An alternative approach worth considering may be to use NotImplemented >> as a model instead of StopIteration. With that approach, instead of >> having SkipStatement be an exception, have it be a singleton that can be >> returned from __enter__ to indicate that the with statement body would >> be skipped. >> >> That has a big advantage over using an exception when it comes to >> execution speed. The statement semantics in that case would become: >> >> mgr = (EXPR) >> exit = mgr.__exit__ # Not calling it yet >> value = mgr.__enter__() >> if value is not SkipStatement: >> exc = True >> try: >> try: >> VAR = value # Only if "as VAR" is present >> BLOCK >> except: >> # The exceptional case is handled here >> exc = False >> if not exit(*sys.exc_info()): >> raise >> # The exception is swallowed if exit() returns true >> finally: >> # The normal and non-local-goto cases are handled here >> if exc: >> exit(None, None, None) >> >> (keeping in mind that I already plan to change PEP 377 to drop the idea >> of assigning anything to VAR when the statement body is skipped) >> >> The major drawback of that approach is that it becomes a little trickier >> to write a context manager like nested() correctly - it would need to >> check all of the __enter__() return values and start unwinding the >> context manager stack if it encountered SkipStatement. The fix isn't >> particularly complicated*, but it does contrast with the fact that >> having SkipStatement as an exception means that the current >> implementation of nested() will "just work" with the new semantics. >> >> Cheers, >> Nick. >> >> * For reference, to support a "SkipStatement as return value" approach >> the main loop in nested() would have to change from this: >> >> for mgr in managers: >> exit = mgr.__exit__ >> enter = mgr.__enter__ >> vars.append(enter()) >> exits.append(exit) >> yield vars >> >> To this: >> >> for mgr in managers: >> exit = mgr.__exit__ >> enter = mgr.__enter__ >> var = enter() >> if var is SkipStatement: >> break >> vars.append(var) >> exits.append(exit) >> else: >> yield vars >> >> As mentioned above, if SkipStatement is an exception then nested() works >> correctly without any changes. >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> --- >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- --Guido van Rossum (home page: http://www.python.org/~guido/)
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
On Mon, Mar 16, 2009 at 11:06 AM, Guido van Rossum wrote: > Moreover, since the main use case seems to be fixing a corner case of > the nested() context manager, perhaps the effort towards changing the > language would be better directed towards supporting "with a, b:" as a > shorthand for "with a: with b:" . > > On Mon, Mar 16, 2009 at 10:01 AM, Guido van Rossum wrote: >> I have no right to speak because I haven't read through all the >> details of the proposal, but reading this I am very sad that we have >> to introduce a whole new exception (and one with special status as >> well) in order to fix such a niggly corner case of the context manager >> protocol. >> >> Since IIUC the original context manager design was intended to have >> exactly one yield in the body of the context manager -- can't we just >> declare fewer (or more) yields an error and rase an appropriate >> TypeError or something? It's not really a generator specific thing. You can generate similar problems by just defining a class with an __enter__() method that raises an exception. But I agree that it seems like a big change for a small corner case. Is there anything other than contextlib.nested() which needs this? If there is no other use case, then I'm a strong +1 for Guido's suggestion of providing syntactic support for ``with a, b:`` instead. BTW, I think the explanation of the problem isn't as clear as it could be. The core problem, if I understand it right, is that contextlib.nested() is not equivalent to a real nested with statement because it calls the nested __enter__() methods too early. A real nested with statement translates into something like:: mgr1.__enter__() try: mgr2.__enter__() try: BLOCK except: ... except: if not mgr1.__exit__(*sys.exc_info()): raise But contextlib.nested() calls all the __enter__() methods in its own __enter__() so it translates into something like:: mgr1.__enter__() mgr2.__enter__() try: BLOCK except: ... The key problem here is that ``mgr2.__enter__()`` is outside of the try block, and the context manager has no way to put it inside. So the thing that contextlib.nested() really needs is a way to be able to insert statements into the BLOCK part of the code. (I'm not actually suggesting we go this route, but that seems to be what contextlib.nested() is really after.) Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
On Mon, Mar 16, 2009 at 11:26 AM, Steven Bethard wrote: > On Mon, Mar 16, 2009 at 11:06 AM, Guido van Rossum wrote: >> Moreover, since the main use case seems to be fixing a corner case of >> the nested() context manager, perhaps the effort towards changing the >> language would be better directed towards supporting "with a, b:" as a >> shorthand for "with a: with b:" . >> >> On Mon, Mar 16, 2009 at 10:01 AM, Guido van Rossum wrote: >>> I have no right to speak because I haven't read through all the >>> details of the proposal, but reading this I am very sad that we have >>> to introduce a whole new exception (and one with special status as >>> well) in order to fix such a niggly corner case of the context manager >>> protocol. >>> >>> Since IIUC the original context manager design was intended to have >>> exactly one yield in the body of the context manager -- can't we just >>> declare fewer (or more) yields an error and rase an appropriate >>> TypeError or something? > > It's not really a generator specific thing. You can generate similar > problems by just defining a class with an __enter__() method that > raises an exception. Huh? According to PEP 343, if __enter__ raises an exception, that's the end of the story. __exit__ shouldn't be called, the exception is not modified, the flow is interrupted right there. > But I agree that it seems like a big change for a small corner case. > Is there anything other than contextlib.nested() which needs this? If > there is no other use case, then I'm a strong +1 for Guido's > suggestion of providing syntactic support for ``with a, b:`` instead. > > BTW, I think the explanation of the problem isn't as clear as it could > be. The core problem, if I understand it right, is that > contextlib.nested() is not equivalent to a real nested with statement > because it calls the nested __enter__() methods too early. A real > nested with statement translates into something like:: > > mgr1.__enter__() > try: > mgr2.__enter__() > try: > BLOCK > except: > ... > except: > if not mgr1.__exit__(*sys.exc_info()): > raise > > But contextlib.nested() calls all the __enter__() methods in its own > __enter__() so it translates into something like:: > > mgr1.__enter__() > mgr2.__enter__() > try: > BLOCK > except: > ... > > The key problem here is that ``mgr2.__enter__()`` is outside of the > try block, and the context manager has no way to put it inside. So the > thing that contextlib.nested() really needs is a way to be able to > insert statements into the BLOCK part of the code. (I'm not actually > suggesting we go this route, but that seems to be what > contextlib.nested() is really after.) Yeah, it really seems pretty much limited to contextlib.nested(). I'd be happy to sacrifice the possibility to *exactly* emulate two nested with-statements. The cost of a new exception is huge -- everyone will have to explain its existence, and historically "you don't need to know about this little detail" isn't acceptable for Python docs. Little details you don't need to know about add up. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Ext4 data loss
On Mar 15, 2009, at 3:25 PM, Greg Ewing wrote:
with renaming_file("blarg.txt", "w") as f:
...
By chance during the weekend I actually wrote something like that:
from __future__ import with_statement
import os
import codecs
import shutil
import tempfile
from contextlib import contextmanager
TDIR = tempfile.mktemp(dir='/tmp/')
@contextmanager
def topen(filepath, mode='wb', bufsize=-1, encoding=None,
inplace=False, tmpd=TDIR, sync=False):
"""
C{topen} is a transactional version of the Python builtin C{open}
function for file IO. It manages transactionality by using a
temporary file and then moving it to the final position once its
content has been written to disk.
If the mode used to open the file doesn't modify the file, this
function is equivalent to the built-in C{open} with automatic
file closing behavior.
@param filepath: The path of the file that you want to open
@type filepath: C{str}
@param mode: POSIX mode in which you want to open the file.
@type mode: C{str} see documentation for the format.
@param bufsize: Buffer size for file IO
@type bufsize: C{int} see documentation for the meaning
@param encoding: Encoding that should be used to read the file
@type encoding: C{str}
@param inplace: Indicates if the temporary file should reside
in the same directory of the final file.
@type inplace: C{bool}
@param tmpd: The temporary directory in which file IO is
performed. Then files are moved from here to
their original destination.
@type tmpd: C{str}
@param sync: Force topen to fsync the file before closing it
@type sync: C{bool}
"""
if 'r' in mode or 'a' in mode:
fp = filepath
else:
if inplace:
source_dir, _ = os.path.split(filepath)
tmpd = source_dir
if not os.path.exists(tmpd):
os.makedirs(tmpd)
_f, fp = tempfile.mkstemp(dir=tmpd)
if encoding is not None:
f = codecs.open(fp, mode, encoding=encoding)
else:
f = open(fp, mode, bufsize)
try:
yield f
finally:
if 'r' in mode:
if "+" in mode:
f.flush()
if sync:
os.fsync(f.fileno())
f.close()
return
f.flush()
if sync:
os.fsync(f.fileno())
f.close()
if 'w' in mode:
shutil.move(fp, filepath)
if __name__ == "__main__":
with topen("a_test") as f:
f.write("hello")
assert file("a_test", "rb").read() == 'hello'
assert os.path.exists(TDIR)
os.rmdir(TDIR)
with topen("a_test", mode="rb") as f:
assert f.read() == "hello"
assert not os.path.exists(TDIR)
os.remove("a_test")
--
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com
PGP.sig
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Nick Coghlan a écrit : Implementing __with__ instead would give the CM complete control over whether or not to execute the block. please note, however, that this is an important change in the semantics of the with statement. As things are today, barring exceptional circunstances, the body of the with statement *will* be executed immediately. This allows to forget about the with statement when first reading code, as it doesn't change the intent of the programmer. What Glyph is proposing is more akin to Ruby code blocks. cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Guido van Rossum wrote:
> Yeah, it really seems pretty much limited to contextlib.nested(). I'd
> be happy to sacrifice the possibility to *exactly* emulate two nested
> with-statements.
Then I really haven't explained the problem well at all. One of the
premises of PEP 343 was "Got a frequently recurring block of code that
only has one variant sequence of statements somewhere in the middle?
Well, now you can factor that out by putting it in a generator,
replacing the part that varies with a yield statement and decorating the
generator with contextlib.contextmanager."
It turns out that there's a caveat that needs to go on the end of that
though: "Be very, very sure that the yield statement can *never* be
skipped or your context manager based version will raise a RuntimeError
in cases where the original code would have just skipped over the
variant section of code and resumed execution afterwards."
Nested context managers (whether through contextlib.nested or through
syntactic support) just turns out to be a common case where you *don't
necessarily know* just by looking at the code whether it can skip over
the body of the code or not.
Suppose you have 3 context managers that are regularly used together
(call them cmA(), cmB(), cmC() for now).
Writing that as:
with cmA():
with cmB():
with cmC():
do_something()
Or the tentatively proposed:
with cmA(), cmB(), cmC():
do_something()
is definitely OK, regardless of the details of the context managers.
However, whether or not you can bundle that up into a *new* context
manager (regardless of nesting syntax) depends on whether or not an
outer context manager can suppress an exception raised by an inner one.
@contextmanager
def cmABC():
with cmA():
with cmB():
with cmC():
yield
with cmABC():
do_something()
The above is broken if cmB().__enter__() or cmC.__enter__() can raise an
exception that cmA().__exit__() suppresses, or cmB.__enter__() raises an
exception that cmB().__exit__() suppresses. So whereas the inline
versions were clearly correct, the correctness of the second version
currently depends on details of the context managers themselves.
Changing the syntax to allow the three context managers to be written on
one line does nothing to fix that semantic discrepancy between the
original inline code and the factored out version.
PEP 377 is about changing the with statement semantics and the
@contextmanager implementation so that the semantics of the factored out
version actually matches that of the original inline code.
You can get yourself into similar trouble without nesting context
managers - all it takes is some way of skipping the variant code in a
context manager that wouldn't have raised an exception if the code was
written out inline instead of being factored out into the context manager.
Suppose for instance you wanted to use a context manager as a different
way of running tests:
@contextmanager
def inline_test(self, *setup_args):
try:
self.setup(*setup_args)
except:
# Setup exception occurred, trap it and log it
return
try:
yield
except:
# Test exception occurred, trap it and log it
finally:
try:
self.teardown()
except:
# Teardown exception occurred, trap it and log it
with inline_test(setup1):
test_one()
with inline_test(setup2):
test_two()
with inline_test(setup3):
test_three()
That approach isn't actually valid - a context manager is not permitted
to decide in it's __enter__() method that executing the body of the with
statement would be a bad idea.
The early return in the above makes it obvious that that CM is broken
under the current semantics, but what about the following one:
@contextmanager
def broken_cm(self):
try:
call_user_setup()
try:
yield
finally:
call_user_teardown()
except UserCancel:
show_message("Operation aborted by user")
That CM will raise RuntimeError if the user attempts to cancel an
operation during the execution of the "call_user_setup()" method.
Without SkipStatement or something like it, that can't be fixed.
Hell, I largely wrote PEP 377 to try to get out of having to document
these semantic problems with the with statement - if I'm having trouble
getting *python-dev* to grasp the problem, what hope do other users of
Python have?
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
On Mon, Mar 16, 2009 at 2:37 PM, Nick Coghlan wrote:
> Guido van Rossum wrote:
>> Yeah, it really seems pretty much limited to contextlib.nested(). I'd
>> be happy to sacrifice the possibility to *exactly* emulate two nested
>> with-statements.
>
> Then I really haven't explained the problem well at all. One of the
> premises of PEP 343 was "Got a frequently recurring block of code that
> only has one variant sequence of statements somewhere in the middle?
> Well, now you can factor that out by putting it in a generator,
> replacing the part that varies with a yield statement and decorating the
> generator with contextlib.contextmanager."
>
> It turns out that there's a caveat that needs to go on the end of that
> though: "Be very, very sure that the yield statement can *never* be
> skipped or your context manager based version will raise a RuntimeError
> in cases where the original code would have just skipped over the
> variant section of code and resumed execution afterwards."
Well, I don't think you can take that premise (promise? :-) literally
anyways, since you cannot turn a loop into a with-statement either. So
I would be fine with just adding the condition that the variant
sequence should be executed exactly once.
> Nested context managers (whether through contextlib.nested or through
> syntactic support) just turns out to be a common case where you *don't
> necessarily know* just by looking at the code whether it can skip over
> the body of the code or not.
>
> Suppose you have 3 context managers that are regularly used together
> (call them cmA(), cmB(), cmC() for now).
>
> Writing that as:
>
> with cmA():
> with cmB():
> with cmC():
> do_something()
>
> Or the tentatively proposed:
>
> with cmA(), cmB(), cmC():
> do_something()
>
> is definitely OK, regardless of the details of the context managers.
>
> However, whether or not you can bundle that up into a *new* context
> manager (regardless of nesting syntax) depends on whether or not an
> outer context manager can suppress an exception raised by an inner one.
>
> �...@contextmanager
> def cmABC():
> with cmA():
> with cmB():
> with cmC():
> yield
>
> with cmABC():
> do_something()
While all this may make sense to the original inventor of context
managers (== you ;-), I personally find this example quite perverse.
Do you have an example taken from real life?
> The above is broken if cmB().__enter__() or cmC.__enter__() can raise an
> exception that cmA().__exit__() suppresses, or cmB.__enter__() raises an
> exception that cmB().__exit__() suppresses. So whereas the inline
> versions were clearly correct, the correctness of the second version
> currently depends on details of the context managers themselves.
> Changing the syntax to allow the three context managers to be written on
> one line does nothing to fix that semantic discrepancy between the
> original inline code and the factored out version.
I think I understand that, I just don't see a use case so important as
to warrant introducing a brand new exception deriving from
BaseException.
> PEP 377 is about changing the with statement semantics and the
> @contextmanager implementation so that the semantics of the factored out
> version actually matches that of the original inline code.
>
> You can get yourself into similar trouble without nesting context
> managers - all it takes is some way of skipping the variant code in a
> context manager that wouldn't have raised an exception if the code was
> written out inline instead of being factored out into the context manager.
Yeah, see above -- you can't write a context manager that implements a
loop either.
> Suppose for instance you wanted to use a context manager as a different
> way of running tests:
>
> �...@contextmanager
> def inline_test(self, *setup_args):
> try:
> self.setup(*setup_args)
> except:
> # Setup exception occurred, trap it and log it
> return
> try:
> yield
> except:
> # Test exception occurred, trap it and log it
> finally:
> try:
> self.teardown()
> except:
> # Teardown exception occurred, trap it and log it
>
> with inline_test(setup1):
> test_one()
> with inline_test(setup2):
> test_two()
> with inline_test(setup3):
> test_three()
>
> That approach isn't actually valid
(but your proposal would make it valid right?)
> - a context manager is not permitted
> to decide in it's __enter__() method that executing the body of the with
> statement would be a bad idea.
A setup failure sounds like a catastrophic error to me.
> The early return in the above makes it obvious that that CM is broken
> under the current semantics, but what about the following one:
>
> �...@contextmanager
> def broken_cm(self):
> try:
> call_user_setup()
> try:
> yield
> finally:
> call_user_teardown()
> except UserCancel:
> show_message("Operation aborted by user")
>
> T
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Baptiste Carvello wrote: > Nick Coghlan a écrit : >> >> Implementing __with__ instead would give the CM complete control over >> whether or not to execute the block. >> > please note, however, that this is an important change in the semantics > of the with statement. As things are today, barring exceptional > circunstances, the body of the with statement *will* be executed > immediately. This allows to forget about the with statement when first > reading code, as it doesn't change the intent of the programmer. What > Glyph is proposing is more akin to Ruby code blocks. Yep - the idea is intriguing, powerful, and far beyond the scope of what I'm trying to achieve with PEP 377. I suspect such blocks would have to be more along the lines of what PJE suggested in order to be practical anyway (i.e. nested functions where all "local" variables were implicitly declared as nonlocal), which then leads to the question of how much of a gain they would actually provide now that you *can* explicitly declare nonlocal variables. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Guido van Rossum wrote: >> Hell, I largely wrote PEP 377 to try to get out of having to document >> these semantic problems with the with statement - if I'm having trouble >> getting *python-dev* to grasp the problem, what hope do other users of >> Python have? > > Hell, if you can't come up with a real use case, why bother? :-) I figured I'd try for a solution that didn't offend my sense of aesthetics before caving in and working out how to better document the limitations of the status quo :) > Perhaps you could address my worry about introducing an obscure > BaseException subclass that will forever add to the weight of the list > of built-in exceptions in all documentation? Since this is really just a matter of the aesthetics of the underlying design from my point of view rather than solving a real world problem, I don't have a good answer for you. In the absence of reports of actual problems caused by this limitation, should I consider the PEP rejected? Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
On Mon, Mar 16, 2009 at 3:19 PM, Nick Coghlan wrote: > Guido van Rossum wrote: >>> Hell, I largely wrote PEP 377 to try to get out of having to document >>> these semantic problems with the with statement - if I'm having trouble >>> getting *python-dev* to grasp the problem, what hope do other users of >>> Python have? >> >> Hell, if you can't come up with a real use case, why bother? :-) > > I figured I'd try for a solution that didn't offend my sense of > aesthetics before caving in and working out how to better document the > limitations of the status quo :) > >> Perhaps you could address my worry about introducing an obscure >> BaseException subclass that will forever add to the weight of the list >> of built-in exceptions in all documentation? > > Since this is really just a matter of the aesthetics of the underlying > design from my point of view rather than solving a real world problem, I > don't have a good answer for you. > > In the absence of reports of actual problems caused by this limitation, > should I consider the PEP rejected? Yes -- sorry for your wasted efforts. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] What level of detail wanted for import and the language reference?
At this point importlib is done for its public API for Python 3.1. That means it's time to turn my attention to making sure the semantics of import are well documented. But where to put all of the details? The language reference for import ( http://docs.python.org/dev/py3k/reference/simple_stmts.html#the-import-statement) explains the basics, but is lacking all of the details of PEP 302 and other stuff like __path__ that have existed for ages. My question is if I should flesh out the details in the language reference or do it in importlib's intro docs. The main reason I could see not doing it in the langauge reference (or at least duplicating it) is it would be somewhat easier to reference specific objects in importlib but I am not sure if the language reference should try to stay away from stdlib references. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Guido van Rossum wrote: > On Mon, Mar 16, 2009 at 3:19 PM, Nick Coghlan wrote: >> Guido van Rossum wrote: Hell, I largely wrote PEP 377 to try to get out of having to document these semantic problems with the with statement - if I'm having trouble getting *python-dev* to grasp the problem, what hope do other users of Python have? >>> Hell, if you can't come up with a real use case, why bother? :-) >> I figured I'd try for a solution that didn't offend my sense of >> aesthetics before caving in and working out how to better document the >> limitations of the status quo :) >> >>> Perhaps you could address my worry about introducing an obscure >>> BaseException subclass that will forever add to the weight of the list >>> of built-in exceptions in all documentation? >> Since this is really just a matter of the aesthetics of the underlying >> design from my point of view rather than solving a real world problem, I >> don't have a good answer for you. >> >> In the absence of reports of actual problems caused by this limitation, >> should I consider the PEP rejected? > > Yes -- sorry for your wasted efforts. Not wasted - I prefer having this as a recognised limitation of the semantics rather than as an accident of the implementation. Who knows, maybe somebody will come up with a real world use case some day and we can drag the PEP out and dust it off a bit :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What level of detail wanted for import and the language reference?
Brett Cannon wrote: At this point importlib is done for its public API for Python 3.1. That means it's time to turn my attention to making sure the semantics of import are well documented. But where to put all of the details? The language reference for import (http://docs.python.org/dev/py3k/reference/simple_stmts.html#the-import-statement) explains the basics, but is lacking all of the details of PEP 302 and other stuff like __path__ that have existed for ages. My question is if I should flesh out the details in the language reference or do it in importlib's intro docs. The main reason I could see not doing it in the langauge reference (or at least duplicating it) is it would be somewhat easier to reference specific objects in importlib but I am not sure if the language reference should try to stay away from stdlib references. Having the Python import semantics well documented will be an *excellent* side effect of importlib. Thank you for your astonishing efforts on this project. Personally I would rather see the import semantics themselves in the language reference. Michael -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What level of detail wanted for import and the language reference?
2009/3/16 Brett Cannon : > At this point importlib is done for its public API for Python 3.1. That > means it's time to turn my attention to making sure the semantics of import > are well documented. But where to put all of the details? The language > reference for import > (http://docs.python.org/dev/py3k/reference/simple_stmts.html#the-import-statement) > explains the basics, but is lacking all of the details of PEP 302 and other > stuff like __path__ that have existed for ages. > > My question is if I should flesh out the details in the language reference > or do it in importlib's intro docs. The main reason I could see not doing it > in the langauge reference (or at least duplicating it) is it would be > somewhat easier to reference specific objects in importlib but I am not sure > if the language reference should try to stay away from stdlib references. Thanks so much for doing this! Personally, I think you should put it in the language reference. (I think it deserves it's own file if it's as big as I suspect it will be.) If I wanted to use importlib, I wouldn't really want to slog through a in-depth description of how exactly import works. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What level of detail wanted for import and the language reference?
On Mon, Mar 16, 2009, Michael Foord wrote: > > Personally I would rather see the import semantics themselves in the > language reference. Either way is fine with me, but it needs to be cross-referenced. -- Aahz (a...@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
Nick Coghlan wrote: One of the premises of PEP 343 was "Got a frequently recurring block of code that only has one variant sequence of statements somewhere in the middle? Well, now you can factor that out Um, no -- it says explicitly right at the very top of PEP 343 that it's only about factoring out try/finally statements. There's no way that try: code_block finally: ... can fail to enter the code block if you get as far as the "try". So it's not reasonable to expect the with statement to provide this ability. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What level of detail wanted for import and the language reference?
Brett Cannon wrote: > At this point importlib is done for its public API for Python 3.1. That > means it's time to turn my attention to making sure the semantics of > import are well documented. But where to put all of the details? The > language reference for import > (http://docs.python.org/dev/py3k/reference/simple_stmts.html#the-import-statement) > explains the basics, but is lacking all of the details of PEP 302 and > other stuff like __path__ that have existed for ages. > > My question is if I should flesh out the details in the language > reference or do it in importlib's intro docs. The main reason I could > see not doing it in the langauge reference (or at least duplicating it) > is it would be somewhat easier to reference specific objects in > importlib but I am not sure if the language reference should try to stay > away from stdlib references. Why not just put a section in both places that says "can't be bothered to spell this out right now" and put a URL in referring to this thread on Google ... that appears to have been the traditional approach to import semantics :) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Want to know? Come to PyCon - soon! http://us.pycon.org/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 377 - allow __enter__() methods to skip the statement body
On Mar 16, 2009, at 3:40 PM, Nick Coghlan wrote: Not wasted - I prefer having this as a recognised limitation of the semantics rather than as an accident of the implementation. Well, I'm glad some good came from the issue report. =) Who knows, maybe somebody will come up with a real world use case some day and we can drag the PEP out and dust it off a bit :) I don't expect this to be at all compelling, but FWIW: The identification of this issue came from an *experiment* attempting to create a *single* "daemonized()" CM that would execute the with- statement's block in a new child process and, of course, not execute it in the parent. At first, I ran into the RuntimeError in the parent process, and then after rewriting the CMs as classes, I realized the futility. with daemonized(): run_some_subprocess() Of course it was all possible if I used the component CMs directly: with parent_trap(): with fork_but_raise_in_parent(): run_some_subprocess() And thus: def daemonized(): return contextlib.nested(parent_trap(), fork_but_raise_in_parent()) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Daemon process context as a ‘with ’ context manager (was: PEP 377 - allow __enter __() methods to skip the statement body)
James Pye writes: > The identification of this issue came from an *experiment* attempting > to create a *single* "daemonized()" CM that would execute the with- > statement's block in a new child process and, of course, not execute > it in the parent. At first, I ran into the RuntimeError in the parent > process, and then after rewriting the CMs as classes, I realized the > futility. > > with daemonized(): > run_some_subprocess() This use case is addressed by my up-coming PEP, currently in draft form but submitted to the PEP editor. A reference implementation is at http://pypi.python.org/pypi/python-daemon/>. I'd like to know whether this meets your needs; follow-ups privately to me or in ‘python-ideas’ as you think appropriate. -- \ “Holy tintinnabulation, Batman!” —Robin | `\ | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daemon process context as a ‘with ’ context manager
Ben Finney writes: > James Pye writes: > > > with daemonized(): > > run_some_subprocess() > > This use case is addressed by my up-coming PEP, currently in draft > form but submitted to the PEP editor. A reference implementation is > at http://pypi.python.org/pypi/python-daemon/>. On second blush, it addresses a smaller use case, with the same syntax as your example. It should still be applicable at some level. > I'd like to know whether this meets your needs; follow-ups privately > to me or in ‘python-ideas’ as you think appropriate. Ditto. -- \ “Life does not cease to be funny when people die any more than | `\ it ceases to be serious when people laugh.” —George Bernard Shaw | _o__) | Ben Finney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
