Re: [Python-Dev] PEP 492: async/await in Python; version 4
tds333 gmail.com gmail.com> writes: > > Hi, > > still watching progress here. Read all posts and changes. > > Everything improved and I know it is a lot of work. Thx for doing this. > > But I still think this PEP goes to far. > > [...] > > We forget to address the major problems here. How can someone in a > "sync" script use this async stuff easy. How can async and sync stuff > cooperate and we don't need to rewrite the world for async stuff. > How can a normal user access the power of async stuff without rewriting > all his code. So he can use a simple asyc request library in his code. > How can a normal user learn and use all this in an easy way. > > [...] Hi Wolfgang, You may want to see what I just posted on python-ideas. What I wrote about is related to several things you mention, and might provide a remedy. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v4
On Thu, Sep 7, 2017 at 10:54 AM, Greg Ewing
wrote:
> Yury Selivanov wrote:
>
>> def foo():
>> var = ContextVar()
>> var.set(1)
>>
>> for _ in range(10**6): foo()
>>
>> If 'var' is strongly referenced, we would have a bunch of them.
>>
>
> Erk. This is not how I envisaged context vars would be
> used. What I thought you would do is this:
>
>my_context_var = ContextVar()
>
>def foo():
> my_context_var.set(1)
>
> This problem would also not arise if context vars
> simply had names instead of being magic key objects:
>
>def foo():
> contextvars.set("mymodule.myvar", 1)
>
> That's another thing I think would be an improvement,
> but it's orthogonal to what we're talking about here
> and would be best discussed separately.
>
>
There are lots of things in this discussion that I should have commented
on, but here's one related to this.
PEP 555 does not have the resource-management issue described above and
needs no additional tricks to achieve that:
# using PEP 555
def foo():
var = contextvars.Var()
with var.assign(1):
# do something [*]
for _ in range(10**6):
foo()
Every time foo is called, a new context variable is created, but that's
perfectly fine, and lightweight. As soon as the context manager exits,
there are no references to the Assignment object returned by var.assign(1),
and as soon as foo() returns, there are no references to var, so everything
should get cleaned up nicely.
And regarding string keys, they have pros and cons, and they can be added
easily, so let's not go there now.
-- Koos
[*] (nit-picking) without closures that would keep the var reference alive
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Consolidate stateful runtime globals
On Wed, Sep 6, 2017 at 8:17 PM, Benjamin Peterson wrote: > On Wed, Sep 6, 2017, at 10:08, Antoine Pitrou wrote: > > On Wed, 06 Sep 2017 09:42:29 -0700 > > Benjamin Peterson wrote: > > > On Wed, Sep 6, 2017, at 03:14, Antoine Pitrou wrote: > > > > > > > > Hello, > > > > > > > > I'm a bit concerned about > > > > https://github.com/python/cpython/commit/76d5abc8684bac4f2fc > 7cccfe2cd940923357351 > > > > > > > > My main gripe is that makes writing C code more tedious. Simple C > > > > global variables such as "_once_registry" are now spelled > > > > "_PyRuntime.warnings.once_registry". The most egregious example > seems > > > > to be "_PyRuntime.ceval.gil.locked" (used to be simply "gil_locked"). > > > > > > > > Granted, C is more verbose than Python, but it doesn't have to become > > > > that verbose. I don't know about you, but when code becomes annoying > > > > to type, I tend to try and take shortcuts. > > > > > > How often are you actually typing the names of runtime globals, though? > > > > Not very often, but if I want to experiment with some low-level > > implementation details, it is nice to avoid the hassle. > > It seems like this could be remediated with some inline functions or > macros, which would also help safely encapsulate state. > > > > > There's also a readability argument: with very long names, expressions > > can become less easy to parse. > > > > > If you are using a globals, perhaps the typing time will allow you to > > > fully consider the gravity of the situation. > > > > Right, I needed to be reminded of how perilous the use of C globals is. > > Perhaps I should contact the PSRT the next time I contemplate using a C > > global. > > It's not just you but future readers. Great. Related to this, there is also discussion on dangers of globals and other widely-scoped variables in the Rationale section of PEP 555 (Context-local variables), for anyone interested. But if you read the draft I posted on python-ideas last Monday, you've already seen it. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Consolidate stateful runtime globals
On Wed, Sep 6, 2017 at 8:17 PM, Benjamin Peterson wrote: > On Wed, Sep 6, 2017, at 10:08, Antoine Pitrou wrote: > > On Wed, 06 Sep 2017 09:42:29 -0700 > > Benjamin Peterson wrote: > > > On Wed, Sep 6, 2017, at 03:14, Antoine Pitrou wrote: > > > > > > > > Hello, > > > > > > > > I'm a bit concerned about > > > > https://github.com/python/cpython/commit/ > 76d5abc8684bac4f2fc7cccfe2cd940923357351 > > > > > > > > My main gripe is that makes writing C code more tedious. Simple C > > > > global variables such as "_once_registry" are now spelled > > > > "_PyRuntime.warnings.once_registry". The most egregious example > seems > > > > to be "_PyRuntime.ceval.gil.locked" (used to be simply "gil_locked"). > > > > > > > > Granted, C is more verbose than Python, but it doesn't have to become > > > > that verbose. I don't know about you, but when code becomes annoying > > > > to type, I tend to try and take shortcuts. > > > > > > How often are you actually typing the names of runtime globals, though? > > > > Not very often, but if I want to experiment with some low-level > > implementation details, it is nice to avoid the hassle. > > It seems like this could be remediated with some inline functions or > macros, which would also help safely encapsulate state. > > > > > There's also a readability argument: with very long names, expressions > > can become less easy to parse. > > > > > If you are using a globals, perhaps the typing time will allow you to > > > fully consider the gravity of the situation. > > > > Right, I needed to be reminded of how perilous the use of C globals is. > > Perhaps I should contact the PSRT the next time I contemplate using a C > > global. > > It's not just you but future readers. Great. Related to this, there is also discussion on dangers of globals and other widely-scoped variables in the Rationale section of PEP 555 (Context-local variables), for anyone interested. But if you read the draft I posted on python-ideas last Monday, you've already seen it. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 559 - built-in noop()
On Sat, Sep 9, 2017 at 10:54 PM, Victor Stinner
wrote:
> I always wanted this feature (no kidding).
>
> Would it be possible to add support for the context manager?
>
> with noop(): ...
>
> Maybe noop can be an instance of:
>
> class Noop:
> def __enter__(self, *args, **kw): return self
> def __exit__(self, *args): pass
> def __call__(self, *args, **kw): return self
>
>
This worries me. Clearly, assuming a None-coercing noop, we must have:
noop(foo) is None
noop[foo] is None
noop * foo is None
foo * noop is None
noop + foo is None
foo + noop is None
noop - noop is None
...
noop / 0 is None
...
(noop == None) is None
which can all sort of be implicitly extrapolated to be in the PEP.
But how are you planning to implement:
(noop is None) is None
(obj in noop) is None
(noop in obj) is None
or
(None or noop) is None
(None and noop) is None
and finally:
foo(noop) is None
?
Sooner or later, someone will need all these features, and the PEP does not
seem to address the issue in any way.
-- Koos
> Victor
>
> Le 9 sept. 2017 11:48 AM, "Barry Warsaw" a écrit :
>
>> I couldn’t resist one more PEP from the Core sprint. I won’t reveal
>> where or how this one came to me.
>>
>> -Barry
>>
>> PEP: 559
>> Title: Built-in noop()
>> Author: Barry Warsaw
>> Status: Draft
>> Type: Standards Track
>> Content-Type: text/x-rst
>> Created: 2017-09-08
>> Python-Version: 3.7
>> Post-History: 2017-09-09
>>
>>
>> Abstract
>>
>>
>> This PEP proposes adding a new built-in function called ``noop()`` which
>> does
>> nothing but return ``None``.
>>
>>
>> Rationale
>> =
>>
>> It is trivial to implement a no-op function in Python. It's so easy in
>> fact
>> that many people do it many times over and over again. It would be
>> useful in
>> many cases to have a common built-in function that does nothing.
>>
>> One use case would be for PEP 553, where you could set the breakpoint
>> environment variable to the following in order to effectively disable it::
>>
>> $ setenv PYTHONBREAKPOINT=noop
>>
>>
>> Implementation
>> ==
>>
>> The Python equivalent of the ``noop()`` function is exactly::
>>
>> def noop(*args, **kws):
>> return None
>>
>> The C built-in implementation is available as a pull request.
>>
>>
>> Rejected alternatives
>> =
>>
>> ``noop()`` returns something
>>
>>
>> YAGNI.
>>
>> This is rejected because it complicates the semantics. For example, if
>> you
>> always return both ``*args`` and ``**kws``, what do you return when none
>> of
>> those are given? Returning a tuple of ``((), {})`` is kind of ugly, but
>> provides consistency. But you might also want to just return ``None``
>> since
>> that's also conceptually what the function was passed.
>>
>> Or, what if you pass in exactly one positional argument, e.g.
>> ``noop(7)``. Do
>> you return ``7`` or ``((7,), {})``? And so on.
>>
>> The author claims that you won't ever need the return value of ``noop()``
>> so
>> it will always return ``None``.
>>
>> Coghlin's Dialogs (edited for formatting):
>>
>> My counterargument to this would be ``map(noop, iterable)``,
>> ``sorted(iterable, key=noop)``, etc. (``filter``, ``max``, and
>> ``min`` all accept callables that accept a single argument, as do
>> many of the itertools operations).
>>
>> Making ``noop()`` a useful default function in those cases just
>> needs the definition to be::
>>
>>def noop(*args, **kwds):
>>return args[0] if args else None
>>
>> The counterargument to the counterargument is that using ``None``
>> as the default in all these cases is going to be faster, since it
>> lets the algorithm skip the callback entirely, rather than calling
>> it and having it do nothing useful.
>>
>>
>> Copyright
>> =
>>
>> This document has been placed in the public domain.
>>
>>
>> ..
>>Local Variables:
>>mode: indented-text
>>indent-tabs-mode: nil
>>sentence-end-double-space: t
>>fill-column: 70
>>coding: utf-8
>>End:
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.
>> stinner%40gmail.com
>>
>>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>
>
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 559 - built-in noop()
On Sun, Sep 10, 2017 at 8:21 PM, Barry Warsaw wrote: > On Sep 9, 2017, at 15:12, Guido van Rossum wrote: > > > > I can't tell whether this was meant seriously, but I don't think it's > worth it. People can easily write their own dummy function and give it any > damn semantics they want. Let's reject the PEP. > > Alrighty then! (Yes, it was serious, but I claim post-sprint > euphoria/delirium). > > Just for future reference, here's a slightly more serious comment: I think the "pass" statement wasn't mentioned yet, but clearly noop() would be duplication of functionality. So maybe the closest thing without duplication would be to make "pass" an expression which evaluates to a no-op function, but which the compiler could perhaps optimize away if it's a statement by itself, or is a builtin. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Mon, Sep 11, 2017 at 8:51 AM, Nick Coghlan wrote: > On 10 September 2017 at 04:04, Nathaniel Smith wrote: > > On Sep 8, 2017 4:06 PM, "Eric Snow" wrote: > > > > > >run(code): > > > > Run the provided Python code in the interpreter, in the current > > OS thread. If the interpreter is already running then raise > > RuntimeError in the interpreter that called ``run()``. > > > > The current interpreter (which called ``run()``) will block until > > the subinterpreter finishes running the requested code. Any > > uncaught exception in that code will bubble up to the current > > interpreter. > > > > > > This phrase "bubble up" here is doing a lot of work :-). Can you > elaborate > > on what you mean? The text now makes it seem like the exception will just > > pass from one interpreter into another, but that seems impossible – it'd > > mean sharing not just arbitrary user defined exception classes but full > > frame objects... > > Indeed, I think this will need to be something more akin to > https://docs.python.org/3/library/subprocess.html# > subprocess.CalledProcessError, > where the child interpreter is able to pass back encoded text data > (perhaps including a full rendered traceback), but the exception > itself won't be able to be propagated. > It would be helpful if at least the exception type could somehow be preserved / restored / mimicked. Otherwise you need if-elif statements in your try-excepts and other annoying stuff. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
On Sep 29, 2017 18:21, "Guido van Rossum" wrote: PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis. This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
On Oct 1, 2017 19:26, "Guido van Rossum" wrote: Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it. I can withdraw it if you ask me to, but I don't want to withdraw it without any reason. I haven't changed my mind about the big picture. OTOH, PEP 521 is elegant and could be used to implement PEP 555, but 521 is almost certainly less performant and has some problems regarding context manager wrappers that use composition instead of inheritance. -- Koos On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" wrote: > On Sep 29, 2017 18:21, "Guido van Rossum" wrote: > > > PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and > Elvis. > > > This is getting really off-topic, but I do have updates to add to PEP 555 > if there is interest in that. IMO, 555 is better and most likely faster > than 550, but on the other hand, the issues with PEP 550 are most likely > not going to be a problem for me personally. > > -- Koos > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 552 soon (deterministic pyc files)
On Mon, Oct 2, 2017 at 6:42 AM, Guido van Rossum wrote:
> On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven wrote:
>
>> On Oct 1, 2017 19:26, "Guido van Rossum" wrote:
>>
>> Your PEP is currently incomplete. If you don't finish it, it is not even
>> a contender. But TBH it's not my favorite anyway, so you could also just
>> withdraw it.
>>
>>
>> I can withdraw it if you ask me to, but I don't want to withdraw it
>> without any reason. I haven't changed my mind about the big picture. OTOH,
>> PEP 521 is elegant and could be used to implement PEP 555, but 521 is
>> almost certainly less performant and has some problems regarding context
>> manager wrappers that use composition instead of inheritance.
>>
>
> It is my understanding that PEP 521 (which proposes to add optional
> __suspend__ and __resume__ methods to the context manager protocol, to be
> called whenever a frame is suspended or resumed inside a `with` block) is
> no longer a contender because it would be way too slow. I haven't read it
> recently or thought about it, so I don't know what the second issue you
> mention is about (though it's presumably about the `yield` in a context
> manager implemented using a generator decorated with
> `@contextlib.contextmanager`).
>
>
Well, it's not completely unrelated to that. The problem I'm talking about
is perhaps most easily seen from a simple context manager wrapper that uses
composition instead of inheritance:
class Wrapper:
def __init__(self):
self._wrapped = SomeContextManager()
def __enter__(self):
print("Entering context")
return self._wrapped.__enter__()
def __exit__(self):
self._wrapped.__exit__()
print("Exited context")
Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__
and __resume__, the Wrapper class is broken, because it does not respect
__suspend__ and __resume__. So actually this is a backwards compatiblity
issue.
But if the wrapper is made using inheritance, the problem goes away:
class Wrapper(SomeContextManager):
def __enter__(self):
print("Entering context")
return super().__enter__()
def __exit__(self):
super().__exit__()
print("Exited context")
Now the wrapper cleanly inherits the new optional __suspend__ and
__resume__ from the wrapped context manager type.
––Koos
>
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Inheritance vs composition in backcompat (PEP521)
Hi all, It was suggested that I start a new thread, because the other
thread drifted away from its original topic. So here, in case someone is
interested:
On Oct 2, 2017 17:03, "Koos Zevenhoven wrote:
On Mon, Oct 2, 2017 at 6:42 AM, Guido van Rossum wrote:
On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven wrote:
On Oct 1, 2017 19:26, "Guido van Rossum" wrote:
Your PEP is currently incomplete. If you don't finish it, it is not even a
contender. But TBH it's not my favorite anyway, so you could also just
withdraw it.
I can withdraw it if you ask me to, but I don't want to withdraw it without
any reason. I haven't changed my mind about the big picture. OTOH, PEP 521
is elegant and could be used to implement PEP 555, but 521 is almost
certainly less performant and has some problems regarding context manager
wrappers that use composition instead of inheritance.
It is my understanding that PEP 521 (which proposes to add optional
__suspend__ and __resume__ methods to the context manager protocol, to be
called whenever a frame is suspended or resumed inside a `with` block) is
no longer a contender because it would be way too slow. I haven't read it
recently or thought about it, so I don't know what the second issue you
mention is about (though it's presumably about the `yield` in a context
manager implemented using a generator decorated with
`@contextlib.contextmanager`).
Well, it's not completely unrelated to that. The problem I'm talking about
is perhaps most easily seen from a simple context manager wrapper that uses
composition instead of inheritance:
class Wrapper:
def __init__(self):
self._wrapped = SomeContextManager()
def __enter__(self):
print("Entering context")
return self._wrapped.__enter__()
def __exit__(self):
self._wrapped.__exit__()
print("Exited context")
Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__
and __resume__, the Wrapper class is broken, because it does not respect
__suspend__ and __resume__. So actually this is a backwards compatiblity
issue.
But if the wrapper is made using inheritance, the problem goes away:
class Wrapper(SomeContextManager):
def __enter__(self):
print("Entering context")
return super().__enter__()
def __exit__(self):
super().__exit__()
print("Exited context")
Now the wrapper cleanly inherits the new optional __suspend__ and
__resume__ from the wrapped context manager type.
––Koos
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Oct 3, 2017 00:02, "Guido van Rossum" wrote:
On Mon, Oct 2, 2017 at 10:13 AM, Koos Zevenhoven wrote:
> Hi all, It was suggested that I start a new thread, because the other
> thread drifted away from its original topic. So here, in case someone is
> interested:
>
> On Oct 2, 2017 17:03, "Koos Zevenhoven wrote:
>
> On Mon, Oct 2, 2017 at 6:42 AM, Guido van Rossum wrote:
>
> On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven wrote:
>
> On Oct 1, 2017 19:26, "Guido van Rossum" wrote:
>
> Your PEP is currently incomplete. If you don't finish it, it is not even a
> contender. But TBH it's not my favorite anyway, so you could also just
> withdraw it.
>
>
> I can withdraw it if you ask me to, but I don't want to withdraw it
> without any reason. I haven't changed my mind about the big picture. OTOH,
> PEP 521 is elegant and could be used to implement PEP 555, but 521 is
> almost certainly less performant and has some problems regarding context
> manager wrappers that use composition instead of inheritance.
>
>
> It is my understanding that PEP 521 (which proposes to add optional
> __suspend__ and __resume__ methods to the context manager protocol, to be
> called whenever a frame is suspended or resumed inside a `with` block) is
> no longer a contender because it would be way too slow. I haven't read it
> recently or thought about it, so I don't know what the second issue you
> mention is about (though it's presumably about the `yield` in a context
> manager implemented using a generator decorated with
> `@contextlib.contextmanager`).
>
>
> Well, it's not completely unrelated to that. The problem I'm talking
> about is perhaps most easily seen from a simple context manager wrapper
> that uses composition instead of inheritance:
>
> class Wrapper:
> def __init__(self):
> self._wrapped = SomeContextManager()
>
> def __enter__(self):
> print("Entering context")
> return self._wrapped.__enter__()
>
> def __exit__(self):
> self._wrapped.__exit__()
> print("Exited context")
>
>
> Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__
> and __resume__, the Wrapper class is broken, because it does not respect
> __suspend__ and __resume__. So actually this is a backwards compatiblity
> issue.
>
>
Why is it backwards incompatible? I'd think that without PEP 521 it would
be broken in exactly the same way because there's no __suspend__/__resume__
at all.
The wrapper is (would be) broken because it depends on the internal
implementation of the wrapped CM.
Maybe the author of SomeContextManager wants to upgrade the CM to also work
in coroutines and generators. But it could be a more subtle change in the
CM implementation.
The problem becomes more serious and more obvious if you don't know which
context manager you are wrapping:
class Wrapper:
def __init__(self, contextmanager):
self._wrapped = contextmanager
def __enter__(self):
print("Entering context")
return self._wrapped.__enter__()
def __exit__(self):
self._wrapped.__exit__()
print("Exited context")
The wrapper is (would be) broken because it does not work for all CMs
anymore.
But if the wrapper is made using inheritance, the problem goes away:
>
>
> class Wrapper(SomeContextManager):
> def __enter__(self):
> print("Entering context")
> return super().__enter__()
>
> def __exit__(self):
> super().__exit__()
> print("Exited context")
>
>
> Now the wrapper cleanly inherits the new optional __suspend__ and
> __resume__ from the wrapped context manager type.
>
>
In any case this is completely academic because PEP 521 is not going to
happen. Nathaniel himself has said so (I think in the context of discussing
PEP 550).
I don't mind this (or Nathaniel ;-) being academic. The backwards
incompatibility issue I've just described applies to any extension via
composition, if the underlying type/protocol grows new members (like the CM
protocol would have gained __suspend__ and __resume__ in PEP521).
-- Koos (mobile)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Oct 3, 2017 01:00, "Guido van Rossum" wrote: Mon, Oct 2, 2017 at 2:52 PM, Koos Zevenhoven wrote I don't mind this (or Nathaniel ;-) being academic. The backwards > incompatibility issue I've just described applies to any extension via > composition, if the underlying type/protocol grows new members (like the CM > protocol would have gained __suspend__ and __resume__ in PEP521). > Since you seem to have a good grasp on this issue, does PEP 550 suffer from the same problem? (Or PEP 555, for that matter? :-) Neither has this particular issue, because they don't extend an existing protocol. If this thread has any significance, it will most likely be elsewhere. -- Koos (mobile) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Oct 3, 2017 01:11, "Koos Zevenhoven" wrote: On Oct 3, 2017 01:00, "Guido van Rossum" wrote: Mon, Oct 2, 2017 at 2:52 PM, Koos Zevenhoven wrote I don't mind this (or Nathaniel ;-) being academic. The backwards > incompatibility issue I've just described applies to any extension via > composition, if the underlying type/protocol grows new members (like the CM > protocol would have gained __suspend__ and __resume__ in PEP521). > Since you seem to have a good grasp on this issue, does PEP 550 suffer from the same problem? (Or PEP 555, for that matter? :-) Neither has this particular issue, because they don't extend an existing protocol. If this thread has any significance, it will most likely be elsewhere. That said, I did come across this thought while trying to find flaws in my own PEP ;) -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
I've probably missed a lot of this discussion, but this lazy import discussion confuses me. We already have both eager import (import at the top of the file), and lazy import (import right before use). The former is good when you know you need the module, and the latter is good when you having the overhead at first use is preferable over having the overhead at startup. But like Raymond was saying, this is of course especially relevant when that import is likely never used. Maybe the fact that the latter is not recommended gives people the feeling that we don't have lazy imports, although we do. What we *don't* have, however, is *partially* lazy imports and partially executed code, something like: on demand: class Foo: # a lot of stuff here def foo_function(my_foo, bar): # more stuff here When executed, the `on demand` block would only keep track of which names are being bound to (here, "Foo" and "foo_function"), and on the lookup of those names in the namespace, the code would actually be run. Then you could also do on demand: import sometimes_needed_module Or on demand: from . import all, submodules, of, this, package This would of course drift away from "namespaces are simply dicts". But who cares, if they still provide the dict interface. See e.g. this example with automatic lazy imports: https://gist.github.com/k7hoven/21c5532ce19b306b08bb4e82cfe5a609 Another thing we *don't* have is unimporting. What if I know that I'm only going to need some particular module in this one initialization function. Why should I keep it in memory for the whole lifetime of the program? ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Wed, Oct 4, 2017 at 8:07 AM, Nick Coghlan wrote:
> On 3 October 2017 at 03:13, Koos Zevenhoven wrote:
> > Well, it's not completely unrelated to that. The problem I'm talking
> about
> > is perhaps most easily seen from a simple context manager wrapper that
> uses
> > composition instead of inheritance:
> >
> > class Wrapper:
> > def __init__(self):
> > self._wrapped = SomeContextManager()
> >
> > def __enter__(self):
> > print("Entering context")
> > return self._wrapped.__enter__()
> >
> > def __exit__(self):
> > self._wrapped.__exit__()
> > print("Exited context")
> >
> >
> > Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__
> > and __resume__, the Wrapper class is broken, because it does not respect
> > __suspend__ and __resume__. So actually this is a backwards compatiblity
> > issue.
>
> This is a known problem, and one of the main reasons that having a
> truly transparent object proxy like
> https://wrapt.readthedocs.io/en/latest/wrappers.html#object-proxy as
> part of the standard library would be highly desirable.
>
>
This is barely related to the problem I describe. The wrapper is not
supposed to pretend to *be* the underlying object. It's just supposed to
extend its functionality.
Maybe it's just me, but using a transparent object proxy for this sounds
like someone trying to avoid inheritance for no reason and at any cost.
Inheritance probably has faster method access, and makes it more obvious
what's going on:
def Wrapper(contextmanager):
class Wrapper(type(contextmanager)):
def __enter__(self):
print("Entering context")
return contextmanager.__enter__()
def __exit__(self):
contextmanager.__exit__()
print("Exited context")
return Wrapper()
A wrapper based on a transparent object proxy is just a non-transparent
replacement for inheritance. Its wrapper nature is non-transparent because
it pretends to `be` the original object, while it's actually a wrapper.
But an object cannot `be` another object as long as the `is` operator won't
return True. And any straightforward way to implement that would add
performance overhead for normal objects.
I do remember sometimes wanting a transparent object proxy. But not for
normal wrappers. But I don't think I've gone as far as looking for a
library to do that, because it seems that you can only go half way anyway.
––Koos
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Wed, Oct 4, 2017 at 3:33 PM, Nick Coghlan wrote:
> On 4 October 2017 at 20:22, Koos Zevenhoven wrote:
> > On Wed, Oct 4, 2017 at 8:07 AM, Nick Coghlan wrote:
> >>
> >> On 3 October 2017 at 03:13, Koos Zevenhoven wrote:
> >> > Well, it's not completely unrelated to that. The problem I'm talking
> >> > about
> >> > is perhaps most easily seen from a simple context manager wrapper that
> >> > uses
> >> > composition instead of inheritance:
> >> >
> >> > class Wrapper:
> >> > def __init__(self):
> >> > self._wrapped = SomeContextManager()
> >> >
> >> > def __enter__(self):
> >> > print("Entering context")
> >> > return self._wrapped.__enter__()
> >> >
> >> > def __exit__(self):
> >> > self._wrapped.__exit__()
> >> > print("Exited context")
> >> >
> >> >
> >> > Now, if the wrapped contextmanager becomes a PEP 521 one with
> >> > __suspend__
> >> > and __resume__, the Wrapper class is broken, because it does not
> respect
> >> > __suspend__ and __resume__. So actually this is a backwards
> compatiblity
> >> > issue.
> >>
> >> This is a known problem, and one of the main reasons that having a
> >> truly transparent object proxy like
> >> https://wrapt.readthedocs.io/en/latest/wrappers.html#object-proxy as
> >> part of the standard library would be highly desirable.
> >>
> >
> > This is barely related to the problem I describe. The wrapper is not
> > supposed to pretend to *be* the underlying object. It's just supposed to
> > extend its functionality.
>
> If a wrapper *isn't* trying to act as a transparent object proxy, and
> is instead adapting it to a particular protocol, then yes, you'll need
> to update the wrapper when the protocol is extended.
>
>
Yes, but it still means that the change in the dependency (in this case a
standard Python protocol) breaks the wrapper code.
Remember that the wrappeR class and the wrappeD class can be implemented in
different libraries.
> That's not a backwards compatibility problem, because the only way to
> encounter it is to update your code to rely on the new extended
> protocol - your *existing* code will continue to work fine, since
> that, by definition, can't be relying on the new protocol extension.
>
>
No, not all code is "your" code. Clearly this is not a well-known problem.
This is a backwards-compatibility problem for the author of the wrappeR,
not for the author of the wrappeD object.
––Koos
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Wed, Oct 4, 2017 at 4:04 PM, Nick Coghlan wrote: > On 4 October 2017 at 22:45, Koos Zevenhoven wrote: > > On Wed, Oct 4, 2017 at 3:33 PM, Nick Coghlan wrote: > >> That's not a backwards compatibility problem, because the only way to > >> encounter it is to update your code to rely on the new extended > >> protocol - your *existing* code will continue to work fine, since > >> that, by definition, can't be relying on the new protocol extension. > >> > > > > No, not all code is "your" code. Clearly this is not a well-known > problem. > > This is a backwards-compatibility problem for the author of the wrappeR, > not > > for the author of the wrappeD object. > > No, you're misusing the phrase "backwards compatibility", and > confusing it with "feature enablement". > > Preserving backwards compatibility just means "existing code and > functionality don't break". It has nothing to do with whether or not > other support libraries and frameworks might need to change in order > to enable full access to a new language feature. > > It's not about full access to a new language feature. It's about the wrappeR promising it can wrap any context manager, which it then no longer can. If the __suspend__ and __resume__ methods are ignored, that is not about "not having full access to a new feature" — that's broken code. The error message you get (if any) may not contain any hint of what went wrong. Take the length hint protocol defined in PEP 424 for example: that > extended the iterator protocol to include a new optional > __length_hint__ method, such that container constructors can make a > more reasonable guess as to how much space they should pre-allocate > when being initialised from an iterator or iterable rather than > another container. > > This is slightly similar, but not really. Not using __length_hint__ does not affect the correctness of code. > That protocol means that many container wrappers break the > optimisation. That's not a compatibility problem, it just means those > wrappers don't support the feature, and it would potentially be a > useful enhancement if they did. > > Again, ignoring __length_hint__ does not lead to broken code, so that just means the wrapper is as slow or as fast as it was before. So I still think it's an issue for the author of the wrapper to fix––even if just by documenting that the wrapper does not support the new protocol members. But that would not be necessary if the wrapper uses inheritance. (Of course there may be another reason to not use inheritance, but just overriding two methods seems like a good case for inheritance.). This discussion seems pretty pointless by now. It's true that *some* code needs to change for this to be a problem. Updating only the Python version does not break a codebase if libraries aren't updated, and even then, breakage is not very likely, I suppose. It all depends on the kind of change that is made. For __length_hint__, you only risk not getting the performance improvement. For __suspend__ and __resume__, there's a small chance of problems. For some other change, it might be even riskier. But this is definitely not the most dangerous type of compatibility issue. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
On Wed, Oct 4, 2017 at 4:51 PM, Eric Snow wrote: > On Tue, Oct 3, 2017 at 11:36 PM, Nick Coghlan wrote: > > The problem relates to the fact that there aren't any memory barriers > > around CPython's INCREF operations (they're implemented as an ordinary > > C post-increment operation), so you can get the following scenario: > > > > * thread on CPU A has the sole reference (ob_refcnt=1) > > * thread on CPU B acquires a new reference, but hasn't pushed the > > updated ob_refcnt value back to the shared memory cache yet > > * original thread on CPU A drops its reference, *thinks* the refcnt is > > now zero, and deletes the object > > * bad things now happen in CPU B as the thread running there tries to > > use a deleted object :) > > I'm not clear on where we'd run into this problem with channels. > Mirroring your scenario: > > * interpreter A (in thread on CPU A) INCREFs the object (the GIL is still > held) > * interp A sends the object to the channel > * interp B (in thread on CPU B) receives the object from the channel > * the new reference is held until interp B DECREFs the object > > From what I see, at no point do we get a refcount of 0, such that > there would be a race on the object being deleted. > > So what you're saying is that when Larry finishes the gilectomy, subinterpreters will work GIL-free too?-) ––Koos The only problem I'm aware of (it dawned on me last night), is in the > case that the interpreter that created the object gets deleted before > the object does. In that case we can't pass the deletion back to the > original interpreter. (I don't think this problem is necessarily > exclusive to the solution I've proposed for Bytes.) > > -eric > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > k7hoven%40gmail.com > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inheritance vs composition in backcompat (PEP521)
On Tue, Oct 3, 2017 at 1:11 AM, Koos Zevenhoven wrote: > On Oct 3, 2017 01:00, "Guido van Rossum" wrote: > > Mon, Oct 2, 2017 at 2:52 PM, Koos Zevenhoven wrote > > I don't mind this (or Nathaniel ;-) being academic. The backwards >> incompatibility issue I've just described applies to any extension via >> composition, if the underlying type/protocol grows new members (like the CM >> protocol would have gained __suspend__ and __resume__ in PEP521). >> > > Since you seem to have a good grasp on this issue, does PEP 550 suffer > from the same problem? (Or PEP 555, for that matter? :-) > > > > Neither has this particular issue, because they don't extend an existing > protocol. If this thread has any significance, it will most likely be > elsewhere. > Actually, I realize I should be more precise with terminology regarding "extending an existing protocol"/"growing new members". Below, I'm still using PEP 521 as an example (sorry). In fact, in some sense, "adding" __suspend__ and __resume__ to context managers *does not* extend the context manager protocol, even though it kind of looks like it does. There would instead be two separate protocols: (A) The traditional PEP 343 context manager: __enter__ __exit__ (B) The hyphothetical PEP 521 context manager: __enter__ __suspend__ __resume__ __exit__ Protocols A and B are incompatible in both directions: * It is generally not safe to use a type-A context manager assuming it implements B. * It is generally not safe to use a type-B context manager assuming it implements A. But if you now have a type-B object, it looks like it's also type-A, especially for code that is not aware of the existence of B. This is where the problems come from: a wrapper for type A does the wrong thing when wrapping a type-B object (except when using inheritance). [Side note: Another interpretation of the situation is that, instead of adding protocol B, A is removed and is replaced with: (C) The hypothetical PEP 521 context manager with optional members: __enter__ __suspend__ (optional) __resume__ (optional) __exit__ But now the same problems just come from the fact that A no longer exists while there is code out there that assumes A. But this is only a useful interpretation if you are the only user of the protocol or if it's otherwise ok to remove A. So let's go back to the A-B interpretation.] Q: Could the problem of protocol conflict be solved? One way to tell A and B apart would be to always explicitly mark the protocol with a base class. Obviously this is not the case with existing uses of context managers. But there's another way, which is to change the naming: (A) The traditional PEP 343 context manager: __enter__ __exit__ (Z) The *modified* hyphothetical PEP 521 context manager: __begin__ __suspend__ __resume__ __end__ Now, A and Z are easy to tell apart. A context manager wrapper designed for type A immediately fails if used to wrap a type-Z object. But of course the whole context manager concept now suddenly became a lot more complicated. It is interesting that, in the A-B scheme, making a general context manager wrapper using inheritance *just works*, even if A is not a subprotocol of B and B is not a subprotocol of A. Anyway, a lot of this is amplified by the fact that the methods of the context manager protocols are not independent functionality. Instead, calling one of them leads to the requirement that the other methods are also called at the right moments. --Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v3 (new interpreters module)
rk through subclassing or a registry >> + a lot of conceptual similarity with tp_share+tp_free >> * a CIV-like proxy >> + you wrap the object, send() the proxy, and recv() a proxy >> + this is entirely compatible with tp_share() >> > > * Allow for multiple channel types, such that MemChannel is merely the > *first* channel type, rather than the *only* channel type > + Allows PEP 554 to be restricted to things we already know can be made > to work > + Doesn't block the introduction of an object-sharing based Channel in > some future release > + Allows for at least some channel types to be adapted for use with > shared memory and multiprocessing > > >> Here are what I consider the key metrics relative to the utility of a >> solution (not in any significant order): >> >> * how hard to understand as a Python programmer? >> > > Not especially important yet - this is more a criterion for the final API, > not the initial experimental platform. > > >> * how much extra work (if any) for folks calling Channel.send()? >> * how much extra work (if any) for folks calling Channel.recv()? >> > > I don't think either are particularly important yet, although we also > don't want to raise any pointless barriers to experimentation. > > >> * how complex is the CPython implementation? >> > > This is critical, since we want to minimise any potential for undesirable > side effects on regular single interpreter code. > > >> * how hard to understand as a type author (wanting to add support for >> their type)? >> * how hard to add support for a new type? >> * what variety of types could be supported? >> * what breadth of experimentation opens up? >> > > You missed the big one: what risk does the initial channel design pose to > the underlying objective of making the GIL a genuinely per-interpreter lock? > > If we don't eventually reach the latter goal, then subinterpreters won't > really offer much in the way of compelling benefits over just using a > thread pool and queue.Queue. > > MemChannel poses zero additional risk to that, since we wouldn't be > sharing actual Python objects between interpreters, only C pointers and > structs. > > By contrast, introducing an object channel early poses significant new > risks to that goal, since it will force you to solve hard protocol design > and refcount management problems *before* making the switch, rather than > being able to defer the design of the object channel protocol until *after* > you've already enabled the ability to run subinterpreters in completely > independent threads. > > >> The most important thing to me is keeping things simple for Python >> programmers. After that is ease-of-use for type authors. However, I >> also want to put us in a good position in 3.7 to experiment >> extensively with subinterpreters, so that's a big consideration. >> >> Consequently, for PEP 554 my goal is to find a solution for object >> sharing that keeps things simple in Python while laying a basic >> foundation we can build on at the C level, so we don't get locked in >> but still maximize our opportunities to experiment. :) >> > > I think our priorities are quite different then, as I believe PEP 554 > should be focused on defining a relatively easy to implement API that > nevertheless makes it possible to write interesting programs while working > on the goal of making the GIL per-interpreter, without worrying too much > about whether or not the initial cross-interpreter communication channels > closely resemble the final ones that will be intended for more general use. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > k7hoven%40gmail.com > > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
On Sun, Oct 8, 2017 at 11:02 AM, David Cournapeau wrote: > > On Mon, Oct 2, 2017 at 6:42 PM, Raymond Hettinger < > raymond.hettin...@gmail.com> wrote: > >> >> > On Oct 2, 2017, at 12:39 AM, Nick Coghlan wrote: >> > >> > "What requests uses" can identify a useful set of >> > avoidable imports. A Flask "Hello world" app could likely provide >> > another such sample, as could some example data analysis notebooks). >> >> Right. It is probably worthwhile to identify which parts of the library >> are typically imported but are not ever used. And likewise, identify a >> core set of commonly used tools that are going to be almost unavoidable in >> sufficiently interesting applications (like using requests to access a REST >> API, running a micro-webframework, or invoking mercurial). >> >> Presumably, if any of this is going to make a difference to end users, we >> need to see if there is any avoidable work that takes a significant >> fraction of the total time from invocation through the point where the user >> first sees meaningful output. That would include loading from nonvolatile >> storage, executing the various imports, and doing the actual application. >> >> I don't expect to find anything that would help users of Django, Flask, >> and Bottle since those are typically long-running apps where we value >> response time more than startup time. >> >> For scripts using the requests module, there will be some fruit because >> not everything that is imported is used. However, that may not be >> significant because scripts using requests tend to be I/O bound. In the >> timings below, 6% of the running time is used to load and run python.exe, >> another 16% is used to import requests, and the remaining 78% is devoted to >> the actual task of running a simple REST API query. It would be interesting >> to see how much of the 16% could be avoided without major alterations to >> requests, to urllib3, and to the standard library. >> > > It is certainly true that for a CLI tool that actually makes any network > I/O, especially SSL, import times will quickly be negligible. It becomes > tricky for complex tools, because of error management. For example, a > common pattern I have used in the past is to have a high level "catch all > exceptions" function that dispatch the CLI command: > > try: > main_function(...) > except ErrorKind1: > > except requests.exceptions.SSLError: > # gives complete message about options when receiving SSL errors, e.g. > invalid certificate > > This pattern requires importing requests every time the command is run, > even if no network IO is actually done. For complex CLI tools, maybe most > command don't use network IO (the tool in question was a complete packages > manager), but you pay ~100 ms because of requests import for every command. > It is particularly visible because commands latency starts to be felt > around 100-150 ms, and while you can do a lot in python in 100-150 ms, you > can't do much in 0-50 ms. > > Yes. OTOH, it can also happen that the *imports* are in fact what use the network IO. At the office, I usually import from a network drive. For instance, `import requests` takes a little less than a second, and `import IPython` usually takes more than a second, with some variation. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
On Sun, Oct 8, 2017 at 2:44 PM, Chris Angelico wrote: > On Sun, Oct 8, 2017 at 7:02 PM, David Cournapeau > wrote: > > It is certainly true that for a CLI tool that actually makes any network > > I/O, especially SSL, import times will quickly be negligible. It becomes > > tricky for complex tools, because of error management. For example, a > common > > pattern I have used in the past is to have a high level "catch all > > exceptions" function that dispatch the CLI command: > > > > try: > > main_function(...) > > except ErrorKind1: > > > > except requests.exceptions.SSLError: > > # gives complete message about options when receiving SSL errors, > e.g. > > invalid certificate > > > > This pattern requires importing requests every time the command is run, > even > > if no network IO is actually done. For complex CLI tools, maybe most > command > > don't use network IO (the tool in question was a complete packages > manager), > > but you pay ~100 ms because of requests import for every command. It is > > particularly visible because commands latency starts to be felt around > > 100-150 ms, and while you can do a lot in python in 100-150 ms, you > can't do > > much in 0-50 ms. > > This would be a perfect use-case for lazy importing, then. You'd pay > the price of the import only if you get an error that isn't caught by > one of the preceding except blocks. > I suppose it might be convenient to be able to do something like: with autoimport: try: main_function(...) except ErrorKind1: ... except requests.exceptions.SLLError: ... The easiest workaround at the moment is still pretty clumsy: def import_SLLError(): from requests.exceptions import SLLError return SLLError ... except import_SLLError(): But what happens if that gives you an ImportError? ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
as the dataclass interaction with `__slots__` goes, that's a > problem largely specific to slots (and `__metaclass__` before it), in that > they're the only characteristics of a class definition that affect how > CPython allocates memory for the class object itself (the descriptors for > the slots are stored as a pointer array after the class struct, rather than > only in the class dict). > > Given PEP 526 variable annotations, __slots__ could potentially benefit > from a __metaclass__ style makeover, allowing an "infer_slots=True" keyword > argument to type.__new__ to request that the list of slots be inferred from > __annotations__ (Slot inference would conflict with setting class level > default values, but that's a real conflict, as you'd be trying to use the > same name on the class object for both the slot descriptor and the default > value) > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > k7hoven%40gmail.com > > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On Thu, Nov 9, 2017 at 9:51 PM, Guido van Rossum wrote: > If we have to change the name I'd vote for string_annotations -- "lazy" > has too many other connotations (e.g. it might cause people to think it's > the thunks). I find str_annotations too abbreviated, and > stringify_annotations is too hard to spell. > > I can't say I disagree. And maybe importing string_annotations from the __future__ doesn't sound quite as sad as importing something from the __past__. Anyway, it's not obvious to me that it is the module author that should decide how the annotations are handled. See also this quote below: (Quoted from the end of https://mail.python.org/pipermail/python-ideas/2017-October/047311.html ) On Thu, Oct 12, 2017 at 3:59 PM, Koos Zevenhoven wrote: > > [*] Maybe somehow make the existing functionality a phantom easter > egg––a blast from the past which you can import and use, but which is > otherwise invisible :-). Then later give warnings and finally remove it > completely. > > But we need better smooth upgrade paths anyway, maybe something like: > > from __compat__ import unintuitive_decimal_contexts > > with unintuitive_decimal_contexts: > do_stuff() > > Now code bases can more quickly switch to new python versions and make > the occasional compatibility adjustments more lazily, while already > benefiting from other new language features. > > > ––Koos > > > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On Fri, Nov 10, 2017 at 7:50 PM, Ethan Furman wrote: > On 11/10/2017 07:48 AM, Guido van Rossum wrote: > > I don't mind the long name. Of all the options so far I really only like >> 'string_annotations' so let's go with that. >> > > As someone else mentioned, we have function annotations and variable > annotations already, which makes string_annotations sound like it's > annotations for strings. > > > Contriwise, "annotation_strings" sounds like a different type of > annotation -- they are now being stored as strings, instead of something > else. > > Or a step further (longer), with annotations_as_strings. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Analog of PEP 448 for dicts (unpacking in assignment with dict rhs)
Oops, forgot to reply to the list.
On Nov 12, 2017 03:35, "Koos Zevenhoven" wrote:
On Nov 12, 2017 02:12, "Joao S. O. Bueno" wrote:
Ben, I have a small package which enables one to do:
with MapGetter(my_dictionary):
from my_dictionary import a, b, parameter3
If this interests you, contributions so it can get hardenned for
mainstram acceptance are welcome.
https://github.com/jsbueno/extradict
Your VersionDict in fact has some similarities to what I have thought of
implementing using the PEP 555 machinery, but it is also a bit different.
Interesting...
-- Koos (mobile)
On 11 November 2017 at 04:26, Ben Usman wrote:
> Got it, thank you. I'll go and check it out!
>
> On Nov 11, 2017 01:22, "Jelle Zijlstra" wrote:
>>
>>
>>
>> 2017-11-10 19:53 GMT-08:00 Ben Usman :
>>>
>>> The following works now:
>>>
>>> seq = [1, 2]
>>> d = {'c': 3, 'a': 1, 'b': 2}
>>>
>>> (el1, el2) = *seq
>>> el1, el2 = *seq
>>> head, *tail = *seq
>>>
>>> seq_new = (*seq, *tail)
>>> dict_new = {**d, **{'c': 4}}
>>>
>>> def f(arg1, arg2, a, b, c):
>>> pass
>>>
>>> f(*seq, **d)
>>>
>>> It seems like dict unpacking syntax would not be fully coherent with
>>> list unpacking syntax without something like:
>>>
>>> {b, a, **other} = **d
>>>
>>> Because iterables have both syntax for function call unpacking and
>>> "rhs in assignment unpacking" and dict has only function call
>>> unpacking syntax.
>>>
>>> I was not able to find any PEPs that suggest this (search keywords:
>>> "PEP 445 dicts", "dictionary unpacking assignment", checked PEP-0),
>>> however, let me know if I am wrong.
>>>
>> It was discussed at great length on Python-ideas about a year ago. There
>> is a thread called "Unpacking a dict" from May 2016.
>>
>>>
>>> The main use-case, in my understating, is getting shortcuts to
>>> elements of a dictionary if they are going to be used more then
>>> ones later in the scope. A made-up example is using a config to
>>> initiate a bunch of things with many config arguments with long
>>> names that have overlap in keywords used in initialization.
>>>
>>> One should either write long calls like
>>>
>>> start_a(config['parameter1'], config['parameter2'],
>>> config['parameter3'], config['parameter4'])
>>>
>>> start_b(config['parameter3'], config['parameter2'],
>>> config['parameter3'], config['parameter4'])
>>>
>>> many times or use a list-comprehension solution mentioned above.
>>>
>>> It becomes even worse (in terms of readability) with nested structures.
>>>
>>> start_b(config['group2']['parameter3'], config['parameter2'],
>>> config['parameter3'], config['group2']['parameter3'])
>>>
>>>
>>> ## Rationale
>>>
>>> Right now this problem is often solved using [list] comprehensions,
>>> but this is somewhat verbose:
>>>
>>> a, b = (d[k] for k in ['a', 'b'])
>>>
>>> or direct per-instance assignment (looks simple for with
>>> single-character keys, but often becomes very verbose with
>>> real-world long key names)
>>>
>>> a = d['a']
>>> b = d['b']
>>>
>>> Alternatively one could have a very basic method\function
>>> get_n() or __getitem__() accepting more then a single argument
>>>
>>> a, b = d.get_n('a', 'b')
>>> a, b = get_n(d, 'a', 'b')
>>> a, b = d['a', 'b']
>>>
>>> All these approaches require verbose double-mentioning of same
>>> key. It becomes even worse if you have nested structures
>>> of dictionaries.
>>>
>>> ## Concerns and questions:
>>>
>>> 0. This is the most troubling part, imho, other questions
>>> are more like common thoughts. It seems (to put it mildly)
>>> weird that execution flow depends on names of local variables.
>>>
>>> For example, one can not easily refactor these variable names. However,
>>> same is true for dictionary keys anyway: you can not suddenly decide
>>> and refactor your code to expect dictionaries wi
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On Sun, Nov 12, 2017 at 7:07 AM, Guido van Rossum wrote: > On Fri, Nov 10, 2017 at 11:02 PM, Nick Coghlan wrote: > >> On 11 November 2017 at 01:48, Guido van Rossum wrote: >> > I don't mind the long name. Of all the options so far I really only like >> > 'string_annotations' so let's go with that. >> >> +1 from me. >> > > I'd like to reverse my stance on this. We had `from __future__ import > division` for many years in Python 2, and nobody argued that it implied > that Python 2 doesn't have division -- it just meant to import the future > *version* of division. So I think the original idea, `from __future__ > import annotations` is fine. I don't expect there will be *other* things > related to annotations that we'll be importing from the future. > > Furthermore, *nobody* expects the majority of programmers to look at __annotations__ either. But those who do need to care about the 'implementation detail' of whether it's a string won't be surprised to find nested strings like "'ForwardReferencedThing'". But one might fear that those cases get ruthlessly converted into being equivalent to just "ForwardReferencedThing". So actually my question is: What should happen when the annotation is already a string literal? -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On Nov 12, 2017 19:10, "Guido van Rossum" wrote: On Sun, Nov 12, 2017 at 4:14 AM, Koos Zevenhoven wrote: > So actually my question is: What should happen when the annotation is > already a string literal? > The PEP answers that clearly (under Implementation): > If an annotation was already a string, this string is preserved > verbatim. Oh sorry, I was looking for a spec, so I somehow assumed I can ignore the gory implementation details just like I routinely ignore things like headers and footers of emails. There's two thing I don't understand here: * What does it mean to preserve the string verbatim? No matter how I read it, I can't tell if it's with quotes or without. Maybe I'm missing some context. -- Koos (mobile) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 562
On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi
wrote:
[..]
> Rationale
> =
>
> It is sometimes convenient to customize or otherwise have control over
> access to module attributes. A typical example is managing deprecation
> warnings. Typical workarounds are assigning ``__class__`` of a module
> object
> to a custom subclass of ``types.ModuleType`` or replacing the
> ``sys.modules``
> item with a custom wrapper instance. It would be convenient to simplify
> this
> procedure by recognizing ``__getattr__`` defined directly in a module that
> would act like a normal ``__getattr__`` method, except that it will be
> defined
> on module *instances*. For example::
>
>
# lib.py
>
> from warnings import warn
>
> deprecated_names = ["old_function", ...]
>
> def _deprecated_old_function(arg, other):
> ...
>
> def __getattr__(name):
> if name in deprecated_names:
> warn(f"{name} is deprecated", DeprecationWarning)
> return globals()[f"_deprecated_{name}"]
> raise AttributeError(f"module {__name__} has no attribute {name}")
>
> # main.py
>
> from lib import old_function # Works, but emits the warning
>
>
Deprecating functions is already possible, so I assume the reason for this
would be performance? If so, are you sure this would help for performance?
Deprecating module attributes / globals is indeed difficult to do at
present. This PEP would allow deprecation warnings for accessing
attributes, which is nice! However, as thread-unsafe as it is, many
modules use module attributes to configure the state of the module. In that
case, the user is more likely to *set* the attribute that to *get* it. Is
this outside the scope of the PEP?
[..]
> There is a related proposal PEP 549 that proposes to support instance
> properties for a similar functionality. The difference is this PEP proposes
> a faster and simpler mechanism, but provides more basic customization.
>
I'm not surprised that the comparison is in favor of this PEP ;-).
[..]
> Specification
> =
>
> The ``__getattr__`` function at the module level should accept one argument
> which is the name of an attribute and return the computed value or raise
> an ``AttributeError``::
>
> def __getattr__(name: str) -> Any: ...
>
> This function will be called only if ``name`` is not found in the module
> through the normal attribute lookup.
>
>
The Rationale (quoted in the beginning of this email) easily leaves a
different impression of this.
[..]
>
> Discussion
> ==
>
> Note that the use of module ``__getattr__`` requires care to keep the
> referred
> objects pickleable. For example, the ``__name__`` attribute of a function
> should correspond to the name with which it is accessible via
> ``__getattr__``::
>
> def keep_pickleable(func):
> func.__name__ = func.__name__.replace('_deprecated_', '')
> func.__qualname__ = func.__qualname__.replace('_deprecated_', '')
> return func
>
> @keep_pickleable
> def _deprecated_old_function(arg, other):
> ...
>
> One should be also careful to avoid recursion as one would do with
> a class level ``__getattr__``.
>
>
Off-topic: In some sense, I'm happy to hear something about pickleability.
But in some sense not.
I think there are three kinds of people regarding pickleability:
1. Those who don't care about anything being pickleable
2. Those who care about some things being picklable
3. Those who care about all things being picklable
Personally, I'd like to belong to group 3, but because group 3 cannot even
attempt to coexist with groups 1 and 2, I actually belong to group 1 most
of the time.
––Koos
> References
> ==
>
> .. [1] PEP 484 section about ``__getattr__`` in stub files
>(https://www.python.org/dev/peps/pep-0484/#stub-files)
>
> .. [2] The reference implementation
>(https://github.com/ilevkivskyi/cpython/pull/3/files)
>
>
> Copyright
> =
>
> This document has been placed in the public domain.
>
>
>
> ..
>Local Variables:
> mode: indented-text
>indent-tabs-mode: nil
>sentence-end-double-space: t
>fill-column: 70
>coding: utf-8
>End:
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> k7hoven%40gmail.com
>
>
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 560: bases classes / confusion
For anyone confused about similar things, I expect you to be interested in my post on python-ideas from today: https://mail.python.org/pipermail/python-ideas/2017-November/047896.html ––Koos On Wed, Nov 15, 2017 at 4:20 PM, Jim J. Jewett wrote: > (1) I found the following (particularly "bases classes") very confusing: > > """ > If an object that is not a class object appears in the bases of a class > > definition, then ``__mro_entries__`` is searched on it. If found, > it is called with the original tuple of bases as an argument. The result > of the call must be a tuple, that is unpacked in the bases classes in place > of this object. (If the tuple is empty, this means that the original bases > is > simply discarded.) > """ > > Based on the following GenericAlias/NewList/Tokens example, I think I > now I understand what you mean, and would have had somewhat less > difficulty if it were expressed as: > > """ > When an object that is not a class object appears in the (tuple of) > bases of a class > definition, then attribute ``__mro_entries__`` is searched on that > non-class object. If ``__mro_entries__`` found, > it is called with the entire original tuple of bases as an argument. The > result > of the call must be a tuple, which is unpacked and replaces only the > non-class object in the tuple of bases. (If the tuple is empty, this > means that the original bases > is > simply discarded.) > """ > > Note that this makes some assumptions about the __mro_entries__ > signature that I wasn't quite sure about from the example. So > building on that: > > class ABList(A, NewList[int], B): > > I *think* the following will happen: > > "NewList[int]" will be evaluated, and __class_getitem__ called, so > that the bases tuple will be (A, GenericAlias(NewList, int), B) > > # (A) I *think* __mro_entries__ gets called with the full tuple, > # instead of just the object it is found on. > # (B) I *think* it is called on the results of evaluating > # the terms within the tuple, instead of the original > # string representation. > _tmp = __mro_entries__(A, GenericAlias(NewList, int), B) > > # (C) I *think* __mro_entries__ returns a replacement for > # just the single object, even though it was called on > # the whole tuple, without knowing which object it > # represents. > bases = (A, _tmp, B) > > # (D) If there are two non-class objects, I *think* the > # second one gets the same arguments as the first, > # rather than an intermediate tuple with the first such > # object already substituted out. > > -jJ > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > k7hoven%40gmail.com > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 560: bases classes / confusion
On Wed, Nov 15, 2017 at 5:37 PM, Nick Coghlan wrote: > On 16 November 2017 at 00:20, Jim J. Jewett wrote: > >> I *think* the following will happen: >> >> "NewList[int]" will be evaluated, and __class_getitem__ called, so >> that the bases tuple will be (A, GenericAlias(NewList, int), B) >> >> # (A) I *think* __mro_entries__ gets called with the full tuple, >> # instead of just the object it is found on. >> # (B) I *think* it is called on the results of evaluating >> # the terms within the tuple, instead of the original >> # string representation. >> _tmp = __mro_entries__(A, GenericAlias(NewList, int), B) >> >> # (C) I *think* __mro_entries__ returns a replacement for >> # just the single object, even though it was called on >> # the whole tuple, without knowing which object it >> # represents. >> bases = (A, _tmp, B) >> > > My understanding of the method signature: > > def __mro_entries__(self, orig_bases): > ... > return replacement_for_self > > My assumption as to the purpose of the extra complexity was: > > - given orig_bases, a method could avoid injecting bases already listed if > it wanted to > - allowing multiple items to be returned provides a way to > programmatically combine mixins without having to define a new subclass for > each combination > > Thanks, this might provide an answer to my question about multiple mro entries here https://mail.python.org/pipermail/python-ideas/2017-November/047897.html ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 562
On Wed, Nov 15, 2017 at 8:02 PM, Ethan Furman wrote: > On 11/15/2017 04:55 AM, Koos Zevenhoven wrote: > >> On Tue, Nov 14, 2017 at 10:34 PM, Ivan Levkivskyi wrote: >> > > >> Rationale >>> = >>> >>> [...] It would be convenient to simplify this >>> procedure by recognizing ``__getattr__`` defined directly in a module >>> that >>> would act like a normal ``__getattr__`` method >>> >> >> > >> [...] > >> > >> Specification >>> = >>> >>> >> The ``__getattr__`` function at the module level should accept one > argument > >> which is the name of an attribute and return the computed value or raise >>> an ``AttributeError``:: >>> >> >>def __getattr__(name: str) -> Any: ... >>> >> >> This function will be called only if ``name`` is not found in the module >>> through the normal attribute lookup. >>> >> >> The Rationale (quoted in the beginning of this email) easily leaves a >> different impression of this. >> > > I don't see how. This is exactly the way normal __getattr__ works. > > > Oh sorry, I think I put this email together too quickly. I was writing down a bunch of thoughts I had earlier but hadn't written down. I think I was mixing this up in my head with overriding __getitem__ for the module namespace dict and __class_getitem__ from PEP 560, which only gets called if the metaclass doesn't implement __getitem__ (IIRC). But I did have another thought related to this. I was wondering whether the lack of passing the module to the methods as `self` would harm future attempts to generalize these ideas. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 560: bases classes / confusion
On Thu, Nov 16, 2017 at 6:28 PM, brent bejot wrote: > Hello all, > > Noticed that "MRO" is not actually defined in the PEP and it seems like it > should be. Probably in the Performance section where the abbreviation is > first used outside of a function name. > > I don't think it will hurt if I suggest that __bases__, bases, "original bases", mro, __orig_bases__, MRO, __mro__ and "concatenated mro entries" are all defined as synonyms of each other, except with different meanings :-) ––Koos > -Brent > > On Thu, Nov 16, 2017 at 7:22 AM, Ivan Levkivskyi > wrote: > >> On 16 November 2017 at 07:56, Nick Coghlan wrote: >> >>> On 16 November 2017 at 04:39, Ivan Levkivskyi >>> wrote: >>> >>>> Nick is exactly right here. Jim, if you want to propose alternative >>>> wording, then we could consider it. >>>> >>> >>> Jim also raised an important point that needs clarification at the spec >>> level: given multiple entries in "orig_bases" with __mro_entries__ methods, >>> do all such methods get passed the *same* orig_bases tuple? Or do they >>> receive partially resolved ones, such that bases listed before them have >>> already been resolved to their MRO entries by the time they run. >>> >>> >>> >> Yes, they all get the same initial bases tuple as an argument. Passing >> updated ones will cost a bit more and I don't think it will be needed (in >> the worst case a base can resolve another base by calling its >> __mro_entries__ manually). >> I will clarify this in the PEP. >> >> -- >> Ivan >> >> >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/brent. >> bejot%40gmail.com >> >> > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > k7hoven%40gmail.com > > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python possible vulnerabilities in concurrency
On Thu, Nov 16, 2017 at 6:53 AM, Guido van Rossum wrote: > On Wed, Nov 15, 2017 at 6:50 PM, Guido van Rossum > wrote: >> >> >> Actually it linked to http://standards.iso.org/ittf/ >> PubliclyAvailableStandards/index.html from which I managed to download >> what looks like the complete c061457_ISO_IEC_TR_24772_2013.pdf (336 >> pages) after clicking on an "I accept" button (I didn't read what I >> accepted :-). The $200 is for the printed copy I presume. >> > > So far I learned one thing from the report. They use the term > "vulnerabilities" liberally, defining it essentially as "bug": > > All programming languages contain constructs that are incompletely >> specified, exhibit undefined behaviour, are implementation-dependent, or >> are difficult to use correctly. The use of those constructs may therefore >> give rise to *vulnerabilities*, as a result of which, software programs >> can execute differently than intended by the writer. >> > > They then go on to explain that sometimes vulnerabilities can be > exploited, but I object to calling all bugs vulnerabilities -- that's just > using a scary word to get attention for a sleep-inducing document > containing such gems as "Use floating-point arithmetic only when absolutely > needed" (page 230). > > I don't like such a definition of "vulnerability" either. Some bugs can be vulnerabilities (those that can be exploited) and some vulnerabilities can be bugs. But there are definitely types of vulnerabilities that are not bugs––the DoS vulnerability that is eliminated by hash randomization is one. There may also be a gray area of bugs that can be vulnerabilities but only in some special situation. I think it's ok to call those vulnerabilities too. ––Koos PS. How come I haven't seen a proposal to remove the float type from builtins yet?-) -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python possible vulnerabilities in concurrency
On Fri, Nov 17, 2017 at 3:40 PM, Koos Zevenhoven wrote: > On Thu, Nov 16, 2017 at 6:53 AM, Guido van Rossum > wrote: > >> On Wed, Nov 15, 2017 at 6:50 PM, Guido van Rossum >> wrote: >>> >>> >>> Actually it linked to http://standards.iso.org/ittf/ >>> PubliclyAvailableStandards/index.html from which I managed to download >>> what looks like the complete c061457_ISO_IEC_TR_24772_2013.pdf (336 >>> pages) after clicking on an "I accept" button (I didn't read what I >>> accepted :-). The $200 is for the printed copy I presume. >>> >> >> So far I learned one thing from the report. They use the term >> "vulnerabilities" liberally, defining it essentially as "bug": >> >> All programming languages contain constructs that are incompletely >>> specified, exhibit undefined behaviour, are implementation-dependent, or >>> are difficult to use correctly. The use of those constructs may therefore >>> give rise to *vulnerabilities*, as a result of which, software programs >>> can execute differently than intended by the writer. >>> >> >> They then go on to explain that sometimes vulnerabilities can be >> exploited, but I object to calling all bugs vulnerabilities -- that's just >> using a scary word to get attention for a sleep-inducing document >> containing such gems as "Use floating-point arithmetic only when absolutely >> needed" (page 230). >> >> > I don't like such a definition of "vulnerability" either. Some bugs can > be vulnerabilities (those that can be exploited) and some vulnerabilities > can be bugs. But there are definitely types of vulnerabilities that are not > bugs––the DoS vulnerability that is eliminated by hash randomization is one. > > There may also be a gray area of bugs that can be vulnerabilities but only > in some special situation. I think it's ok to call those vulnerabilities > too. > > Just to clarify the obvious: By the above, I *don't* mean that one could use the word "vulnerability" for any functionality that can be used in such a way that it creates a vulnerability. For example, `eval` or `exec` or `open` by themselves are not vulnerabilities. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make the stable API-ABI usable
> "stable API" but change its implementation to a *function call* rather > than the existing macro, so the compiled C extension will use a > function call and so don't rely on the ABI anymore. > > > My plan is to have two main milestones: > > (1) Python 3.7: Extend the *existing* opt-in "stable API" which > requires to compile C extensions in a special mode. Add maybe an > option in distutils to ease the compilation of a C extension with the > "stable API"? > > (2) In Python 3.8, --if the project is successful and the performance > overhead is acceptable compared the advantages of having C extensions > working on multiple Python verisons--, make the "stable API (without > implementation details)" the default, but add a new opt-in option to > give access to the "full API (with implementation details)" for > debuggers and other people who understand what they do (like Cython?). > > Note: currently, the "stable API" is accessible using Py_LIMITED_API > define, and the "full API" is accessible using Py_BUILD_CORE define. > No define gives the current C API. > > > My problem is more on the concrete implementation: > > * Need to provide two different API using the same filenames (like: > #include "Python.h") > > * Need to extend distutils to have a flag to compile a C extension > with one specific API (define Py_LIMITED_API or Py_BUILD_CORE?) > > * Need to test many C extensions and check how many extensions are broken > > > My plan for Python 3.7 is to not touch the current API at all. There > is no risk of backward incompatibility. You should only get issues if > you opt-in for the new API without implementation details. > > > Final note: Nothing new under the sun: PyPy already implemented my > "idea"! Where the idea is a C API without macros; PyTuple_GET_ITEM() > is already a function call in PyPy ;-) > > > Final question: Is it acceptable to iterate on many small changes on > the C API implement this idea in Python 3.7? Maybe only write partial > implementation, and finish it in Python 3.8? > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > k7hoven%40gmail.com > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __future__ imports and breaking code (was: PEP 563: Postponed Evaluation of Annotations)
Previously, I expressed some concerns about PEP 563 regarding what should happen when a string is used as an annotation. Since my point here is more general, I'm starting yet another thread. For a lot of existing type-annotated code, adding "from __tuture__ import annotations" [1] *doesn't break anything*. But that doesn't seem right. The whole point of __future__ imports is to break things. Maybe the __future__ import will not give a 100% equivalent functionality to what will be in Python 4 by default, but anyway, it's Python 4 that should break as little as possible. This leaves the breaking business to the future import, if necessary. If someone cares enough to add the future import that avoids needing string annotations for forward references, it shouldn't be such a big deal to get a warning if there's a string annotation left. But the person upgrading to Python 4 (or whatever they might be upgrading) will have a lot less motivation to figure out what went wrong. Then again, code that works in both Python 3 and 4 could still have the future import. But that would defeat the purpose of Python 4 as a clean and high-performance dynamic language. —Koos [1] As defined in the PEP 563 draft: https://mail.python.org/pipermail/python-dev/2017-November/150062.html -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 563: Postponed Evaluation of Annotations
On Mon, Nov 13, 2017 at 11:59 PM, Brett Cannon wrote: [..] > On Sun, Nov 12, 2017, 10:22 Koos Zevenhoven, wrote: > > >> >> There's two thing I don't understand here: >> >> * What does it mean to preserve the string verbatim? No matter how I read >> it, I can't tell if it's with quotes or without. >> >> Maybe I'm missing some context. >> > > I believe the string passes through unchanged (i.e. no quotes). Think of > the PEP as simply turning all non-string annotations into string ones. > > Ok, maybe that was just wishful thinking on my part ;-). More info in the other threads, for example: https://mail.python.org/pipermail/python-dev/2017-November/150642.html https://mail.python.org/pipermail/python-dev/2017-November/150637.html -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Comments on PEP 563 (Postponed Evaluation of Annotations)
On Mon, Nov 20, 2017 at 7:58 PM, Lukasz Langa wrote: > I agree with you. The special handling of outermost strings vs. strings > embedded inside annotations bugged me a lot. Now you convinced me that this > functionality should be moved to `get_type_hints()` and the __future__ > import shouldn't try to special-case this one instance, while leaving > others as is. > > > That's better. I don't necessarily care if there will be a warning when a string is given as annotation, but if the idea is to simplify things for the future and get rid of strings to represent types, then this would be a good moment to gently "enforce" it. ––Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] generator vs iterator etc. (was: How assignment should work with generators?)
On Mon, Nov 27, 2017 at 3:55 PM, Steven D'Aprano wrote: > On Mon, Nov 27, 2017 at 12:17:31PM +0300, Kirill Balunov wrote: > > > > 2. Should this work only for generators or for any iterators? > > I don't understand why you are even considering singling out *only* > generators. A generator is a particular implementation of an iterator. I > can write: > > def gen(): >yield 1; yield 2; yield 3 > > it = gen() > > or I can write: > > it = iter([1, 2, 3]) > > and the behaviour of `it` should be identical. > > > I can see where this is coming from. The thing is that "iterator" and "generator" are mostly synonymous, except two things: (1) Generators are iterators that are produced by a generator function (2) Generator functions are sometimes referred to as just "generators" The concept of "generator" thus overlaps with both "iterator" and "generator function". Then there's also "iterator" and "iterable", which are two different things: (3) If `obj` is an *iterable*, then `it = iter(obj)` is an *iterator* (over the contents of `obj`) ( 4) Iterators yield values, for example on explicit calls to next(it). Personally I have leaned towards keeping a clear distinction between "generator function" and "generator", which leads to the situation that "generator" and "iterator" are mostly synonymous for me. Sometimes, for convenience, I use the term "generator" to refer to "iterators" more generally. This further seems to have a minor benefit that "generators" and "iterables" are less easily confused with each other than "iterators" and "iterables". I thought about this issue some time ago for the `views` package, which has a separation between sequences (seq) and other iterables (gen): https://github.com/k7hoven/views The functionality provided by `views.gen` is not that interesting—it's essentially a subset of itertools functionality, but with an API that parallels `views.seq` which works with sequences (iterable, sliceable, chainable, etc.). I used the name `gen`, because iterator/iterable variants of the functionality can be implemented with generator functions (although also with other kinds of iterators/iterables). Calling the thing `iter` would have conflicted with the builtin `iter`. HOWEVER, this naming can be confusing for those that lean more towards using "generator" to also mean "generator function", and for those that are comfortable with the term "iterator" despite its resemblance to "iterable". Now I'm actually seriously considering to consider renaming `views.gen` to ` views.iter` when I have time. After all, there's already `views.range` which "conflicts" with the builtin range. Anyway, the point is that the naming is suboptimal. SOLUTION: Maybe (a) all iterators should be called iterators or (b) all iterators should be called generators, regardless of whether they are somehow a result of a generator function having been called in the past. (I'm not going into the distinction between things that can receive values via `send` or any other possible distinctions between different types of iterators and iterables.) —Koos (discussion originated from python-ideas, but cross-posted to python-dev in case there's more interest there) -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Thoughts on "contexts". PEPs 550, 555, 567, 568
Hi all, I feel like I should write some thoughts regarding the "context" discussion, related to the various PEPs. I like PEP 567 (+ 567 ?) better than PEP 550. However, besides providing cvar.set(), I'm not really sure about the gain compared to PEP 555 (which could easily have e.g. a dict-like interface to the context). I'm still not a big fan of "get"/"set" here, but the idea was indeed to provide those on top of a PEP 555 type thing too. "Tokens" in PEP 567, seems to resemble assignment context managers in PEP 555. However, they feel a bit messy to me, because they make it look like one could just set a variable and then revert the change at any point in time after that. PEP 555 is in fact a simplification of my previous sketch that had a .set(..) in it, but was somewhat different from PEP 550. The idea was to always explicitly define the scope of contextvar values. A context manager / with statement determined the scope of .set(..) operations inside the with statement: # Version A: cvar.set(1) with context_scope(): cvar.set(2) assert cvar.get() == 2 assert cvar.get() == 1 Then I added the ability to define scopes for different variables separately: # Version B cvar1.set(1) cvar2.set(2) with context_scope(cvar1): cvar1.set(11) cvar2.set(22) assert cvar1.get() == 1 assert cvar2.get() == 22 However, in practice, most libraries would wrap __enter__, set and __exit__ into another context manager. So maybe one might want to allow something like # Version C: assert cvar.get() == something with context_scope(cvar, 2): assert cvar.get() == 2 assert cvar.get() == something But this then led to combining "__enter__" and ".set(..)" into Assignment.__enter__ -- and "__exit__" into Assignment.__exit__ like this: # PEP 555 draft version: assert cvar.value == something with cvar.assign(1): assert cvar.value == 1 assert cvar.value == something Anyway, given the schedule, I'm not really sure about the best thing to do here. In principle, something like in versions A, B and C above could be done (I hope the proposal was roughly self-explanatory based on earlier discussions). However, at this point, I'd probably need a lot of help to make that happen for 3.7. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thoughts on "contexts". PEPs 550, 555, 567, 568
On Jan 10, 2018 07:17, "Yury Selivanov" wrote: Wasn't PEP 555 rejected by Guido? What's the point of this post? I sure hope there is a point. I don't think mentioning PEP 555 in the discussions should hurt. A typo in my post btw: should be "PEP 567 (+568 ?)" in the second paragraph of course. -- Koos (mobile) Yury On Wed, Jan 10, 2018 at 4:08 AM Koos Zevenhoven wrote: > Hi all, > > I feel like I should write some thoughts regarding the "context" > discussion, related to the various PEPs. > > I like PEP 567 (+ 567 ?) better than PEP 550. However, besides providing > cvar.set(), I'm not really sure about the gain compared to PEP 555 (which > could easily have e.g. a dict-like interface to the context). I'm still not > a big fan of "get"/"set" here, but the idea was indeed to provide those on > top of a PEP 555 type thing too. > > "Tokens" in PEP 567, seems to resemble assignment context managers in PEP > 555. However, they feel a bit messy to me, because they make it look like > one could just set a variable and then revert the change at any point in > time after that. > > PEP 555 is in fact a simplification of my previous sketch that had a > .set(..) in it, but was somewhat different from PEP 550. The idea was to > always explicitly define the scope of contextvar values. A context manager > / with statement determined the scope of .set(..) operations inside the > with statement: > > # Version A: > cvar.set(1) > with context_scope(): > cvar.set(2) > > assert cvar.get() == 2 > > assert cvar.get() == 1 > > Then I added the ability to define scopes for different variables > separately: > > # Version B > cvar1.set(1) > cvar2.set(2) > with context_scope(cvar1): > cvar1.set(11) > cvar2.set(22) > > assert cvar1.get() == 1 > assert cvar2.get() == 22 > > > However, in practice, most libraries would wrap __enter__, set and > __exit__ into another context manager. So maybe one might want to allow > something like > > # Version C: > assert cvar.get() == something > with context_scope(cvar, 2): > assert cvar.get() == 2 > > assert cvar.get() == something > > > But this then led to combining "__enter__" and ".set(..)" into > Assignment.__enter__ -- and "__exit__" into Assignment.__exit__ like this: > > # PEP 555 draft version: > assert cvar.value == something > with cvar.assign(1): > assert cvar.value == 1 > > assert cvar.value == something > > > Anyway, given the schedule, I'm not really sure about the best thing to do > here. In principle, something like in versions A, B and C above could be > done (I hope the proposal was roughly self-explanatory based on earlier > discussions). However, at this point, I'd probably need a lot of help to > make that happen for 3.7. > > -- Koos > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > yselivanov.ml%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thoughts on "contexts". PEPs 550, 555, 567, 568
The status of PEP 555 is just a side track. Here, I took a step back compared to what went into PEP 555. —Koos On Wed, Jan 10, 2018 at 6:21 PM, Guido van Rossum wrote: > The current status of PEP 555 is "Withdrawn". I have no interest in > considering it any more, so if you'd rather see a decision from me I'll be > happy to change it to "Rejected". > > On Tue, Jan 9, 2018 at 10:29 PM, Koos Zevenhoven > wrote: > >> On Jan 10, 2018 07:17, "Yury Selivanov" wrote: >> >> Wasn't PEP 555 rejected by Guido? What's the point of this post? >> >> >> I sure hope there is a point. I don't think mentioning PEP 555 in the >> discussions should hurt. >> >> A typo in my post btw: should be "PEP 567 (+568 ?)" in the second >> paragraph of course. >> >> -- Koos (mobile) >> >> >> Yury >> >> On Wed, Jan 10, 2018 at 4:08 AM Koos Zevenhoven >> wrote: >> >>> Hi all, >>> >>> I feel like I should write some thoughts regarding the "context" >>> discussion, related to the various PEPs. >>> >>> I like PEP 567 (+ 567 ?) better than PEP 550. However, besides providing >>> cvar.set(), I'm not really sure about the gain compared to PEP 555 (which >>> could easily have e.g. a dict-like interface to the context). I'm still not >>> a big fan of "get"/"set" here, but the idea was indeed to provide those on >>> top of a PEP 555 type thing too. >>> >>> "Tokens" in PEP 567, seems to resemble assignment context managers in >>> PEP 555. However, they feel a bit messy to me, because they make it look >>> like one could just set a variable and then revert the change at any point >>> in time after that. >>> >>> PEP 555 is in fact a simplification of my previous sketch that had a >>> .set(..) in it, but was somewhat different from PEP 550. The idea was to >>> always explicitly define the scope of contextvar values. A context manager >>> / with statement determined the scope of .set(..) operations inside the >>> with statement: >>> >>> # Version A: >>> cvar.set(1) >>> with context_scope(): >>> cvar.set(2) >>> >>> assert cvar.get() == 2 >>> >>> assert cvar.get() == 1 >>> >>> Then I added the ability to define scopes for different variables >>> separately: >>> >>> # Version B >>> cvar1.set(1) >>> cvar2.set(2) >>> with context_scope(cvar1): >>> cvar1.set(11) >>> cvar2.set(22) >>> >>> assert cvar1.get() == 1 >>> assert cvar2.get() == 22 >>> >>> >>> However, in practice, most libraries would wrap __enter__, set and >>> __exit__ into another context manager. So maybe one might want to allow >>> something like >>> >>> # Version C: >>> assert cvar.get() == something >>> with context_scope(cvar, 2): >>> assert cvar.get() == 2 >>> >>> assert cvar.get() == something >>> >>> >>> But this then led to combining "__enter__" and ".set(..)" into >>> Assignment.__enter__ -- and "__exit__" into Assignment.__exit__ like this: >>> >>> # PEP 555 draft version: >>> assert cvar.value == something >>> with cvar.assign(1): >>> assert cvar.value == 1 >>> >>> assert cvar.value == something >>> >>> >>> Anyway, given the schedule, I'm not really sure about the best thing to >>> do here. In principle, something like in versions A, B and C above could be >>> done (I hope the proposal was roughly self-explanatory based on earlier >>> discussions). However, at this point, I'd probably need a lot of help to >>> make that happen for 3.7. >>> >>> -- Koos >>> >>> ___ >>> Python-Dev mailing list >>> Python-Dev@python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: https://mail.python.org/mailma >>> n/options/python-dev/yselivanov.ml%40gmail.com >>> >> >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% >> 40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Thoughts on "contexts". PEPs 550, 555, 567, 568
The timing of all of this is unfortunate. I'm sorry that my participation
in the discussion has been a bit "on-off" lately. But my recent
contributions have involved studying things like the interaction of
threading/concurrency aspects of signal handling, as well as investigating
subtleties of various proposals for context variables, including my own.
Those are not exactly low-hanging fruit, and I'm sorry about not being able
to eat them.
It is also unfortunate that I haven't written down this proposal
("versions" A-C) to anywhere near the amount of precision than I did for
PEP 555, which wasn't 100% specified in the first draft either. For
consideration, I just thought it's better to at least mention it, so that
those that now have a good understanding of the issues involved could
perhaps understand it. I can add more detail, but to make it a full
proposal now, I would probably need to join forces with a coauthor (with a
good understanding of these issues) to figure out missing parts. I could
tune in later to finish the PEP and write docs in case the approach gets
implemented.
-- Koos
On Wed, Jan 10, 2018 at 7:17 PM, Guido van Rossum wrote:
> I'm sorry, Koos, but based on your past contributions I am not interested
> in discussing this topic with you.
>
> On Wed, Jan 10, 2018 at 8:58 AM, Koos Zevenhoven
> wrote:
>
>> The status of PEP 555 is just a side track. Here, I took a step back
>> compared to what went into PEP 555.
>>
>> —Koos
>>
>>
>> On Wed, Jan 10, 2018 at 6:21 PM, Guido van Rossum
>> wrote:
>>
>>> The current status of PEP 555 is "Withdrawn". I have no interest in
>>> considering it any more, so if you'd rather see a decision from me I'll be
>>> happy to change it to "Rejected".
>>>
>>> On Tue, Jan 9, 2018 at 10:29 PM, Koos Zevenhoven
>>> wrote:
>>>
>>>> On Jan 10, 2018 07:17, "Yury Selivanov"
>>>> wrote:
>>>>
>>>> Wasn't PEP 555 rejected by Guido? What's the point of this post?
>>>>
>>>>
>>>> I sure hope there is a point. I don't think mentioning PEP 555 in the
>>>> discussions should hurt.
>>>>
>>>> A typo in my post btw: should be "PEP 567 (+568 ?)" in the second
>>>> paragraph of course.
>>>>
>>>> -- Koos (mobile)
>>>>
>>>>
>>>> Yury
>>>>
>>>> On Wed, Jan 10, 2018 at 4:08 AM Koos Zevenhoven
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I feel like I should write some thoughts regarding the "context"
>>>>> discussion, related to the various PEPs.
>>>>>
>>>>> I like PEP 567 (+ 567 ?) better than PEP 550. However, besides
>>>>> providing cvar.set(), I'm not really sure about the gain compared to PEP
>>>>> 555 (which could easily have e.g. a dict-like interface to the context).
>>>>> I'm still not a big fan of "get"/"set" here, but the idea was indeed to
>>>>> provide those on top of a PEP 555 type thing too.
>>>>>
>>>>> "Tokens" in PEP 567, seems to resemble assignment context managers in
>>>>> PEP 555. However, they feel a bit messy to me, because they make it look
>>>>> like one could just set a variable and then revert the change at any point
>>>>> in time after that.
>>>>>
>>>>> PEP 555 is in fact a simplification of my previous sketch that had a
>>>>> .set(..) in it, but was somewhat different from PEP 550. The idea was to
>>>>> always explicitly define the scope of contextvar values. A context manager
>>>>> / with statement determined the scope of .set(..) operations inside the
>>>>> with statement:
>>>>>
>>>>> # Version A:
>>>>> cvar.set(1)
>>>>> with context_scope():
>>>>> cvar.set(2)
>>>>>
>>>>> assert cvar.get() == 2
>>>>>
>>>>> assert cvar.get() == 1
>>>>>
>>>>> Then I added the ability to define scopes for different variables
>>>>> separately:
>>>>>
>>>>> # Version B
>>>>> cvar1.set(1)
>>>>> cvar2.set(2)
>>>>> with context_scope(cvar1):
>>>>> cvar1.set(11)
>>>>> cvar2.set(22)
>>&g
Re: [Python-Dev] Thoughts on "contexts". PEPs 550, 555, 567, 568
I'll quickly add a few things below just in case there's anyone that cares. On Wed, Jan 10, 2018 at 2:06 AM, Koos Zevenhoven wrote: > > The idea was to always explicitly define the scope of contextvar values. A > context manager / with statement determined the scope of .set(..) > operations inside the with statement: > > # Version A: > cvar.set(1) > with context_scope(): > cvar.set(2) > > assert cvar.get() == 2 > > assert cvar.get() == 1 > > Then I added the ability to define scopes for different variables > separately: > > # Version B > cvar1.set(1) > cvar2.set(2) > with context_scope(cvar1): > cvar1.set(11) > cvar2.set(22) > > assert cvar1.get() == 1 > assert cvar2.get() == 22 > > > However, in practice, most libraries would wrap __enter__, set and > __exit__ into another context manager. So maybe one might want to allow > something like > > # Version C: > assert cvar.get() == something > with context_scope(cvar, 2): > assert cvar.get() == 2 > > assert cvar.get() == something > > Note here, that the point is to get a natural way to "undo" changes made to variables when exiting the scope. Undoing everything that is done within the defined scope is a very natural way to do it. Undoing individual .set(..) operations is more problematic. Features B+C could be essentially implemented as described in PEP 555, except with context_scope(cvar) being essentially the same as pushing and popping an empty Assignment object onto the reverse-linked stack. By empty, I mean a "key-value pair with a missing value". Then any set operations would replace the topmost assignment object for that variable with a new key-value pair (or push a new Assignment if there isn't one). However, to also get feature A, the stack may have to contain full mappings instead of assignemnt objects with just one key-value pair. I hope that clarifies some parts. Otherwise, in terms of semantics, the same things apply as for PEP 555 when it comes to generator function calls and next(..) etc., so we'd need to make sure it works well enough for all use cases. For instance, I'm not quite sure if I have a good enough understanding of the timeout example that Nathaniel wrote in the PEP 550 discussion to tell what would be required in terms of semantics, but I suppose it should be fine. -- Koos > But this then led to combining "__enter__" and ".set(..)" into > Assignment.__enter__ -- and "__exit__" into Assignment.__exit__ like this: > > # PEP 555 draft version: > assert cvar.value == something > with cvar.assign(1): > assert cvar.value == 1 > > assert cvar.value == something > > > Anyway, given the schedule, I'm not really sure about the best thing to do > here. In principle, something like in versions A, B and C above could be > done (I hope the proposal was roughly self-explanatory based on earlier > discussions). However, at this point, I'd probably need a lot of help to > make that happen for 3.7. > > -- Koos > > -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 567 v3
On Thu, Jan 18, 2018 at 3:53 AM, Yury Selivanov wrote: [] > > Given the time frame of the Python 3.7 release schedule it was decided > to defer this proposal to Python 3.8. > It occurs to me that I had misread this to refer to the whole PEP. Although I thought it's kind of sad that after all this, contextvars still would not make it into 3.7, I also thought that it might be the right decision. As you may already know, I think there are several problems with this PEP. Would it be worth it to write down some thoughts on this PEP in the morning? -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Intention to accept PEP 567 (Context Variables)
On Tue, Jan 23, 2018 at 2:23 AM, Victor Stinner wrote: > The PEP 555 looks a competitor PEP of the PEP 567. Since the Yury's > PEP 567 was approved, I understand that Koos's PEP 555 should be > rejected, no? > > If Guido prefers to reject it, I assume he'll say so. Anyway, it's still waiting for me to add references to earlier discussions and perhaps summaries of some discussions. Personally, I need to find some time to properly catch up with the latest discussion to figure out why PEP 567 is better than PEP 555 (or similar with .set(..), or PEP 550), despite problems of reasoning about the scopes of variables and unset tokens. In any case, congrats, Yury! This hasn't been an easy one for any of us, and it seems like the implementation required quite a beastly patch too in the end. —Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
Nick Coghlan wrote: > On 7 April 2016 at 03:26, Brett Cannon wrote: >> >> Name: __path__, __fspath__, or something else? > > __fspath__ > I think I might like this dunder name because it does not clutter the list of regular methods and attributes, and is perhaps more pythonic. >> Method or attribute? (changes what kind of one-liner you might use in >> libraries, but I think historically all protocols have been methods and the >> serialized string representation might be costly to build) > > Method, as long as there's a helper function somewhere As a further minor benefit of it being a method, it may be easier to distinguish it from from `__path__`, which is an iterable attribute. >> Built-in? (name is dependent on #1 if we add one) > > os.fspath (alongside os.fsencode and os.fsdecode) > > (Putting this in a module low in the dependency stack makes it easy > for other modules to access without pulling in all of pathlib's > dependencies) Strong +1 on putting it in os. This should also be implemented in DirEntry, instances of which are "yielded" by os.scandir. Also, you have a strong case regarding naming with the 'fs' prefix. It is also easier to read fspath as f-s-path than it is to read ospath as o-s-path, because ospath could also be pronounced as a single (meaningless?) word. I'm still thinking a little bit about 'pathname', which to me sounds more like a string than fspath does [1]. It would be nice to have the string/path distinction especially when pathlib adoption grows larger. But who knows, maybe somewhere in the far future, no-one will care much about fspath, fsencode, fsdecode or os.path. >> Add the method/attribute to str? (I assume so, much like __index__() is on >> int, but I have not seen it explicitly stated so I would rather clarify it) > > Makes sense If added to str, it should also be added to bytes. But will that then return str or bytes? See also the next point. > Expand the C API to have something like PyObject_Path()? > > PyUnicode_FromFSPath, perhaps? The return type is well-defined here, > so it can be done as an alternate constructor, and the C API > counterparts of os.fsdecode and os.fsencode are PyUnicode functions > (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) What about DirEntry, which may have a bytes representation? I would expect the function return type of os.fspath to be Union[str, bytes], unless bytes pathnames are decoded with surrogate escapes. [1] https://mail.python.org/pipermail/python-ideas/2016-April/039595.html PS. I have been reading this list occasionally on the google groups mirror, and I now subscribed to it just to send this. (BTW, I probably broke the thread, as I did not have Nick's email in my inbox to reply to. Sorry about that.) I'll have to mention that I was surprised, to say the least, to find that the pathlib discussion had moved here from python-ideas, where I had mentioned I was working on a proposal. Then, I also found that the solution discussed here was seemingly an improved version of what I had proposed on python-ideas somewhat earlier [1], but did not get any reactions to. While I can only make guesses about what happened, these kinds of things easily make you go from "Hey, maybe I'll be able to do something to improve Python!" to "These people don't seem to want me here or appreciate my efforts.". Not to accuse anyone in particular; just to let people know. Anyway, I somehow got sucked into thinking deeply about pathlib etc. (which I do use). Not that I really have much at stake here, except spending ridiculous amounts of time thinking about paths, mainly during my Easter holidays and after that. I really had a hard time explaining to friends and family what the heck I was doing ;). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: > On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: >> >> I'm still thinking a little bit about 'pathname', which to me sounds >> more like a string than fspath does [1]. > > > I like that a lot - or even "__pathstr__" or "__pathstring__" > > after all, we're making a big deal out of the fact that a path is *not a > string*, but rather a string is a *representation* (or serialization) of a > path. For me, the point here is the reverse: that any str is not a path, and that it is misleading to call it *path* when whole point is to make it *not* a specialized path object but a plain string. I think it's ok to think of a path as special kind of string. For instance, an URI is explicitly defined as a *sequence of characters*, and URIs can be thought of as a more recent, improved and broadened concept than paths. This is the point of view I took in my recent proposal, but I don't think it's the only valid way to think about paths "in theory". I like the "serialization" interpretation as well, but i tend to think that that string serialization is what is called a path. Anyway, I don't think these philosophical considerations should dictate how Python is implemented. But it is always good to also have a valid theoretical point of view to back up a design decision. > For the record, this is pretty rare -- and it was announced on -ideas that > the discussion had started up here -- maybe you missed that post? If you mean in Ethan's response to my proposal, I noticed that, but the discussions here had already gone quite far by that time. Even more so by the time I had time to see what was going on. I do have to say this is not the first time I felt there was some sort of hostility towards newcomers on python-ideas. Sure, it might be partly because those people don't know the culture on the list, but I'm not sure if that should be used as an excuse. -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 9:20 PM, Chris Barker wrote: > > we rejected plain old __path__ because this is already ued in another > context, but if we add "str" on the end, that's not longer an issue, so do > we need the "fs"? > > __pathstr__ # pathstring > Or perhaps __pathstring__ in case it may be or return byte strings. -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Fri, Apr 8, 2016 at 11:39 PM, R. David Murray wrote: > On Fri, 08 Apr 2016 19:24:44 -, Brett Cannon wrote: >> On Fri, 8 Apr 2016 at 12:10 Chris Angelico wrote: >> >> > On Sat, Apr 9, 2016 at 5:03 AM, Chris Barker >> > wrote: >> > > On Fri, Apr 8, 2016 at 11:34 AM, Koos Zevenhoven >> > wrote: >> > >> >> > >> > >> > >> > __pathstr__ # pathstring >> > >> > >> > >> >> > >> Or perhaps __pathstring__ in case it may be or return byte strings. > > But there are other paths than OS file system paths. I prefer > __fspath__ or __os_path__ myself. I think the fact that it is a string > is implied by the fact that it is getting us the thing we can pass > to the os (since Python3 deals with os paths as strings unless you > specify otherwise, only converting them back to bytes, on unix, at the last > moment). > > Heh, although I suppose one could make the argument that it should > return whatever the native OS wants, and save the low level code > from having to do that? Pass the path object all the way down > to that "final step" in the C layer? (Just ignore me, I'm sure > I'm only making trouble :) My favorites are fspath and pathname, and since this is a dunder methdod, it is not as crucial what it is called. I have the feeling the consensus is converging towards fspath? I'll comment on the bytes issue in the other thread. Boy these threads are all over the place! -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Fri, Apr 8, 2016 at 8:34 PM, Brett Cannon wrote: > On Fri, 8 Apr 2016 at 09:39 Ethan Furman wrote: >> > I thought the whole point off all this is that not any old string can be >> > a path! (whereas any int can be an index). Unless we go with Chris A's >> > suggestion that this be a more generic lossless string protocol, rather >> > than just for paths. >> >> That does seem to be a valid point against str.__fspath__. > > Yep, and I'm expecting we won't want that at this point. The fact that paths > need strings for low-level OS stuff is a historical and technical detail, so > no need to drag the entire str type into it if we can provide a reasonable > helper function (for either the ABC or magic method solution). I'm not sure I understand what these points are about. Anyway, disallowing str or bytes as pathnames will break backwards compatibility if done at some point in the future. There's no way around that. But regarding all this talk of mine about bytes is because it has not been completely clear to me if something can break when converting a bytes path to str. I did originally propose guaranteeing a str, but I am so far only 85% convinced that that does not cause any problems. I understand that fsencode(fsdecode(bytes_path)) should always be equal to bytes_path. But can some other path operations fail when there are surrogates in the strings? And again, not to forget DirEntry, which may have a byte string path. Either way, I suppose os.fspath should accept anything that has __fspath__ or is a str or bytes (whether these have the dunder method or not). Then the options are either to return Union[str, bytes] or to always return str. And if the latter does not cause any problems, I like it way better, and it seems others would do too. And in that case it would probably be time to deprecate bytes paths on posix too (on Windows, this is already the case). But do we know that converting all paths to str does not cause any problems? -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol
On Sat, Apr 9, 2016 at 12:53 AM, Brett Cannon wrote: > On Fri, 8 Apr 2016 at 14:23 Koos Zevenhoven wrote: > > At this point no one wants to touch bytes paths. If you need that level of > control because of multiple encodings within a single file system then you > will probably have to stick with managing bytes paths on your own to get the > encoding right. What does this mean? I assume you don't mean os.path.* would stop dealing with bytes? And if not, then you seem to mean that os.fspath would do nothing except call .__fspath__(). In that case, I think we should go back to it being an attribute (or property) and a variation of the now very famous idiom getattr(path, '__fspath__', path) and perhaps have os.fspath do exactly that. > And just because DirEntry supports bytes doesn't mean that any magic method > it gains has to carry that forward (it can always raise a TypeError if > necessary). No, but what if some code gets pathnames from whatever other places and passes them on to os.scandir. Whenever it happens to get a bytes path, a TypeError gets raised, but only when it picks one of the DirEntry objects and for instance tries to open(...) it. Of course, I'm not sure how common this is. > It really depends on how we choose to structure the > function in terms of just doing the right thing for objects that follow the > protocol or if we want to introduce some required structure for the > resulting path and implement some type guarantees so you have a better idea > of what you will be working with after calling the function. Do you have an example of potential 'required structure'? >> Then the options are either to return Union[str, bytes] or to >> always return str. And if the latter does not cause any problems, I >> like it way better, and it seems others would do too. > > You don't have to convert byte paths to str, you can simply raise an > exception in the face of them. > I thought the point was for existing APIs to start supporting path objects, wouldn't raising an exception break the API? -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
On Sat, Apr 9, 2016 at 10:16 AM, Ethan Furman wrote: > On 04/09/2016 12:07 AM, Victor Stinner wrote: >> >> os.DirEntry doesn't support bytes: os.scandir() only accept str. It's a >> deliberate choice. > > > 3.5.0 scandir supports bytes: > > --> huh = list(scandir(b'.')) > --> huh > [, , b'__MACOSX'>, , , b'index.html'>] > > --> huh[0].path > b'./minicourse-ajax-project' > > Maybe it's the bytes support in scandir that should be deprecated? (And not bytes support in general, which cannot be done on posix, as I hear Stephen T. will tell me). -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancments - method name only
On Sat, Apr 9, 2016 at 10:48 AM, Nick Coghlan wrote: > On 9 April 2016 at 04:25, Brett Cannon wrote: >> On Fri, 8 Apr 2016 at 11:13 Ethan Furman wrote: >>> On 04/08/2016 10:46 AM, Koos Zevenhoven wrote: >>> > On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: >>> >> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: >>> >>> >>> >>> I'm still thinking a little bit about 'pathname', which to me sounds >>> >>> more like a string than fspath does. >>> >> >>> >> >>> >> I like that a lot - or even "__pathstr__" or "__pathstring__" >>> >> after all, we're making a big deal out of the fact that a path is >>> >> *not a string*, but rather a string is a *representation* (or >>> >> serialization) of a path. >>> >>> That's a decent point. >>> >>> So the plausible choices are, I think: >>> >>> - __fspath__ # File System Path -- possible confusion with Path >> >> +1 > > I like __fspath__, but I'm also sympathetic to Koos' point that we're > really dealing with path *names* being produced via this protocol, > rather than the paths themselves. > > That would bring the completely explicit "__fspathname__" into the > mix, which would be comparable in length to "__getattribute__" as a > magic method name (both in terms of number of syllable and number of > characters). > > Considering the helper function usage, here's some examples in > combination with os.fsencode and os.fsdecode: > > # Status quo for binary/text path conversions > text_path = os.fsdecode(bytes_path) > bytes_path = os.fsencode(text_path) > > # Getting a text path from an arbitrary object > text_path = os.fspath(obj) # This doesn't scream "returns text!" to me > text_path = os.fspathname(obj) # This does > > # Getting a binary path from an arbitrary object > bytes_path = os.fsencode(os.fspath(obj)) > bytes_path = os.fsencode(os.fspathname(obj)) > > I'm starting to think the semantic nudge from the "name" suffix when > reading the code is worth the extra four characters when writing it > (keeping in mind that the whole point of this exercise is that most > folks *won't* be writing explicit conversions - the stdlib will handle > it on their behalf). > Regarding the name, I completely agree with Nick's reasoning (above). I'm not sure it's a high priority to make dunder-method names short. They are not typed very often, and when the number of these "protocols" increases, you face potentially ambiguous names more and more often (there already is a '__path__' and a '__file__' etc., as has been brought up earlier in these threads.). In other words, it's a good idea to have some information in the name. > I also think the more explicit name helps answer some of the type > signature questions that have arisen: > > 1. Does os.fspathname return rich Path objects? No, it returns names > as str objects Or byte strings, it seems, unfortunately. > 2. Will file descriptors pass through os.fspathname? No, as they're > not names, they're numeric descriptors. > 3. Will bytes-like objects pass through os.fspathname? No, as they're > not names, they're encodings of names > If fspathname(...) is to be used in os.path.*, it will break things if it starts to turn encoded bytes pathnames into str pathnames, which it did not previously do. And if fspathname is not to be used in os.path.*, who would be our intended user of fspathname? I assume we we don't want to encourage typical 'users' to manipulate pathnames by hand. >> I personally still like __ospath__ as well. > > That one fails the "Is it ambiguous when spoken aloud?" test for me: > if someone mentions "oh-ess-path", are they talking about os.path or > __ospath__? With "eff-ess-path" or "eff-ess-path-name", that problem > doesn't arise. > +1 to this too. -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Mon, Apr 11, 2016 at 9:27 AM, Nick Coghlan wrote:
> On 11 April 2016 at 02:16, Ethan Furman wrote:
>>
>> I guess I don't see the point of this. Either DirEntry's [1] only get
>> partial support (which is only marginally better than the no support pathlib
>> currently has), or stdlib code will need to catch those errors and then do
>> an isinstance check to see if knows what the type is and how to deal with it
>> [1].
>
> What's wrong with only gaining partial support? Standard library code
> that doesn't currently support DirEntry at all will gain the ability
> to support str-based DirEntry objects, while bytes-based DirEntry
> objects will continue to be a low level object that isn't
> interoperable with most other APIs (which is fine - anyone writing low
> level POSIX-specific code can deal with unpacking the values
> explicitly, it just won't happen implicitly anywhere).
>
While I'm also tempted to lean towards 'marginalizing bytes support',
it seems a little bit dangerous to me. Currently, os.path is heavily
based on duck typing of str and bytes, so there may be code out there
that does all kinds of things with paths without knowing whether it
deals with bytes or str objects. If such code gets in contact with
this pathname protocol, it will raise an exception whenever it happens
to be fed a bytes path. That is, if the approach of 'partial support'
is taken.
And still there is the question I just posted in another branch of
this mess: Who should use os.fspathname(...)? If it's os.path.* and
other traditional (low-level?) functions that deal with paths, then
fspathname should, in the name of backwards compatiblity, be able to
deal with bytes and return bytes in those cases. Otherwise fspathname
would do nothing for you, and all the work of
isinstance/hasattr/whatever would be left to the caller of
os.fspathname (or maybe this is what you want?).
So a somewhat useful fspathname might indeed look something like this:
def fspathname(pathlike) -> Union[str, bytes]:
pathname = getattr(pathlike, '__fspathname__', pathlike)
if not isinstance(pathname, (str, bytes)):
raise TypeError("your thing is not pathlike")
return pathname
But maybe it is enough to have the __fspathname__ attribute, and make
fspathname() some internal implementation detail of os.path.* and the
like.
-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib - current status of discussions
On Tue, Apr 12, 2016 at 11:56 AM, Nick Coghlan wrote:
> One possible way to address this concern would be to have the
> underlying protocol be bytes/str (since boundary code frequently needs
> to handle the paths-are-bytes assumption in POSIX), but offer an
> "os.fspathname" API that rejected bytes output from os.fspath. That
> is, it would be equivalent to:
>
> def fspathname(path):
> name = os.fspath(path)
> if not isinstance(name, str):
> raise TypeError("Expected str for pathname, not
> {}".format(type(name)))
> return name
>
> That way folks that wanted the clean "must be str" signature could use
> os.fspathname, while those that wanted to accept either could use the
> lower level os.fspath.
I'm not necessarily opposed to this. I kept bringing up bytes in the
discussion because os.path.* etc. and DirEntry support bytes and will
need to keep doing so for backwards compatibility. I have no
intention to use bytes pathnames myself. But it may break existing
code if functions, for instance, began to decode bytes paths to str if
they did not previously do so (or to reject them). It is indeed a lot
safer to make new code not support bytes paths than to change the
behavior of old code.
But then again, do we really recommend new code to use os.fspath (or
os.fspathname)? Should they not be using either pathlib or os.path.*
etc. so they don't have to care? I'm sure Ethan and his library (or
some other path library) will manage without the function in the
stdlib, as long as the dunder attribute is there.
So I'm, once again, posing this question (that I don't think got any
reactions previously): Is there a significant audience for this new
function, or is it enough to keep it a private function for the stdlib
to use? That handful of third-party path libraries can decide for
themselves if they want to (a) reject bytes or (b) implicitly fsdecode
them or (c) pass them through just like str, depending on whatever
their case requires in terms of backwards compatiblity or other goals.
If we forget about the os.fswhatever function, we only have to decide
whether the magic dunder attribute can be str or bytes or just str.
-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib - current status of discussions
On Tue, Apr 12, 2016 at 7:19 PM, Chris Barker wrote: > > One more though came up just now: there are different level sof abstractions > and representations for paths. We don't want to make Path a subclass of > string, because Path is supposed to be a higher level abstraction -- good. > > then at the bottom of the stack, we NEED the bytes level path, because that > what ultimately gets passed to the OS. > > THe legacy from the single-byte encoding days is that bytes and strings were > the same, so we could let people work with nice human readable strings, > while also working with byte paths in the same way -- but those days are > gone -- py3 make s clear (and important) distiction between nice human > readable strings and the bytes that represent them. > > So: why use strings as the lingua franca of paths? i.e. the basis of the > path protocol. maybe we should support only two path representations: > > 1) A "proper" path object -- i.e. pathlib.Path or anything else that > supports the path protocol. > > 2) the bytes that the OS actually needs. > You do have a point there. But since bytes pathnames are deprecated on windows, this seems to lead to supporting both str and bytes in the protocol, or having two protocols __fspathbytes__ and __fspathstr__ (and one being preferred over the other, potentially even depending on the platform)., -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib - current status of discussions
On Tue, Apr 12, 2016 at 6:52 PM, Stephen J. Turnbull wrote: > > (A) Why does anybody need bytes out of a pathlib.Path (or other > __fspath__-toting, higher-level API) *inside* the boundary? Note > that the APIs in os (etc) *don't need* bytes because they are > already polymorphic. > Indeed not from pathlib.*Path , but from DirEntry, which may have a path as bytes. So the options for DirEntry (or things like Ethan's 'antipathy') are: (1) Provide bytes or str via the protocol, depending on which type this DirEntry has Downside: The protocol needs to support str and bytes. (2) Decode bytes using os.fsdecode and provide a str via the protocol Downside: The user passed in bytes and maybe had a reason to do so. This might lead to a weird mixture of str and bytes in the same code. (3) Do not implement the protocol when dealing with bytes Downside: If a function calling os.scandir accepts both bytes and str in a duck-typing fashion, then, if this adopted something that uses the new protocol, it will lose its bytes compatiblity. This risk might not be huge, so perhaps (3) is an option? > (B) If they do, why can't they just apply bytes() to the object? I > understand that that would offend Ethan's aesthetic sense, so it's > worth looking for a nice way around it. But allowing __fspath__ > to return bytes or str is hideous, because Paths are clearly on > the application side of the boundary. > > Note that bytes() may not have the serious problem that str() does of > being too catholic about its argument: nothing in __builtins__ has a > __bytes__! Of course there are a few things that do work: ints, and > sequences of ints. Good point. But this only applies to when the user _explicitly_ deals with bytes. But when the user just deals with the type (str or bytes) that is passed in, as os.path.* as well as DirEntry now do, this does not work. -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] List posting custom [was: current status of discussions]
On Wed, Apr 13, 2016 at 5:56 AM, Stephen J. Turnbull wrote: > The following is my opinion, as will become obvious, but it's based on > over a decade of observing these lists, and other open source > development lists. In a context where some core developers have > unsubscribed from these lists, and others regularly report muting > threads with a certain air of asperity, I think it's worth the risk of > seeming arrogant to explain some of the customs (which are complex and > subtle) around posting to Python developer lists. I'm posting > publicly because there are several new developers whose activity and > fresh perspective is very welcome, but harmony *is* being disturbed, > IMO unnecessarily. > Thank you for this thoughtful post. While none of the quotes you refer to are mine, I did try to find whether any of the advice is something I should learn from. While I didn't find a whole lot (please do correct me if you think otherwise), it is also valuable to hear these things from someone more experienced, even just to confirm what I may have thought or guessed. I can't really tell, but possibly some of the thoughts are interesting even to people significantly more experienced than me. I know you are not interested in discussing this further here, but I'll add some inexperienced points of view inline below, just in case someone is interested: > This particular post caught my eye, but it's only an example of one of > the most unharmonious posting styles that has become common recently. > Attribution deliberately removed. > > > Sorry for disturbing this thread's harmony. > > *sigh* There is way too much of this on Python-Ideas recently, and > there shouldn't be any on Python-Dev. So please don't. Specifically, > disagreement with an apparently developing consensus is fine but > please avoid this: > > > >> Path is an alternative to os.path -- you don't need to use both. > > > > I agree with that quote of Chris. > > It's a waste of time to post *what* you agree with.[1] Decisions are > not taken by vote in this community, except for the color of the > bikeshed, where it is agreed that *what* decision is taken doesn't > matter, but that some decision should be taken expeditiously.[2] > Chris already stated this position clearly and it's not a "color", so > there is no need to reiterate. It simply wastes others' time to read > it. (Whether it was a waste of the poster's time is not for me to > comment on.) > > What matters to the decision is *why* you agree (or disagree). If you > think that some of Chris's arguments are bogus (and should be > disregarded) and others are important, that is valuable information. > It's even better if you can shed additional light on the matter > (example below). > > Also, expression of agreement is often a prelude to a request for > information. "I agree with Z's post. At least, I have never needed > X. *When* do you need X? Let's look for a better way than X!" > That's what I thought too. I remember several times recently that I have mentioned I agreed about something, then continuing to add more to it, or even saying I disagree about something else. Part of the reason to also state that I agree is an attempt to keep the overall tone more positive. After all, the other person might be a highly experienced core developer who just did not happen to have gone though all the same thoughts regarding that specific question recently. I hope that has not been interpreted as arrogance such as "I know better than these people". For me, as one of the (many?) newcomers, especially on -dev, it can sometimes be difficult to tell whether not getting a reaction means "Good point, I agree", "I did not understand so I'll just ignore it", "I don't want to argue with you" or something else. Then again, someone just saying essentially the same thing without a reference a few posts later just feels strange. Also, if the only thing people apparently do is disagree about things, it makes the overall tone of the discussions at least *seem* very negative. From this point of view there seems to be some good in positive comments. > Unsupported (dis)agreement to statements about "needs" also may be > taken as *rude*, because others may infer your arrogant claim to know > what *they* do or don't need. Admittedly there's a difficult > distinction here between Chris's *idiom* where "you don't need to" > translates to "In my understanding, it is generally not necessary to", > and your *unsupported* agreement, which in my dialect of English > changes the emphasis to imply you know better than those who disagree > with you and Chris. And, of course, the position that others are "too > easily offended" is often reasonable, but you should be aware that > there will be an impact on your reputation and ability to influence > development of Python (even if it doesn't come near the point where > a moderator invokes "Code of Conduct"). > > "Me too" posts aren't entirely forbidden, but I feel that
Re: [Python-Dev] pathlib - current status of discussions
On Thu, Apr 14, 2016 at 7:46 PM, Ethan Furman wrote: > > What many folks seem to be missing is that *you* (generic you) have control > of your data. > > If you are not working at the bytes layer, you shouldn't be getting bytes > objects because: > > - you specified str when asking for data from the OS, or > - you transformed the incoming bytes from whatever external source > to str when you received them. There is an apparent contradiction of the above with some previous posts, including your own. Let me try to fix it: Code that deals with paths can be divided in groups as follows: (1) Code that has access to pathname/filename data and has some level of control over what data type comes in. This code may for instance choose to deal with either bytes or str (2) Code that takes the path or file name that it happens to get and does something with it. This type of code can be divided into subgroups as follows: (2a) Code that accepts only one type of paths (e.g. str, bytes or pathlib) and fails if it gets something else. (2b) Code that wants to support different types of paths such as str, bytes or pathlib objects. This includes os.path.*, os.scandir, and various other standard library code. Presumably there is also third-party code that does the same. These functions may want to preserve the str-ness or bytes-ness of the paths in case they return paths, as the stdlib now does. But new code may even want to return pathlib objects when they get such objects as inputs. This is the duck-typing or polymorphic code we have been talking about. Code of this type (2b) may want to avoid implicit conversions because it makes the life of code of the other types more difficult. (feel free to fill in more categories of code) So the code of type (2b) is trying to make all categories happy by returning objects of the same type that it gets as input, while the other categories are probably in the situation where they don't necessarily need to make other categories of code happy. And the question is this: Do we need to make code using both bytes *and* scandir happy? This is largely the same question as whether we have to support bytes in addition to str in the protocol. (We may of course talk about third-party path libraries that have the same problem as scandir's DirEntry. Ethan's library is not exactly in the same category as DirEntry since its path objects *are* instances of bytes or str and therefore do not need this protocol to begin with, except perhaps for conversions from other high-level path types so that different path libraries work together nicely). -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pathlib - current status of discussions
On Thu, Apr 14, 2016 at 9:35 PM, Random832 wrote: > On Thu, Apr 14, 2016, at 13:56, Koos Zevenhoven wrote: >> (1) Code that has access to pathname/filename data and has some level >> of control over what data type comes in. This code may for instance >> choose to deal with either bytes or str >> >> (2) Code that takes the path or file name that it happens to get and >> does something with it. This type of code can be divided into >> subgroups as follows: >> >> (2a) Code that accepts only one type of paths (e.g. str, bytes or >> pathlib) and fails if it gets something else. > > Ideally, these should go away. > I don't think so. (1) might even be the most common type of all code. This is code that gets a path from user input, from a config file, from a database etc. and then does things with it, typically including passing it to type (2) code and potentially getting a path back from there too. >> (2b) Code that wants to support different types of paths such as >> str, bytes or pathlib objects. This includes os.path.*, os.scandir, >> and various other standard library code. Presumably there is also >> third-party code that does the same. These functions may want to >> preserve the str-ness or bytes-ness of the paths in case they return >> paths, as the stdlib now does. But new code may even want to return >> pathlib objects when they get such objects as inputs. > > Hold on. None of the discussion I've seen has included any way to > specify how to construct a new object representing a different path > other than the ones passed in. Surely you're not suggesting type(a)(b). > That's right. This protocol is not solving the issue of returning 'rich' path objects. It's solving the issue of passing those objects to lower-level functions or to interact with other 'rich' path types. What I meant by this is that there may be code that *does* want to do type(a)(b), which is out of our control. Maybe I should not have mentioned that. > Also, how does DirEntry fit in with any of this? > os.scandir + DirEntry are one of the many things in the stdlib that give you pathnames of the same type as those that were put in. >> This is the >> duck-typing or polymorphic code we have been talking about. Code of >> this type (2b) may want to avoid implicit conversions because it makes >> the life of code of the other types more difficult. > > As long as the type it returns is still a path/bytes/str (and therefore > can be accepted when the caller passes it somewhere else) what's the > problem? No, because not all paths are passed to the function that does the implicit conversion, and then when for instance os.path.joining two paths of a differenty type, it raises an error. In other words: Most non-library code (even library code?) deals with one specific type and does not want implicit conversions to other types. Some code (2b) deals with several types and, at least in the stdlib, such code returns paths of the same type as they are given, which makes said "most non-library code" happy, because it does not force the programmer to think about type conversions. (Then there is also code that explicitly deals with type conversions, such as os.fsencode and os.fsdecode.) -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Sun, Apr 17, 2016 at 11:03 AM, Stephen J. Turnbull
wrote:
> Nick Coghlan writes:
>
> > str and bytes aren't going to implement __fspath__ (since they're
> > only *sometimes* path objects), so asking people to call the
> > protocol method directly for any purpose would be a pain.
>
> It *should* be a pain. People who need bytes should call fsencode,
> people who need str should call fsdecode, and Ethan's antipathy checks
> for bytes and str, then calls __fspath__ if needed. Who's left? Just
> the bartender and the janitor, last call was hours ago. OK, maybe
> there are enough clients to make it worthwhile to provide the utility,
> but it should be clearly marked as "double opt-in, for experts only
> (consenting adults must show proof of insurance)".
My doubts, expressed several times in these threads, about the need
for a *public* os.fspath function to complement the __fspath__
protocol, are now perhaps gone. I'll explain why (and how). The
reasons for my doubts were that
(1) The audience outside the stdlib for such a function should be
small, because it is preferred to either use existing tools in
os.path.* or pathlib (or similar) for manipulating paths.
(2) There are just too many different possible versions of this
function: rejecting str, rejecting bytes, coercion to str, coercion to
bytes, and accepting both str and bytes. That's a total of 5 different
cases. People also used to talk about versions that would not allow
passing through objects that are already bytes or str. That would make
it a total of 10 different versions!
(in principle, there could be even more, but let's not go there :-).
In other words, this argument was that it is probably best to
implement whatever flavor is needed for the context, perhaps based on
documented recipes.
Regarding (2), we can first rule out half of the 10 cases---the ones
that reject plain instances of bytes and/or str---because they would
not be very useful as all the isinstance/hasattr checking etc. would
be left to the caller. And here are the remaining five, explained
based on what they accept as argument, what they return, and where
they would be used:
(A) "polymorphic"
*Accept*: str and bytes, provided via __fspath__ as well as plain str
and bytes instances.
*Return*: str/bytes depending on input.
*Audience*: the stdlib, including os.path.things, os.things,
shutil.things, open, ... (some functions would need a C version).
There may even be a small audience outside the stdlib.
(B) "str-based only"
*Accept*: str, provided via __fspath__ as well as plain str.
*Return*: str.
*Audience*: relatively low-level code that works exclusively with str
paths but accepts specialized path objects as input.
(C) "bytes-based only"
*Accept*: bytes, provided via __fspath__ as well as plain bytes.
*Return*: bytes.
*Audience*: low-level code that explicitly deals with paths as bytes
(probably to deal with undefined/ill-defined encodings).
(D) "coerce to str"
*Accept*: str and bytes, provided via __fspath__ as well as plain str
and bytes instances.
*Return*: str (coerced / decoded if needed).
*Audience*: code that deals explicitly with str but wants to 'try'
supporting bytes-based path inputs too via implicit decoding (even if
it may result in surrogate escapes, which one cannot for instance
print(...).)
(E) "coerce to bytes"
*Accept*: str and bytes, provided via __fspath__ as well as plain str
and bytes instances.
*Return*: bytes (coerced / encoded if needed).
*Audience*: low-level code that explicitly deals with bytes paths but
wants to accept str-based path inputs too via implicit encoding.
Even if all options (A-E) probably have small audiences (compared to
e.g. os.path.*), some of them have larger audiences than others. But
all of them have at least *some* reasonable audience (as desribed
above).
Recently (well, a few days ago, but 'recently', considering the scale
of these discussions anyway ;-), Nick pointed out something I hadn't
realized---os.fsencode and os.fsdecode actually already implement
coercion to bytes and str, respectively. With those two functions made
compatible with the __fspath__ protocol [using (A) above], they would
in fact *be* (D) and (E), respectively.
Now, we only have options (A-C) left. They could all be implemented
roughly as follows:
def fspath(pathlike, *, output_types = (str,)):
if hasattr(pathlike, '__fspath__'):
ret = pathlike.__fspath__() # or pathlike.__fspath__ if it's not a method
else:
ret = pathlike
if not isinstance(ret, output_types):
raise TypeError("argument is not and does not provide an
acceptable pathname")
return ret
With an implementation like the above, (A) would correspond to
output_types = (str, bytes), (B) to the default, and (C) to
output_types = (bytes,).
So, with the above considerations as a counterargument, I consider
argument (2) gone.
What about argument (1), that the audience for the os.fspath(...)
function (especially for one selected version of the 5 or 10
variation
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Sun, Apr 17, 2016 at 9:14 PM, Ethan Furman wrote: > On 04/17/2016 06:58 AM, Koos Zevenhoven wrote: > >> So, as a summary: With a str+bytes-polymorphic __fspath__, with the >> above argumentation and the rough implementation of os.fspath(...), >> the conclusion is that the os.fspath function should indeed be public, >> and that no further variations are needed. > > > Nice summation, thank you. :) > Come on, Ethan, that summary was not for you ;) It was for lazy people, people with bad memory, or people not so involved in the topic. I wrote a big post, provided new arguments, with other points collected into the same logical framework, wrote a new version of os.fspath and argued why it is the right one --- and all you do is read the stupid summary. You can do better than that: read the whole thing! ;-). -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Mon, Apr 18, 2016 at 5:03 PM, Ethan Furman wrote:
> On 04/18/2016 12:41 AM, Nick Coghlan wrote:
>
>> Given the variant you [Koos] suggested, what if we defined the API
>> semantics
>> like this:
>>
>> # Offer the simplest possible API as the public vesion
>> def fspath(pathlike) -> str:
>> return os._raw_fspath(pathlike)
>>
>> # Expose the complexity in the "private" variant
>> def _raw_fspath(pathlike, *, output_types = (str,)) -> (str, bytes):
>> # Short-circuit for instances of the output type
>> if isinstance(pathlike, output_types):
>> return pathlike
>> # We'd have a tidier error message here for non-path objects
>> result = pathlike.__fspath__()
>> if not isinstance(result, output_types):
>> raise TypeError("argument is not and does not provide an
>> acceptable pathname")
>> return result
>
> My initial reaction was that this was overly complex, but after thinking
> about it a couple days I /really/ like it. It has a reasonable default for
> the 99% real-world use-case, while still allowing for custom and exact
> tailoring (for the 99% stdlib use-case ;) .
>
While it does seem we finally might be nearly there :), this still
seems to need some further discussion.
As described in that long post of mine, I suppose some third-party
code may need the variations (A-C), while it seems that in the stdlib,
most places need (str, bytes), i.e. (A), except in pathlib, which
needs (str,), i.e. (B). I'm not sure what I think about making the
variations private, even if "hiding" the bytes version is, as I said,
an important role of the public function.
Except for that type hint, there is *nothing* in the function that
might mislead the user to think bytes paths are something important in
Python 3. It's a matter of documentation whether it "supports" bytes
or not. In fact, that function (assuming the name os.fspath) could now
even be documented to support this:
patharg = os.fspath(patharg, output_types = (str, pathlib.PurePath)) # :-)
So are we still going to end up with two functions or can we deal with one?
What should the typehint be? Something new in typing.py? How about
FSPath[...] as follows:
FSPath[bytes] # bytes-based pathlike, including bytes
FSPath[str] # str-based pathlike, including str
pathstring = typing.TypeVar('pathstring', str, bytes) # could be
extended with PurePath or some path ABC
So the above variation might become:
def fspathname(pathlike: FSPath[pathstring],
*, output_types: tuple = (str,)) -> pathstring:
# Short-circuit for instances of the output type
if isinstance(pathlike, output_types):
return pathlike
# We'd have a tidier error message here for non-path objects
result = pathlike.__fspath__()
if not isinstance(result, output_types):
raise TypeError("valid output type not provided via __fspath__")
return result
And similar type hints would apply to os.path functions. For instance,
os.path.dirname:
def dirname(p: FSPath[pathstring]) -> pathstring:
...
This would say pathstring all over and not give anyone any ideas about
bytes, unless they know what they're doing.
Complicated? Yes, typing is. But I think we will need this kind of
hints for os.path functions anyway.
-Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Tue, Apr 19, 2016 at 2:55 PM, Stephen J. Turnbull wrote: > > AFAICS bytes return from __fspath__ is just YAGNI. Show me something > that actually wants it. It might be, but as long as bytes paths are supported polymorphicly all over the stdlib, we won't get rid of supporting bytes paths. So are you proposing to deprecate bytes paths? -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Wed, Apr 20, 2016 at 6:11 AM, Stephen J. Turnbull wrote: > Koos Zevenhoven writes: > > On Tue, Apr 19, 2016 at 2:55 PM, Stephen J. Turnbull > wrote: > > > > > > AFAICS bytes return from __fspath__ is just YAGNI. Show me something > > > that actually wants it. > > > > It might be, > > May I take that as meaning you just jumped to the conclusion that > extending polymorphism is useful on no actual evidence of usefulness? No you may not! YAGNI almost never means "you are *never* going to need it". And if you implement a feature, better implement it well. If a variation of the feature is rarely used, that is perfectly fine. I think leaving bytes out would complicate things. If os.fspath does its job well, everyone should be happy. I kept bringing up bytes paths, because that is already a feature in Python 3. Then (already some time ago in these discussions) I briefly visited the thought of 'can we deprecate bytes paths', and it then quickly became clear to me that is not going to happen any time soon. In other words: As long as bytes paths are supported, they should be supported consistently. I don't want DirEntry to behave differently when the underlying type is bytes, which is one of the things I've been talking about all the time. That would just be broken. And as you also understand, one point is to allow passing DirEntry to open. Or any of the os.path functions. An some more: I don't want open(direntry_obj) to ever raise because it is the bytes flavor of direntry, because, when they are created, DirEntry objects always point to existing objects on the file system. I also don't want implicit conversions between str and bytes paths, because there are cases where they will produce strange results and exceptions. [Yes, way back in the p-string thread, I did first suggest a similiar thing that implied implicit conversion, but I soon abandoned that part.] Not that I will ever use these features---just to do this right. > > but as long as bytes paths are supported polymorphicly all over the > > stdlib, we won't get rid of supporting bytes paths. So are you > > proposing to deprecate bytes paths? > > You claim "almost always want str", Ethan claims "bias against bytes." > Sorry, guys, you can't have it both ways. Either bytes paths are > discouraged (not "deprecated", not yet), or they aren't. > > I say, let's not encourage them. It's all essentially the same thing: "almost always want str": Yes, I still claim this. This is the reason for str (and rejecting bytes) being the default for third-party code. If we wanted to, we could even leave bytes support out of the documentation, so no-one will know about it unless they already deal with bytes paths. However, I dont think we should do that---we should just strongly discourage using the bytes version unless there is a reason to, and you know what you are doing. "bias against bytes": I agree with this too. This is in line with making str (and rejecting bytes) the default for third-party code. "let's not encourage them": And I even agree with this, as you may have noticed. I just don't believe in deliberately making implementations awkward for the bytes-based paths. Bytes paths already exist, not because of Python 2 (as you know), but because not all operating systems guarantee that paths make sense in any encoding, and people may need to work at that level. There is no need to make working with bytes-based paths awkward, and we can support them with little additional work compared to supporting str-based rich path objects. The additional work is mostly this discussion. > Ie, keep the status quo for bytes, > and make things better for the preferred str. Yes, that means > discouraging bytes relative to str in this context. That's a Python 3 > principle, one strong enough to justify the huge compatibility break > involved in making str be Unicode. That compatibility break has been > extremely successful in my personal experience as a sometime Python > teacher and Mailman developer, though the Mercurial developers have a > different POV. Yes. Luckily, people are already using str-based paths. We don't need any more discrete transitions. If linux will start to enforce an encoding, as Guido and Random832 may be suggesting on python-ideas, these already obscure bytes paths will slowly fade away. -Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
On Wed, Apr 20, 2016 at 6:16 AM, Stephen J. Turnbull wrote: > > (1) some really attractive producer of pathlib.Paths will be > published, and > Yes, pathlib is str-only, so this sounds just right. > (2) people will want to plug that producer into their bytes paths > consumers using os.fspath(path) "and be done with it". > No, fspath can't know that is the the right thing to do. There should be *someone* that is aware of the encoding that happens, either the provider or the consumer. That byte path consumer, assuming it wants to support the behavior you describe, should use os.fsencode instead of os.fspath, which will do exactly what you want, and just as easy for the bytes path consumer to implement! (Unless you want to explicitly reject plain str objects, which you would then indeed do *explicitly*, but I'm not sure there is a point in accepting plain bytes and str-based pathlib objects but not str). To avoid further unnecessary discussion, please read [1] carefully, where I already explained this, among other things. -Koos [1] https://mail.python.org/pipermail/python-dev/2016-April/144239.html ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pathlib enhancements - improve fsdecode and fsencode
On Thu, Apr 14, 2016 at 9:55 AM, Stephen J. Turnbull wrote: > Please please please, junk both "filter out bytes" proposals. If you were referring to some of the fspath versions, I think we will need a bytes-rejecting version, for reasons explained in [1-2]. Of course not eveŕyone wants or has to use it. > Since they involve an exception, they impose an unnecessary "try" on > all text applications that fear death on bytes returns. May as well > just wrap all objects with __fspath__ in fsdecode, and all is > happy. > > Counterproposal: make fsdecode and fsencode grok __fspath__. Then: Not being a native English speaker, I'm relying on a Wikipedia explanation of "grok", but if you mean that fsdecode and fsencode would accept objects that implement __fspath__, then I think we all agree on this. Making the stdlib accept path objects, after all, is the whole point of the pathlib discussions :). Anyway, I am happy that Nick [3] (and you [4] ?) pointed out that os.fsencode and os.fsdecode currently implement coercion, i.e., they both accept both str and bytes, and return just one of them. This was important for my conclusion in [1]. When these two functions are made __fspath__ compatible using `fspath(patharg, output_types = (str, bytes))`, like most os functions, they will indeed implement coercion to bytes or str from "any pathlike object". [Side note: One may, for instance, ask why os.fsdecode passes str objects through silently, even if they can't be decoded. Well, that's the way it is, and I'm not expecting that to change. But maybe fsdecode should have an additional keyword-only argument to tell them that it should strictly return something it actually did decode. (And similarly for os.fsencode.) But this has nothing to do with the path protocol we are discussing.] > (1) Bytes-lovers and str-addicts are both safe. I don't think everyone is safe if you cant say "I don't want implicit encoding/decoding". > (2) They can omit fspath, too! I think having *one* additonal function for the non-encoding/non-decoding cases is too much, and as shown in [1], one is enough. > No, that doesn't work if the bytes objects aren't in the file system > encoding, but these are *bytes*, mon ami: you have no way to find out > what that encoding is, so you either know already and you substitute > that + fspath for fsdecode, or you're hosed. And in the only concrete > use case so far, fsdecode Just Works. Well, as you say yourself, fsdecode indeed works if your bytes are in the default fs encoding, and when you know they are, go for it, use fsdecode. But I, for instance, rarely have my paths as bytes. Therefore, I would be happy to get an exception if I'm accidentally passing bytes to some non-bytes-supporting function because I've forgotten to decode some input that I got in an encoding other than the file system encoding. > I suppose a similar argument holds for applications that want bytes > and fsencode, but I leave that as an exercise for the reader. A similar counterargument holds, too :). Unrelated to this particular post, I believe these discussions are almost done and I truly hope we at least won't have to keep addressing the same questions that we have already gone through, unless there is something new on the table. I hope it takes a shorter time to read these emails than it takes to write them :). -Koos [1] https://mail.python.org/pipermail/python-dev/2016-April/144239.html [2] https://mail.python.org/pipermail/python-dev/2016-April/144290.html And somewhat older ones: [3] https://mail.python.org/pipermail/python-dev/2016-April/144101.html [4] https://mail.python.org/pipermail/python-dev/2016-April/144107.html ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Discussion on fspath: please wait for a PEP
On Wed, Apr 20, 2016 at 2:52 PM, Victor Stinner wrote: > Hi, > > I'm unable to count the number of threads about the fspath protocol. > It's even more difficult to count the total number of emails. IMHO > everyone had enough time to give him/her opinion. Couldn't agree more. > We even had multiple > summaries :-) I'm not quite as sure about this. Maybe the meaning of "summary" in the subculture of python lists is different from the one I know. > Can you please wait for a PEP? Brett Canon and Ethan Furman are > working on a PEP. So please give them time to write it. I wonder what happened there... > The PEP should summarize the discussion and help a lot to make > concrete progress on the design (avoid restarting to discuss the same > points forever). I don't expect that more emails would add anything at > the current state of the discussion. Again, agreed, and this part makes me feel relieved. Personally, I got tired of the discussion a long time ago, but felt it had to be finished. > I think that we have enough other topics to discuss in the meanwhile ;-) No doubt about that. > FYI there is already an article about fspath/pathlib on LWN. Here is a > free link until the article is freely accessible: > > "Python looks at paths" By Jake Edge (April 13, 2016) > https://lwn.net/SubscriberLink/683350/4f52334af09653c8/ Wow. Wasn't expecting that. A whole story about the notorious "path discussions"! (well, up to some date). Anyway, the beginning seems fairly accurate, but then, among other things, it fails to mention this for example: https://mail.python.org/pipermail/python-ideas/2016-March/039179.html https://mail.python.org/pipermail/python-ideas/2016-April/039595.html Since I did not get any responses to that suggestion, it felt like a dead end, and I continued experimenting with other things and ended up taking the approach of "subclassing path-types from str gives more complete pathlib support, but the objects should not pretend to be strings in every way". By the way, I even implemented this, which I suppose I failed to mention. Admittedly, it became a little awkward in the end, but the main point was to provide a smooth transition from a str world to a PurePath-subclass world (as opposed to a discrete one like Py3k). While I was working on that, the discussions on -dev seemed to have reopened the gate at exactly that 'dead end' I mentioned before, and had started to step through it. -Koos > > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Discussion on fspath: please wait for a PEP
On Wed, Apr 20, 2016 at 7:34 PM, Victor Stinner wrote: > > 2016-04-20 18:12 GMT+02:00 Brett Cannon : >> >> I thought Chris and I w/ Ethan helping with coding, but if it's just me for >> the PEP then that's fine; Well, just in case you didn't notice this on python-ideas, I offered to work on the PEP in case there turns out to be one. This was when Guido had asked if there is going to be a PEP, in response to my "Type hinting for path-related functions" email. That offer is certainly still valid. >> luckily my firefighter gear is well-worn: >> https://goo.gl/photos/R8oWdLE45d99ebaw8 > > LOL, it seems appropriate for this topic... It sure has been flammable XD >> I'll try to get a PEP draft written and posted prior to PyCon US. I will >> reply to any dangling comments/issues that have appeared overnight to close >> those threads, but otherwise I will start ignoring all discussions so I can >> focus on the PEP. Everyone can now consider themselves spared from any >> further path-related discussions. :) Yes, going from endless discussion to PEP seems like a very healthy direction at this point. > I hesitated to propose to create a fspath-sig mailing list, but I suck > at humor and so I skipped this joke in my email ;-) You did not have to tell that joke, the joke was present all the time ;). > Victor > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
**another deep, calming breath**
On Wed, May 11, 2016 at 7:43 PM, Brett Cannon wrote:
> Open Issues
> ===
>
> Should os.fspath() return bytes?
>
>
In most cases, it of course should not. The section (or the title) do
not represent my view on the topic, but bytes paths are not going away
any time soon, so this requires considerations.
Below is a copy paste of my comment in the discussion on my pull
request ( https://github.com/brettcannon/path-pep/pull/2 ). It is
about whether os.fspath should look like
def fspath(path, *, type_constraint=str): ...
This was already discussed before, receiving positive reactions. It
would be an extension to what is in the draft Brett just posted. Note
that the return type, however, does not depend on `type_constraint`.
It fully depends on `path`. The constraint only determines when an
exception is raised instead of returning a value. When using
str-based paths normally, one would do
os.fspath(path)
Calling it with
os.fspath(path, type_constraint=(str, bytes))
would turn off rejection of bytes paths (and be consistent with the
rest of `os` and `os.path` where functions accept both types). But the
default value for the keyword argument would be a good reminder that
str-based paths should be used by default. This str-constraint is also
present in the current drafted version, but it is not optional or
visible in the signature.
(See the diffs of my pull request for an example implementation of this.)
So below is the copy-paste from the pull request discussion that I promised:
"""
Nobody wants to promote using `bytes` for paths without a proper
reason. However, `os.fspath(path, type_contraint=str)` is already a
compromise from the current `os.*` convention (of symmetrically
supporting both `str` and `bytes`) towards enforcing `str`-based
paths.
As you know, the reason for using `os.fspath` is to switch to a
lower-level representation of paths as strings. When you use it, you
are already deciding that you are lower-level and want to examine or
manipulate path strings manually. I think the `type_constraint=str`
keyword-only argument (with the default) is a good way to remind the
'user' that `str` is the way to go unless you know what you are doing
and, to `bytes` users, that `os.fspath` (by default) rejects bytes.
Looking at the discussions on python-dev, one notices that the active
people in the discussions were mostly in favor of exposing the
bytes-supporting version in one way or the other. Some were even
against bytes-rejecting versions.
Most people (Python programmers), especially in the long term, should
not be using `os.fspath` a lot. If they want to print a path, they
should simply do so: `print(path_obj)`, or `print(f"The path is:
{path_obj}")`. However, there will always be people that for whatever
reason want to convert a pathlib object into a string, and without
even consulting the docs, they may write `str(path_obj)`. And that's
not the end of the world. It's not any worse than `str(integer)` for
instance. After all, if the main point is to convert something into a
generic string, the function should probably be called `str`, not
`fspath`. But if the idea is to convert any path object into something
that `os` and similar code internally understand, then `os.fspath`
sounds fine to me. So I would not go as far as 'never use
`str(path)`'. What we don't want is things like `open(str(path),
'w')`. I know you agree, because you wrote that yourself. But once
this protocol is available, I see no reason why people would
voluntarily wrap their path objects in `str(...)`, so I don't see that
as a problem in the future (thanks to this PEP). So, especially in the
long term, I don't expect `fspath` to be an everyday tool for people.
Even in shorter term, people don't usually need it, because the stdlib
will already support pathlib objects.
Since it is not an everyday tool, I think it's ok to have that extra
keyword-only argument. Even if it were an everyday tool, I can see the
keyword-only argument in the signature as useful thing in a
documentation sense.
[As a side note, I could imagine someone convincing me that `fspath`
should *always* accept both `str`- and `bytes`-based paths, because
that would simplify it and align it with the rest of `os`. This would
not be any worse than the status quo in terms of early failures. That
would, however, still not detect accidental use of bytes paths (when
forgotten to decode from the encoding it was received in).]
"""
-- Koos
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Wed, May 11, 2016 at 11:04 PM, Brett Cannon wrote: > A quick comment about sending me fixes. While I do appreciate them, sending > them as a pull request is much easier for me as (a) I don't have to hunt the > changes down in the text, and (b) you will see the fixes others have done > already to the PEP and I then don't have to figure out what changes have not > already been fixed. And honestly, reading the PEP in its rendered format on > GitHub is easier IMO than in the text format unless you have something > specific to respond to (and even if you do, you can copy and paste the > relevant bits into an email reply). > Personally, I find it more important to settle the open issues first, so I will be looking for typos at a later stage. Besides, more typos may be added in the process. I previously left some typos uncorrected in attempt to keep the diffs of my commits cleaner. By the way, to see what additions and wordings I have suggested, the right way is to look at my PR(s). I might send more PRs to clarify open issues etc., but this will largely be based on my previous commits. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 12:15 AM, Ethan Furman wrote: > On 05/11/2016 01:51 PM, Ethan Furman wrote: >> >> On 05/11/2016 01:44 PM, Serhiy Storchaka wrote: > > os.path ''' The various path-manipulation functions of ``os.path`` [#os-path]_ will be updated to accept path objects. For polymorphic functions that accept both bytes and strings, they will be updated to simply use code very much similar to ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This will allow for their pre-existing type-checking code to continue to function. >>> >>> >>> I afraid that this will hit a performance. Some os.path functions are >>> used in tight loops, they are hard optimized, and adding support of path >>> protocol can have visible negative effect. >> >> >> Do you have an example of os.path functions being used in a tight loop? > I'd be interested in this too. > > Also, the C code for fspath can check types first and take the fast path if > bytes/str are passed in, only falling back to the __fspath__ protocol if > something else was passed in -- which should make any performance hits > negligible. My suggestion for the *python version* already does this too (again, see my PR). This is a win-win, as it also improves error messages (as suggested by Nick in the earlier discussions). -- Koos > > > -- > ~Ethan~ > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 12:28 AM, Nikolaus Rath wrote: > On May 11 2016, Brett Cannon wrote: >> This PEP proposes a protocol for classes which represent a file system >> path to be able to provide a ``str`` or ``bytes`` representation. > [...] > > As I said before, to me this seems like a lot of effort for a very > specific use-case. So let me put forward two hypothetical scenarios to > better understand your position: I think you are touching important points. > - A new module for URL handling is added to the standard library (or > urllib is suitably extended). There is a proposal to add a new > protocol that allows classes to provide a ``str`` or ``bytes`` > representation of URLs. This reminds me of the thread I recently started on python-ideas about extending the concept of paths to URLs. I don't know if you are referring to something like that or not. Anyway, it would be important to know whether the str or bytes representation is to be used a file system path or an URL, so that would need to be a separate protocol. But everyone would have the experience from these discussions, so hopefully less discussion then :). (By the way, this is one reason to have 'fs' in the name of the __fspath__ method, although I have not mentioned it before.) > > - A new (third-party) library for natural language processing arises > that exposes a specific class for representing audio data. Existing > language processing code just uses bytes objects. To ease transition > and interoperability, it is proposed to add a new protocol for classes > that represend audio data to provide a bytes representation. > > Do you think you would you be in favor of adding these protocols to > the stdlib/languange reference as well? If not, what's the crucial > difference to file system paths? > File system paths are very fundamental, and will probably be used in context of your natural language example too. -- Koos > > Thanks, > -Nikolaus ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 1:13 AM, Brett Cannon wrote: > > > On Wed, 11 May 2016 at 13:45 Serhiy Storchaka wrote: >> >> On 11.05.16 19:43, Brett Cannon wrote: >> > os.path >> > ''' >> > >> > The various path-manipulation functions of ``os.path`` [#os-path]_ >> > will be updated to accept path objects. For polymorphic functions that >> > accept both bytes and strings, they will be updated to simply use >> > code very much similar to >> > ``path.__fspath__() if hasattr(path, '__fspath__') else path``. This >> > will allow for their pre-existing type-checking code to continue to >> > function. >> >> I afraid that this will hit a performance. Some os.path functions are >> used in tight loops, they are hard optimized, and adding support of path >> protocol can have visible negative effect. > > > As others have asked, what specific examples do you have that os.path is > used in a tight loop w/o any I/O that would overwhelm the performance? > >> >> >> I suggest first implement other changes and then look whether it is >> worth to add support of path protocol in os.path functions. > > > I see this whole discussion breaking down into a few groups which changes > what gets done upfront and what might be done farther down the line: > > 1. Maximum acceptance: do whatever we can to make all representation of paths > just work, which means making all places working with a path in the stdlib > accept path objects, str, and bytes. Since you are putting me in this camp, there is at least one thing you are wrong about. I don't want all places that work with a path to accept bytes. Only those that already do so, including os/os.path. And yes, I think the stdlib should show a good example in accepting path types (especially those provided in the stdlib itself). Whether Ethan is fully in camp 1, I don't know. Not that I think he would be any closer to the other camps, though. > 2. Safely use path objects: __fspath__() is there to signal an object is a > file > system path and to get back a lower-level representation so people stop > calling str() on everything, providing some interface signaling that someone > doesn't misuse an object as a path and only changing path consumptions APIs > -- e.g. open() -- and not path manipulation APIs -- e.g. os.path -- in the > stdlib. > > 3. It ain't worth it: those that would rather just skip all of this and drop > pathlib from the stdlib. > > Ethan and Koos are in group #1 and I'm personally in group #2 but I tried to > compromise somewhat and find a middle ground in the PEP with the level of > changes in the stdlib but being more restrictive with os.fspath(). If I were > doing a pure group #2 PEP I would drop os.path changes and make os.fspath() > do what Ethan and Koos have suggested and simply pass through without checks > whatever path.__fspath__() returned if the argument wasn't str or bytes. > Related to this, based on the earlier discussions, I had the impression that you were largely in the same camp as me. In fact, I thought you had politely left some things out of the PEP draft so I could fill them in. It turned out I was wrong about that, because you didn't merge them. -- Koos > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 2:05 AM, Ethan Furman wrote: > On 05/11/2016 03:13 PM, Brett Cannon wrote: > >> If [...] I would drop os.path changes and make os.fspath() do what > >> Ethan and Koos have suggested and simply pass through without checks >> >> whatever path.__fspath__() returned if the argument wasn't str or bytes. > > > Not to derail the conversation too much, as I know we're all getting burned > out on the topic, but that last bit is not accurate: my druthers are to have > __fspath__ be able to return str /or/ bytes, and if anything else comes from > the object in question an exception must be raised. Maybe a word got lost > between your thoughts and your fingers -- happens to me all the time. :) Yes. This would also be equivalent to my fspath(path, type_constraint=(str,bytes)). And if the compromise I mentioned about the rejecting (by default or optionally) is lifted, the keyword argument would not be needed. I might be ok with throwing away the isinstance check on the return value of __fspath__() if it has significant impact on performance in realistic cases (with DirEntry most likely, I suppose), but I doubt it. -- Koos > > -- > ~Ethan~ > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven wrote: > On Thu, May 12, 2016 at 2:05 AM, Ethan Furman wrote: >> On 05/11/2016 03:13 PM, Brett Cannon wrote: >> >>> If [...] I would drop os.path changes and make os.fspath() do what >> >>> Ethan and Koos have suggested and simply pass through without checks >>> >>> whatever path.__fspath__() returned if the argument wasn't str or bytes. >> >> >> Not to derail the conversation too much, as I know we're all getting burned >> out on the topic, but that last bit is not accurate: my druthers are to have >> __fspath__ be able to return str /or/ bytes, and if anything else comes from >> the object in question an exception must be raised. Maybe a word got lost >> between your thoughts and your fingers -- happens to me all the time. :) > > Yes. This would also be equivalent to my fspath(path, > type_constraint=(str,bytes)). And if the compromise I mentioned about > the rejecting (by default or optionally) is lifted, the keyword the rejecting -> rejecting bytes Good night. -- Koos > argument would not be needed. I might be ok with throwing away the > isinstance check on the return value of __fspath__() if it has > significant impact on performance in realistic cases (with DirEntry > most likely, I suppose), but I doubt it. > > -- Koos > >> >> -- >> ~Ethan~ >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 2:53 AM, Koos Zevenhoven wrote: > On Thu, May 12, 2016 at 2:49 AM, Koos Zevenhoven wrote: >> On Thu, May 12, 2016 at 2:05 AM, Ethan Furman wrote: >>> On 05/11/2016 03:13 PM, Brett Cannon wrote: >>> >>>> If [...] I would drop os.path changes and make os.fspath() do what >>> >>>> Ethan and Koos have suggested and simply pass through without checks >>>> >>>> whatever path.__fspath__() returned if the argument wasn't str or bytes. >>> >>> >>> Not to derail the conversation too much, as I know we're all getting burned >>> out on the topic, but that last bit is not accurate: my druthers are to have >>> __fspath__ be able to return str /or/ bytes, and if anything else comes from >>> the object in question an exception must be raised. Maybe a word got lost >>> between your thoughts and your fingers -- happens to me all the time. :) >> >> Yes. This would also be equivalent to my fspath(path, >> type_constraint=(str,bytes)). And if the compromise I mentioned about >> the rejecting (by default or optionally) is lifted, the keyword > > the rejecting -> rejecting bytes > >> argument would not be needed. I might be ok with throwing away the >> isinstance check on the return value of __fspath__() if it has >> significant impact on performance in realistic cases (with DirEntry >> most likely, I suppose), but I doubt it. >> >> -- Koos >> I will send a pull request about this tomorrow. -- Koos >>> >>> -- >>> ~Ethan~ >>> >>> ___ >>> Python-Dev mailing list >>> Python-Dev@python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 11:14 AM, Serhiy Storchaka wrote: > > This is cheap in C, but os.path functions are implemented in Python. They > have to make at least one function call (os.fspath(), hasattr() or > isinstance()), not counting a bytecode for retrieving arguments, resolving > attributes, comparing, jumps. Currently os.path functions use tricks to > avoid overheads > I suppose a C-implemented version of fspath *called from python* might be the fastest option at least in some cases. After all, a function call (isinstance or hasattr) is likely anyway, unless of course `try: path.__fspath__` is used. > Yet one problem is that currently many os,path functions work with > duck-typed strings (e.g. UserString). Using os.fspath() likely limit > supported types to str, bytes and types that support the path protocol. > Something like path = path.__fspath__() if hasattr(path, '__fspath__') else path as currently in the PEP, would not have this problem. However, I wonder whether such duck string paths actually exist (although it does remind me of my earlier experiments for solving the pathlib compatibility problem ;-). -- Koos P.S: I think it's great that you are concerned about the performance, and I find it important. However, this does feel like premature optimization to me at this point. We should first decide what functions should support the protocol, and after that, find the performance concerns and fix them. I have the feeling that the cases where this would be a performance bottleneck are quite rare, but if they are not, they may well be fixable. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 11:31 AM, Sven R. Kunze wrote: > On 11.05.2016 18:43, Brett Cannon wrote: >> >> Rationale >> = >> >> Historically in Python, file system paths have been represented as >> strings or bytes. This choice of representation has stemmed from C's >> own decision to represent file system paths as >> ``const char *`` [#libc-open]_. While that is a totally serviceable >> format to use for file system paths, it's not necessarily optimal. At >> issue is the fact that while all file system paths can be represented >> as strings or bytes, not all strings or bytes represent a file system >> path. > > > I can remember this argument being made during the discussion. I am not sure > if that 100% correct as soon as we talk about PurePaths. > I had suggested an alternative wording for this (see my commit on the work on Rationale). >> Proposal >> >> >> This proposal is split into two parts. One part is the proposal of a >> protocol for objects to declare and provide support for exposing a >> file system path representation. > > > https://docs.python.org/3/whatsnew/changelog.html says: > > "Add ‘path’ attribute to pathlib.Path objects, returning the same as str(), > to make it more similar to DirEntry. Library code can now write getattr(p, > ‘path’, p) to get the path as a string from a Path, a DirEntry, or a plain > string. This is essentially a small one-off protocol." > > So, in order to promote the "small one-off protocol" to a more broader > protocol, this PEP proposes a simple rename of .path to .__fspath__, is that > correct? > Well, I have brought this up previously several times. Indeed I see this as a further development of that duck-typing compatiblity approach. However, while the .path attribute is prior art, it has not been in a release yet. > Unfortunately, I don't have anything to contribute to the open issues. All > solutions have their pros and cons and everything that could be said has > been said. I think you need to decide. > Surprising enough, there are new things being said all the time. But luckily there seem to be signs of convergence. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 3:04 PM, Nick Coghlan wrote: > > I'd still like to see this exposed to Python code as os._raw_fspath() > (with the leading underscore just meaning "this probably isn't the API > you want" rather than indicating a private or unstable API), and then > fspath() defined as a wrapper around it which disallows bytes as > output. > I don't remember (should probably check) if you previously proposed implementing exactly that in C, but I indeed agree with what you write above, except that I don't like the "_raw_" prefix in the name. I would be willing to call that simply fspath though, since as mentioned before in this thread (I think by Brett and me), the reasons for rejecting bytes in fspath are really quite minor. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 4:20 PM, Nick Coghlan wrote: > > It's not unusual for me to encounter "POSIX oughtta be enough for > anyone" folks that are not yet entirely convinced that > bytes-are-not-text, so I'm actually in favour of making the default > Python-level API str-only as a healthy nudge away from the > "text-is-just-bytes-with-an-encoding!" school of thought. > This was also how I convinced myself about the default str constraint. However, I'm afraid it would be a weak weapon against using bytes paths, since the people using bytes paths would not be likely to call it, regardless of whether it supports bytes or not. The nice thing is that pathlib is str-only and *that* will push people away from bytes paths. > However, in terms of the three groups Brett articulated (maximum > flexibility, encouraging cross-platform correctness, and forgetting > the whole idea), I'm in both camps 1 & 2 - I work with POSIX enough > that I'm entirely on board with the notion that if you're specifically > modelling *POSIX* paths, then bytes-with-an-assumed-encoding is > frequently a good enough representation, but also deal with other > environments (like Windows, the JVM and the CLR) enough to know that > that particular representation of filesystem paths breaks down the > moment you expand your scope of interest beyond *nix platforms. > I also agree with parts about Brett's "camp 2". -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 7:24 PM, Guido van Rossum wrote: > I am glad this is finally happening. There's quite a bit of noise in the > thread which I have to ignore. The two issues that I want to respond to are > speed and whether os.fspath() can return bytes. > > - Speed: We should trust our ability to optimize the implementations where > necessary. First the API issues need to be settled. > > - Bytes: I strongly believe that os.fspath() should be a thin wrapper around > the __fspath__ protocol, like next() wraps the .__next__ protocol. It should > not get into bytes vs. string politics. If your app really needs strings, > call os.fsdecode(). So this is my version (unoptimized): > :) Thank you for this. I can breathe now. Some questions remain: > def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: > if isinstance(p, (str, bytes)): > return p > try: > return p.__fspath__ > except AttributeError: > raise TypeError(...) > (I know Brett already posted this question, but somehow it did not show up in my mailbox before I had written this. I'm (re)posting because there is some stuff here that is not in Brett's email ) You might be suggesting that __fspath__ should be an attribute, not a method, or did you mean something like: def fspath(p): if isinstance(p, (str, bytes)): return p try: p.__fspath__ except AttributeError: raise TypeError(...) return p.__fspath__() IMO, either is fine, I suppose. As you know, it's mostly a question of whether __fspath__ will be a property or a method (on PurePath for instance). But if you meant the former, that would change also the ABC and the protocol description. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 8:22 PM, Sjoerd Job Postmus wrote: > I would like to make just 1 comment regarding the question of accepting > (or not) bytes as output of `os.fspath`. > > The whole point of adding `os.fspath` is to make it easier to use Path > objects. This is in an effort to gain greater adoption of pathlib in > libraries. Now, this is an excellent idea. > > However, if it were to reject bytes, that would mean that when libraries > start to use pathlib, it would suddenly become harder for people that > actually need bytes-support to use pathlib. > > Now, the claim 'if you need bytes, you should not be using pathlib` is a > reasonable one. But what if I need bytes *and* a specific library (say, > image handling, or a web framework, or ...). It's not up to me if that > library uses pathlib or plain old os.path.join. > > Is using surrogate-escapes enough for this case? I myself am not sure, > (and also not affected), but it sounds to me that rejecting bytes is a > wrong approach if there is no proper workaround (assuming the use-case > of pathlib is somewhere deep in library code). > This is out of the scope of this PEP and probably a very insignificant issue (luckily, this is not the pathlib PEP). Surrogates will probably work and if not, on can "blaim" broken filenames ;). -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
This has just been discussed very recently in this thread (and earlier too). It may make sense, but it's not among our current worries. Besides, we already added the new fspath semantics to the PEP. While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett"). -- Koos On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: > On 12.05.2016 18:24, Guido van Rossum wrote: >> >> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: >> if isinstance(p, (str, bytes)): >> return p >> try: >> return p.__fspath__ >> except AttributeError: >> raise TypeError(...) > > > @Brett > Would you think it makes sense to swap the str/bytes check and the > __fspath__ check? > > > I just thought of a class subclassing str/bytes and defines __fspath__. Its > __fspath__ method would be ignored currently. > > > Best, > Sven > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, May 13, 2016 at 12:24 PM, Sven R. Kunze wrote: > On 13.05.2016 10:36, Koos Zevenhoven wrote: >> >> This has just been discussed very recently in this thread (and earlier >> too). > > > Could you point me to that? It seems I missed that part. I only found posts > related to performance degradation. > This issue is coupled with the future optimization questions. > However, the proposed semantics will change if the checks are swapped. So, > my actual question is: > > Is that an intended API inconsistency or a known bug supposed to be resolved > later? > Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent. I do think the documentation of the protocol should say that str or bytes subclasses should not implement __fspath__. So no API inconsistency there. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
Thanks Brett! Now one thing is that, despite your suggestion, I had not added myself as an author in my big pull request. Originally, it was because I simply forgot to copy and paste it when I split my edits into separate commits ;-). Sorry about that (not sure if you care though, and I've been defending the PEP regardless). Anyway, especially now that my main worry regarding the open questions has been resolved, I would be more than happy to have my name on it. So Brett, could you add me as author? (Koos Zevenhoven and k7ho...@gmail.com will be fine) It looks like this is finally happening :) -- Koos On Thu, May 12, 2016 at 11:53 PM, Brett Cannon wrote: > Second draft that takes Guido's comments into consideration. The biggest > change is os.fspath() now returns whatever path.__fspath__() returns instead > of restricting it to only str. > > Minor changes: > - Renamed the C function to PyOS_FSPath() > - Added an Implementation section with a TODO list > - Bunch of things added to the Rejected Ideas section > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 2:00 PM, Steven D'Aprano wrote:
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
> from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
> care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
I would think these have the b suffix because there is no good way to
infer which type should be returned. In things like os.path.join or
os.path.dirname you pass in the object(s) that determine the return
type.
In os.fspath, you pass in an object, whose type (str/bytes) or
"underlying path string type" (as returned by __fspath__()) determines
the return type of fspath. I think this is well in line with os.path
functions.
-- Koos
>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, May 13, 2016 at 7:06 PM, Sven R. Kunze wrote: > On 13.05.2016 11:48, Koos Zevenhoven wrote: >> >> This issue is coupled with the future optimization questions. >> > > AFAIC coupling API design to optimization is called premature optimization. > I suppose so, but currently I consider the API not to make guarantees about what happens if you pass in something other than specified as valid input. I'm not sure what level of precision is usual at this point, but I think this is sufficient for now. I'm sure we will pay attention to this (as we have been doing, both while optimizing prematurely and for other reasons). -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 8:52 PM, Steven D'Aprano wrote: > On Fri, May 13, 2016 at 03:43:29PM +, Brett Cannon wrote: >> On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote: > > [...] >> > I think it is a bit confusing to refer to "path objects", as that seems >> > like you are referring only to pathlib.Path objects. It took me far too >> > long to realise that here you mean generic path-like objects that obey >> > the __fspath__ protocol rather than a specific concrete class. >> > This terminology is indeed a bit difficult, not least because there are 6 different path classes in pathlib. A couple of months ago, I decided to start to call these pathlib objects and path objects, because I did not know what else to call them without doubling the length. >> > Since the ABC is called "PathLike", I suggest we refer to "path-like >> > objects" rather than "path objects", both in the PEP and in the Python >> > docs for this protocol. >> >> I went back and forth with this in my head while writing the PEP. The >> problem with making "path-like" mean "objects implementing the PathLike >> ABC" becomes how do you refer to an argument of a function that accepts >> anything os.fspath() does (i.e. PathLike, str, and bytes)? > > On further reflection, I think the right language is to use "path-like" > for the union of Pathlike, str and bytes. That will, I think, cover the > majority of cases: most functions which want to work on a file system > path should accept all three. When you want to specify only an object > which implements the PathLike ABC, that's called a (virtual) instance of > PathLike. > As I've told Brett before, I think exactly this reasoning would be a good enough reason not to call the ABC PathLike. [...] >> > Would I be right in saying that in practice this will actually end up >> > being type(path).__fspath__() to match the behaviour of all(?) other >> > dunder methods? >> >> I wasn't planning on it because for most types the accessing of the method >> directly off of the type for magic methods is because of some special >> struct field at the C level that we're pulling from. Since we're not >> planning to have an equivalent struct field I don't see any need to do the >> extra work of avoiding the instance participating in method lookup. >> Obviously if people disagree for some reason then please let me know (maybe >> for perf by avoiding the overhead of checking for the method on the >> instance?). > > The reasons I would disagree are: > > (1) It took me a long time to learn the rule that dunder methods are > always called from the class, not the instance, and now I have to learn > that there are exceptions? G argggh. Who knows, maybe the exception made it take a longer time.. IIRC the docs say that dunder methods are *not guaranteed* to be called on the instance, because they may be called on the class. > (2) If we ever do change to a C struct field, the behaviour will change. > Maybe it's better to emulate the same behaviour from the start? > > (3) If there's a performance speed up, that's a bonus! > > > > -- > Steven > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, 3rd draft
FYI, I recently sent a couple of emails in my earlier type hinting
thread on -ideas. What I wrote now is about the path ABC regarding
type hints.
-- Koos
On Sat, May 14, 2016 at 12:48 AM, Brett Cannon wrote:
>
>
> On Fri, 13 May 2016 at 14:30 Philip Jenvey wrote:
>>
>>
>> On May 13, 2016, at 11:37 AM, Brett Cannon wrote:
>>
>> Biggest changes since the second draft:
>>
>> Resolve __fspath__() from the type, not the instance (for Guido)
>>
>>
>> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
>> path,
>> NULL);
>>
>>
>> _PyObject_LookupSpecial would be preferable then.
>
>
> Yes it would be. :) I'll add it to the PEP.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, May 13, 2016 at 7:43 PM, Chris Angelico wrote:
[...]
> "Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
> is str"). The question is, which one SHOULD be being done?
Indeed it should do "Check", so that path libraries that do inherit
from str will still work (see also below).
> What should this do:
>
> class TmpPath(str):
> def __fspath__(self):
> return "/tmp/"+self
> x = TmpPath("foo/bar")
> open(x, "w")
>
> Does that respect __fspath__, or respect the fact that it's a string?
It should not respect __fspath__ because that would mean the behavior
is different in older versions of Python(!). This also means that such
code should never be written (except for that example you wrote ;-).
So indeed, composition would be the way to go.
So yes, I still think str subclasses should not implement __fspath__,
and that, at the very least, the docs should say that :).
-- Koos
> ChrisA
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the provisional label from pathlib
On Mon, May 23, 2016 at 10:38 PM, Chris Barker wrote: > On Mon, May 23, 2016 at 10:13 AM, Brett Cannon wrote: >> >> 3.5 is still getting bugfixes: >> https://docs.python.org/devguide/#status-of-python-branches >> >> As for backporting __fspath__() for pathlib, you can easily write your own >> subclass that adds it. And since the stdlib won't be updated in 3.5 for >> consumption of fspath > > > OK -- when I said "it", I meant the whole shebang -- i.e. the stdlib stuff > too. But fair enough, we can't be backporting everything, and I that would > be touching a lot of the lib. > > -CHB I guess we might consider adding __fspath__ in maintenance releases, and make open() support it? That would cover a significant share of use cases, although it might be weird if code written for 3.5.2 doesn't run on 3.5.1... So maybe just 3.6-> ? I'm not quite as busy as last week, so I might consider working on the stdlib changes if I find the time. Or is someone already working on this? -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the provisional label from pathlib
On Tue, May 24, 2016 at 4:56 PM, Barry Warsaw wrote: > On May 24, 2016, at 02:03 PM, Koos Zevenhoven wrote: > >>I guess we might consider adding __fspath__ in maintenance releases, >>and make open() support it? That would cover a significant share of >>use cases, although it might be weird if code written for 3.5.2 >>doesn't run on 3.5.1... > > Please, no. We learned that lesson in Python 2.2.1 with True/False. What happened? True was included in 2.2.1 but not False?-). Anyway, I guess you are probably right, and "3.6->" is the way to go. Besides, Guido already wrote that in the first response. -- Koos > Cheers, > -Barry > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Questions about about the DLS 2020
I've had some things going on, and I'm still trying to catch up with the
discussions here. Can someone tell me what would be the best place to look
at the most recent proposal? Is one of the PEPs up to date?
On Mon, Nov 16, 2020 at 7:02 PM Tobias Kohn wrote:
> Hi Mark,
>
> Thank you for your interest and the questions.
>
>
> 1. This really comes down to how you look at it, or how you define
> pattern matching. The issue here is that the concept of pattern matching
> has grown into a large and somewhat diverse flock of interpretations and
> implementations (as a side note: interestingly enough, some of the only
> universally agreed-upon standards are to use `_` as a wildcard and not to
> mark names that capture/bind values---which are quite exactly the points
> most fiercely debatted here).
>
> Anyway, the paper presents the pattern matching structure we are proposing
> as one of three major variants of pattern matching:
> (a) Matching arguments to parameters in a function call,
> (b) Matching elements to elements in iterable unpacking,
> (c) Matching tree-like data to general patterns in a conditional pattern
> matching structure.
>
> The last one is the subject of the PEP and the paper. Nonetheless, in the
> first two cases (a) and (b), we find that indeed the computer will validate
> that the data matched the pattern and raise an exception if this fails.
> This is where this way of looking at it comes from.
>
>
> 2. Yes, that is indeed a deliberate simplification. The idea is to
> abstract away from the details of how exactly Python implements abstract
> syntax trees (which I honestly believe are irrelevant for the sake of the
> entire narrative). Moreover, using strings here allows us to exemplify the
> literal patterns, rather only showcasing only the constructor/class pattern.
>
> Essentially, this is a question of making the most out of the little space
> available.
>
>
> Since you have addressed this email to me directly, I would like to take
> this opportunity and briefly stress that this paper really grew out of a
> team effort. While I might have been the one pushing for an academic
> publication, the DLS'20 paper represents the input and ideas of all the
> authors, as well as the long discussions we had. Of course, I am happy to
> answer any questions about the paper, but it would be wrong to see me as
> the one person behind it.
>
> Cheers,
> Tobias
>
>
>
> Quoting Mark Shannon :
>
> Hi Tobias,
>
> A couple of questions about the DLS 2020 paper.
>
> 1. Why do you use the term "validate" rather than "test" for the process
> of selecting a match?
>
> It seems to me, that this is a test, not a validation, as no exception is
> raised if a case doesn't match.
>
>
> 2. Is the error in the ast matching example, an intentional
> "simplification" or just an oversight?
>
> The example:
>
> ```
> def simplify(node):
>match node:
>case BinOp(Num(left), '+', Num(right)):
>return Num(left + right)
>case BinOp(left, '+' | '-', Num(0)):
>return simplify(left)
>case UnaryOp('-', UnaryOp('-', item)):
>return simplify(item)
>case _:
>return node
>
> ```
>
> is wrong.
>
> The correct version is
>
> ```
> def simplify(node):
>match node:
>case BinOp(Num(left), Add(), Num(right)):
>return Num(left + right)
>case BinOp(left, Add() | Sub(), Num(0)):
>return simplify(left)
>case UnaryOp(USub(), UnaryOp(USub(), item)):
>return simplify(item)
>case _:
>return node
> ```
>
> Cheers,
> Mark.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ETZGYRCF4DR6RJXTHGXIRZXINXJ76J2D/Code
> of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/CUNO57W7KSIM2WRROC5R43ZT7HUQZCZ6/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/FDP476BPKG55W2OP66VPMMGV5OBP67U3/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview
On Tue, Jun 7, 2016 at 11:28 PM, Ethan Furman wrote: > > Minor changes: updated version numbers, add punctuation. > > The current text seems to take into account Guido's last comments. > > Thoughts before asking for acceptance? > > PEP: 467 > Title: Minor API improvements for binary sequences > Version: $Revision$ > Last-Modified: $Date$ > Author: Nick Coghlan > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2014-03-30 > Python-Version: 3.5 > Post-History: 2014-03-30 2014-08-15 2014-08-16 > > > Abstract > > > During the initial development of the Python 3 language specification, the > core ``bytes`` type for arbitrary binary data started as the mutable type > that is now referred to as ``bytearray``. Other aspects of operating in the > binary domain in Python have also evolved over the course of the Python 3 > series. > > This PEP proposes four small adjustments to the APIs of the ``bytes``, > ``bytearray`` and ``memoryview`` types to make it easier to operate entirely > in the binary domain: > > * Deprecate passing single integer values to ``bytes`` and ``bytearray`` > * Add ``bytes.zeros`` and ``bytearray.zeros`` alternative constructors > * Add ``bytes.byte`` and ``bytearray.byte`` alternative constructors > * Add ``bytes.iterbytes``, ``bytearray.iterbytes`` and > ``memoryview.iterbytes`` alternative iterators > Why not bytes.viewbytes (or whatever name) so that one could also subscript it? And if it were a property, one could perhaps conveniently get the n'th byte: b'abcde'.viewbytes[n] # compared to b'abcde'[n:n+1] Also, would it not be more clear to call the int -> bytes method something like bytes.fromint or bytes.fromord and introduce the same thing on str? And perhaps allow multiple arguments to create a str/bytes of length > 1. I guess this may violate TOOWTDI, but anyway, just a thought. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 467: Minor API improvements to bytes, bytearray, and memoryview
On Wed, Jun 8, 2016 at 12:57 AM, Barry Warsaw wrote: > On Jun 07, 2016, at 09:40 PM, Brett Cannon wrote: > >>On Tue, 7 Jun 2016 at 14:38 Paul Sokolovsky wrote: >>> What's wrong with b[i:i+1] ? >>It always succeeds while indexing can trigger an IndexError. > > Right. You want a method with the semantics of __getitem__() but that returns > the desired type. > And if this is called __getitem__ (with slices delegated to bytes.__getitem__) and implemented in a class, one has a view. Maybe I'm missing something, but I fail to understand what makes this significantly more problematic than an iterator. Ok, I guess we might also need __len__. -- Koos > -Barry > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] proposed os.fspath() change
lass Huh: >> ... def __int__(self): >> ... return 'string' >> ... def __index__(self): >> ... return b'bytestring' >> ... def __bool__(self): >> ... return 'true-ish' >> ... >> --> h = Huh() >> >> --> int(h) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: __int__ returned non-int (type str) >> >> --> ''[h] >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: __index__ returned non-int (type bytes) >> >> --> bool(h) >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: __bool__ should return bool, returned str >> >> Arguments in favor or against? >> >> -- >> ~Ethan~ >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (python.org/~guido) > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > -- -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
