Re: [Python-Dev] file system path protocol PEP
On 12.05.2016 19:27, Ethan Furman wrote: Maybe, but a bad idea for two reasons: 1) Reducing a str to the exact same str is silly; and, more importantly Finding something silly is no technical argument. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 12.05.2016 18:24, Guido van Rossum wrote: def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...) @Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check? I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
This has just been discussed very recently in this thread (and earlier too). It may make sense, but it's not among our current worries. Besides, we already added the new fspath semantics to the PEP. While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett"). -- Koos On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: > On 12.05.2016 18:24, Guido van Rossum wrote: >> >> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: >> if isinstance(p, (str, bytes)): >> return p >> try: >> return p.__fspath__ >> except AttributeError: >> raise TypeError(...) > > > @Brett > Would you think it makes sense to swap the str/bytes check and the > __fspath__ check? > > > I just thought of a class subclassing str/bytes and defines __fspath__. Its > __fspath__ method would be ignored currently. > > > Best, > Sven > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 13.05.2016 10:36, Koos Zevenhoven wrote: This has just been discussed very recently in this thread (and earlier too). Could you point me to that? It seems I missed that part. I only found posts related to performance degradation. However, the proposed semantics will change if the checks are swapped. So, my actual question is: Is that an intended API inconsistency or a known bug supposed to be resolved later? It may make sense, but it's not among our current worries. It might not be yours but mine. ;) That's why I was asking. Besides, we already added the new fspath semantics to the PEP. While I hope Brett is asleep in his time zone, I'm guessing he will agree (just saying this because you write "@Brett"). -- Koos On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: On 12.05.2016 18:24, Guido van Rossum wrote: def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: if isinstance(p, (str, bytes)): return p try: return p.__fspath__ except AttributeError: raise TypeError(...) @Brett Would you think it makes sense to swap the str/bytes check and the __fspath__ check? I just thought of a class subclassing str/bytes and defines __fspath__. Its __fspath__ method would be ignored currently. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 05/12/2016 05:42 PM, Ethan Furman wrote: And even given all that, for smoother interoperability with the rest of the stdlib, or at least the os.* portion, those functions would still need to be upgraded to check for .path on the incoming arguments -- at which point we may as well make a protocol to properly support file system paths instead of relying on the rather generic attribute name of 'path'. FWIW, this shouldn't be very difficult. Paths are already complicated enough that the parsing has been factored out into a PyArg_Parse "converter" called path_converter. path_converter isn't used everywhere in posixmodule yet, but it would probably make sense to convert everything to use it anyway. And then we would only need to implement Path object support once. It's remotely possible that converting functions to use path_converter will create slight incompatibilities (though I don't know how, this is just FUD really) so 3.6 would be a good opportunity to make this change. Optimistically, //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, May 13, 2016 at 12:24 PM, Sven R. Kunze wrote: > On 13.05.2016 10:36, Koos Zevenhoven wrote: >> >> This has just been discussed very recently in this thread (and earlier >> too). > > > Could you point me to that? It seems I missed that part. I only found posts > related to performance degradation. > This issue is coupled with the future optimization questions. > However, the proposed semantics will change if the checks are swapped. So, > my actual question is: > > Is that an intended API inconsistency or a known bug supposed to be resolved > later? > Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent. I do think the documentation of the protocol should say that str or bytes subclasses should not implement __fspath__. So no API inconsistency there. -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Thu, May 12, 2016 at 07:22:25PM +0200, Sjoerd Job Postmus wrote: > The whole point of adding `os.fspath` is to make it easier to use Path > objects. This is in an effort to gain greater adoption of pathlib in > libraries. Now, this is an excellent idea. > > However, if it were to reject bytes, that would mean that when libraries > start to use pathlib, it would suddenly become harder for people that > actually need bytes-support to use pathlib. I'm not sure how it would be *harder*. Currently, if you need bytes paths, you can't use pathlib. That won't change, so it will be exactly as hard as it is now. > Now, the claim 'if you need bytes, you should not be using pathlib` is a > reasonable one. But what if I need bytes *and* a specific library (say, > image handling, or a web framework, or ...). It's not up to me if that > library uses pathlib or plain old os.path.join. I feel this is Not Our Problem. Surely the stdlib cannot be held responsible for all the poor decisions made by third-party libraries? If a library hard-codes "\\" as their directory separator, because everyone uses Windows, you'd simply say "this library is not very good for my use-case". Likewise if it rejects bytes paths, then: - perhaps that library is not fit for your purpose; - perhaps you can use it, but with the limitation that you can no longer support all paths, but only the ones which can be accessed via a string path; - perhaps you can build a compatibility layer that lets you use both bytes and the library. I don't doubt that all three options are unpleasant, but I don't see how they are the fault of this PEP. This PEP supports Path-Like objects that return bytes paths. Its up to library authors to support bytes paths, or not, as they see fit. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, May 13, 2016 at 8:19 PM, Steven D'Aprano wrote: > I feel this is Not Our Problem. Surely the stdlib cannot be held > responsible for all the poor decisions made by third-party libraries? If > a library hard-codes "\\" as their directory separator, because everyone > uses Windows, you'd simply say "this library is not very good for my > use-case". Likewise if it rejects bytes paths, then: > > - perhaps that library is not fit for your purpose; > > - perhaps you can use it, but with the limitation that you can no > longer support all paths, but only the ones which can be > accessed via a string path; > > - perhaps you can build a compatibility layer that lets you use > both bytes and the library. AFAICT, the compatibility layer would simply decode the bytes using surrogateescape handling, which should round-trip anything. Or am I wrong here somewhere? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Thu, May 12, 2016 at 08:53:12PM +, Brett Cannon wrote:
> Second draft that takes Guido's comments into consideration. The biggest
> change is os.fspath() now returns whatever path.__fspath__() returns
> instead of restricting it to only str.
Counter suggestion:
- __fspath__() method may return either bytes or str (no change
from the PEP as it stands now);
- but os.fspath() will only return str;
- and os.fspathb() will only return bytes;
- there is no os function that returns "str or bytes, I don't
care which". (If you really need that, call __fspath__ directly.)
Note that this differs from the already rejected suggestion that there
should be two dunder methods, __fspath__() and __fspathb__().
Why?
(1) Normally, the caller knows whether they want str or bytes. (That's
been my experience, you may disagree.) If so, and they call os.fspath()
expecting a str, they won't be surprised by it returning bytes. And visa
versa for when you expect a bytes path.
(2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
Cons:
(3) Polymorphic code that truly doesn't care whether it gets bytes or
str will have a slightly less convenient way of getting it, namely by
calling __fspath__() itself, instead of os.fspath().
A few other comments below:
> builtins
>
>
> ``open()`` [#builtins-open]_ will be updated to accept path objects as
> well as continue to accept ``str`` and ``bytes``.
I think it is a bit confusing to refer to "path objects", as that seems
like you are referring only to pathlib.Path objects. It took me far too
long to realise that here you mean generic path-like objects that obey
the __fspath__ protocol rather than a specific concrete class.
Since the ABC is called "PathLike", I suggest we refer to "path-like
objects" rather than "path objects", both in the PEP and in the Python
docs for this protocol.
> def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str, bytes]:
> """Return the string representation of the path.
>
> If str or bytes is passed in, it is returned unchanged.
> """
I've already suggested a change to this, above, but independent of that,
a minor technical query:
> try:
> return path.__fspath__()
Would I be right in saying that in practice this will actually end up
being type(path).__fspath__() to match the behaviour of all(?) other
dunder methods?
--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 9:00 PM, Steven D'Aprano wrote: > Cons: > (3) Polymorphic code that truly doesn't care whether it gets bytes or > str will have a slightly less convenient way of getting it, namely by > calling __fspath__() itself, instead of os.fspath(). I don't like this; it goes against the general principle that dunders are for defining, not calling. Generally, a given dunder method has approximately one call site, eg __reduce__ in pickle.py, and everyone else defines it. (You might call super's dunder in the definition of your own, but that's still defining it, not calling it.) Having an official statement that it's appropriate to call a dunder confuses this. So this isn't a "slightly less convenient way", it's a bad way (IMO), and this is a very strong con. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On 13 May 2016 at 21:00, Steven D'Aprano wrote: > On Thu, May 12, 2016 at 08:53:12PM +, Brett Cannon wrote: > >> Second draft that takes Guido's comments into consideration. The biggest >> change is os.fspath() now returns whatever path.__fspath__() returns >> instead of restricting it to only str. > > Counter suggestion: > > - __fspath__() method may return either bytes or str (no change > from the PEP as it stands now); > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; We don't have any use cases for a bytes-only API - only str-or-bytes (in the polymorphic low level functions) and str-only (in newer high level APIs like pathlib). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
Thanks Brett! Now one thing is that, despite your suggestion, I had not added myself as an author in my big pull request. Originally, it was because I simply forgot to copy and paste it when I split my edits into separate commits ;-). Sorry about that (not sure if you care though, and I've been defending the PEP regardless). Anyway, especially now that my main worry regarding the open questions has been resolved, I would be more than happy to have my name on it. So Brett, could you add me as author? (Koos Zevenhoven and k7ho...@gmail.com will be fine) It looks like this is finally happening :) -- Koos On Thu, May 12, 2016 at 11:53 PM, Brett Cannon wrote: > Second draft that takes Guido's comments into consideration. The biggest > change is os.fspath() now returns whatever path.__fspath__() returns instead > of restricting it to only str. > > Minor changes: > - Renamed the C function to PyOS_FSPath() > - Added an Implementation section with a TODO list > - Bunch of things added to the Rejected Ideas section > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
> - there is no os function that returns "str or bytes, I don't > care which". (If you really need that, call __fspath__ directly.) os.fspath() in the PEP works when given str or bytes directly, but those don't have a __fspath__ method, so directly calling the dunder method is not equivalent to calling os.fspath(). The whole point of having os.fspath() is to pass str or paths (or bytes) and get an appropriate object for lew-level functions. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
> On 13 May 2016, at 08:07, Ethan Furman wrote: > > On 05/12/2016 01:59 PM, Sjoerd Job Postmus wrote: >>> On 12 May 2016, at 21:30, Ethan Furman wrote: >>> >>> If you need bytes support for your paths, there's at least one [1] that has >>> that support. >> >> So if I would need bytes support, I should submit a pull request to >> > library> which replaces usage of the stdlib pathlib with another variant, > > upon which they > > will decline the pull request because it introduces another "useless" > > dependency. > > My apologies, I thought you were talking about your own code. > > As far as -- it wouldn't be awesome to me if > I couldn't pass in the data types I need. > > -- > ~Ethan~ I just hope you are right. Thank you (and Brett) for clearing things up for me. My apologies if I came off as (a bit) harsh. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016, at 07:00, Steven D'Aprano wrote: > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; And raise an exception if __fspath__ returns the other, I suppose. What's the use case for these functions? When would I call them rather than fsdecode and fsencode? (assuming the latter will support the path protocol - I've got more objections if they're not going to) Also, what happens if you pass a string to os.fspath? Statements like "str will not implement __fspath__" have thus far been light on details of what functions will accept "str or path-like-object-whose-__fspath__returns-str" and which ones will only accept the latter (and what, if any, will continue to only accept the former). If no-one's supposed to directly call dunder methods, and os.fspath accepts str, then what difference does it make whether this is implemented by having a special case within os.fspath or by calling str.__fspath__? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, 13 May 2016 at 02:25 Sven R. Kunze wrote: > On 13.05.2016 10:36, Koos Zevenhoven wrote: > > This has just been discussed very recently in this thread (and earlier > > too). > > Could you point me to that? It seems I missed that part. I only found > posts related to performance degradation. > > However, the proposed semantics will change if the checks are swapped. > So, my actual question is: > > Is that an intended API inconsistency or a known bug supposed to be > resolved later? > Purposeful change. It was what I had in my head after I posted my "groups" breakdown email and what Guido suggested as well independently. This helps alleviate any perf worries as type checks in C are pointer checks that are cheap to make compared to attribute lookup. And even if we do get the community to move on to path objects the check is still cheap enough to not worry. And finally, this is simply how we have almost always coded this kind of special-casing at the C level throughout Python and the stdlib, so it now is more inline with standard coding practices than the original proposal in the PEP. > > > It may make sense, but it's not among our current worries. > > It might not be yours but mine. ;) That's why I was asking. > Yes, I was very much asleep. :) -Brett > > > Besides, we already added the new fspath semantics to the PEP. > > > > While I hope Brett is asleep in his time zone, I'm guessing he will > > agree (just saying this because you write "@Brett"). > > > > -- Koos > > > > > > On Fri, May 13, 2016 at 10:58 AM, Sven R. Kunze wrote: > >> On 12.05.2016 18:24, Guido van Rossum wrote: > >>> def fspath(p: Union[str, bytes, PathLike]) -> Union[str, bytes]: > >>> if isinstance(p, (str, bytes)): > >>> return p > >>> try: > >>> return p.__fspath__ > >>> except AttributeError: > >>> raise TypeError(...) > >> > >> @Brett > >> Would you think it makes sense to swap the str/bytes check and the > >> __fspath__ check? > >> > >> > >> I just thought of a class subclassing str/bytes and defines __fspath__. > Its > >> __fspath__ method would be ignored currently. > >> > >> > >> Best, > >> Sven > >> > >> ___ > >> Python-Dev mailing list > >> Python-Dev@python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote:
> On Thu, May 12, 2016 at 08:53:12PM +, Brett Cannon wrote:
>
> > Second draft that takes Guido's comments into consideration. The biggest
> > change is os.fspath() now returns whatever path.__fspath__() returns
> > instead of restricting it to only str.
>
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
> from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
> care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>
I prefer what's in the PEP. I get where you coming from, Steven, but I
don't think it will be common enough to worry about. Think of os.fspath()
like next() where it truly is a very minor convenience function that
happens to special-case str and bytes.
>
>
>
>
> A few other comments below:
>
>
> > builtins
> >
> >
> > ``open()`` [#builtins-open]_ will be updated to accept path objects as
> > well as continue to accept ``str`` and ``bytes``.
>
> I think it is a bit confusing to refer to "path objects", as that seems
> like you are referring only to pathlib.Path objects. It took me far too
> long to realise that here you mean generic path-like objects that obey
> the __fspath__ protocol rather than a specific concrete class.
>
> Since the ABC is called "PathLike", I suggest we refer to "path-like
> objects" rather than "path objects", both in the PEP and in the Python
> docs for this protocol.
>
I went back and forth with this in my head while writing the PEP. The
problem with making "path-like" mean "objects implementing the PathLike
ABC" becomes how do you refer to an argument of a function that accepts
anything os.fspath() does (i.e. PathLike, str, and bytes)?
>
>
>
>
> > def fspath(path: t.Union[PathLike, str, bytes]) -> t.Union[str,
> bytes]:
> > """Return the string representation of the path.
> >
> > If str or bytes is passed in, it is returned unchanged.
> > """
>
> I've already suggested a change to this, above, but independent of that,
> a minor technical query:
>
> > try:
> > return path.__fspath__()
>
> Would I be right in saying that in practice this will actually end up
> being type(path).__fspath__() to match the behaviour of all(?) other
> dunder methods?
>
I wasn't planning on it because for most types the accessing of the method
directly off of the type for magic methods is because of some special
struct field at the C level that we're pulling from. Since we're not
planning to have an equivalent struct field I don't see any need to do the
extra work of avoiding the instance participating in method lookup.
Obviously if people disagree for some reason then please let me know (maybe
for perf by avoiding the overhead of checking for the method on the
instance?).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 13.05.2016 11:48, Koos Zevenhoven wrote: This issue is coupled with the future optimization questions. AFAIC coupling API design to optimization is called premature optimization. However, the proposed semantics will change if the checks are swapped. So, my actual question is: Is that an intended API inconsistency or a known bug supposed to be resolved later? Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent. From your perspective. As far as I remember, one goal of this proposal was to avoid wallet gardens. During the lengthy discussion on python-ideas people brought up that some third-party libs indeed subclass from str. They are currently locked out. I do think the documentation of the protocol should say that str or bytes subclasses should not implement __fspath__. Indeed. Just one minor note here: str or bytes subclasses *can* implement __fspath__ and currently it will be *ignored*. Maybe that changes in the future. So, that's the reason it should not be implemented. So no API inconsistency there. API consistency is not defined by docs-matching-implementation but by implementation-matching-expectations. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2016-05-06 - 2016-05-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open5501 ( -3) closed 33270 (+47) total 38771 (+44) Open issues with patches: 2383 Issues opened (28) == #22570: Better stdlib support for Path objects http://bugs.python.org/issue22570 reopened by gvanrossum #26857: gethostbyname_r() is broken on android http://bugs.python.org/issue26857 reopened by xdegaye #26971: python v3.5.1: sys.paths not respecting DESTDIRS and DESTSHARE http://bugs.python.org/issue26971 opened by yaro-yaro #26972: mistakes in docstrings in the import machinery http://bugs.python.org/issue26972 opened by Oren Milman #26974: Crash in Decimal.from_float http://bugs.python.org/issue26974 opened by serhiy.storchaka #26975: Decimal.from_float works incorrectly for non-binary floats http://bugs.python.org/issue26975 opened by serhiy.storchaka #26978: Implement pathlib.Path.link (Using os.link) http://bugs.python.org/issue26978 opened by cool-RR #26979: The danger of PyType_FromSpec() http://bugs.python.org/issue26979 opened by serhiy.storchaka #26980: The path argument of asyncio.BaseEventLoop.create_unix_connect http://bugs.python.org/issue26980 opened by texttheater #26981: add compatibility shim for enum34 backport http://bugs.python.org/issue26981 opened by ethan.furman #26982: Clarify forward annotations in PEP 484 http://bugs.python.org/issue26982 opened by James.Tatum #26983: float() can return not exact float instance http://bugs.python.org/issue26983 opened by serhiy.storchaka #26984: int() can return not exact int instance http://bugs.python.org/issue26984 opened by serhiy.storchaka #26985: Information about CodeType in inspect documentation is outdate http://bugs.python.org/issue26985 opened by xiang.zhang #26988: Add AutoNumberedEnum to stdlib http://bugs.python.org/issue26988 opened by John Hagen #26990: file.tell affect decoding http://bugs.python.org/issue26990 opened by mfmain #26991: Possible reference leak in MAKE_FUNCTION http://bugs.python.org/issue26991 opened by xiang.zhang #26992: 64-bit Python 2.7.11 hangs in 64-bit Windows 10 - CMD and Git http://bugs.python.org/issue26992 opened by sam.taylor.bs #26993: Copy idlelib *.py files with new names http://bugs.python.org/issue26993 opened by terry.reedy #26995: Add tests for parsing float and object arguments http://bugs.python.org/issue26995 opened by serhiy.storchaka #26997: Docs for pdb should note that __future__ magic doesn't work http://bugs.python.org/issue26997 opened by Martin Jones #27002: Support different modes in posixpath.realpath() http://bugs.python.org/issue27002 opened by serhiy.storchaka #27004: Handle script shbang options http://bugs.python.org/issue27004 opened by opoplawski #27006: C implementation of Decimal.from_float() bypasses __new__ and http://bugs.python.org/issue27006 opened by serhiy.storchaka #27007: Alternate constructors bytes.fromhex() and bytearray.fromhex() http://bugs.python.org/issue27007 opened by serhiy.storchaka #27010: email library could "recover" from bad mime boundary like (som http://bugs.python.org/issue27010 opened by Fedele Mantuano #27012: Rename the posix module to _os http://bugs.python.org/issue27012 opened by brett.cannon #27014: maximum recursion depth when using typing options http://bugs.python.org/issue27014 opened by Roy Shmueli Most recent 15 issues with no replies (15) == #27014: maximum recursion depth when using typing options http://bugs.python.org/issue27014 #27006: C implementation of Decimal.from_float() bypasses __new__ and http://bugs.python.org/issue27006 #27002: Support different modes in posixpath.realpath() http://bugs.python.org/issue27002 #26997: Docs for pdb should note that __future__ magic doesn't work http://bugs.python.org/issue26997 #26995: Add tests for parsing float and object arguments http://bugs.python.org/issue26995 #26990: file.tell affect decoding http://bugs.python.org/issue26990 #26982: Clarify forward annotations in PEP 484 http://bugs.python.org/issue26982 #26980: The path argument of asyncio.BaseEventLoop.create_unix_connect http://bugs.python.org/issue26980 #26979: The danger of PyType_FromSpec() http://bugs.python.org/issue26979 #26978: Implement pathlib.Path.link (Using os.link) http://bugs.python.org/issue26978 #26964: Incorrect documentation for `-u`-flag http://bugs.python.org/issue26964 #26955: Implement equivalent to `pip.locations.distutils_scheme` in di http://bugs.python.org/issue26955 #26954: Add Guido's rejection notice to the "with" FAQ http://bugs.python.org/issue26954 #26942: android: test_ctypes crashes on armv7 http://bugs.python.org/issue26942 #26937: android: test_tarfile fails http://bugs.python.org/issue26937 Most recent 15 issues waiting f
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 4:00 AM, Steven D'Aprano
wrote:
> On Thu, May 12, 2016 at 08:53:12PM +, Brett Cannon wrote:
>
> > Second draft that takes Guido's comments into consideration. The biggest
> > change is os.fspath() now returns whatever path.__fspath__() returns
> > instead of restricting it to only str.
>
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
> from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
> care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>
More cons:
- It would be confusing that there'd be no direct corresponding between
os.fspath(x) and x.__fspath__(), unlike for most other dunders (next(x) ->
x.__next__(), iter(x) -> x.__iter__(), and so on).
- Your examples like os.getcwd() have no input to determine whether to
return bytes or str, so for those we use an alternate name to request
bytes. However for most os methods the same method handles both, e.g.
os.listdir(b'.') will return a list of bytes objects while os.listdir('.')
will return a list of strings. os.fspath() firmly falls in the latter camp:
it takes an input whose __fspath__() method will return either str or
bytes, so that's all the guidance it needs.
Really, if you want bytes, you should use os.fsencode(); if you want
strings, use os.fsencode(); if you want to be polymorphic, use os.fspath()
and check the type it returns. I find the case where you'd explicitly want
to exclude bytes unusual; Nick already proposed pathlib.Path(os.fspath(x))
for that.
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 2:00 PM, Steven D'Aprano wrote:
> Counter suggestion:
>
> - __fspath__() method may return either bytes or str (no change
> from the PEP as it stands now);
>
> - but os.fspath() will only return str;
>
> - and os.fspathb() will only return bytes;
>
> - there is no os function that returns "str or bytes, I don't
> care which". (If you really need that, call __fspath__ directly.)
>
> Note that this differs from the already rejected suggestion that there
> should be two dunder methods, __fspath__() and __fspathb__().
>
> Why?
>
> (1) Normally, the caller knows whether they want str or bytes. (That's
> been my experience, you may disagree.) If so, and they call os.fspath()
> expecting a str, they won't be surprised by it returning bytes. And visa
> versa for when you expect a bytes path.
>
> (2) This behaviour will match that of os.{environ[b],getcwd[b],getenv[b]}.
I would think these have the b suffix because there is no good way to
infer which type should be returned. In things like os.path.join or
os.path.dirname you pass in the object(s) that determine the return
type.
In os.fspath, you pass in an object, whose type (str/bytes) or
"underlying path string type" (as returned by __fspath__()) determines
the return type of fspath. I think this is well in line with os.path
functions.
-- Koos
>
> Cons:
>
> (3) Polymorphic code that truly doesn't care whether it gets bytes or
> str will have a slightly less convenient way of getting it, namely by
> calling __fspath__() itself, instead of os.fspath().
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On 05/13/2016 08:43 AM, Brett Cannon wrote: a minor technical query: try: return path.__fspath__() Would I be right in saying that in practice this will actually end up being type(path).__fspath__() to match the behaviour of all(?) other dunder methods? I wasn't planning on it because for most types the accessing of the method directly off of the type for magic methods is because of some special struct field at the C level that we're pulling from. Since we're not planning to have an equivalent struct field I don't see any need to do the extra work of avoiding the instance participating in method lookup. Obviously if people disagree for some reason then please let me know (maybe for perf by avoiding the overhead of checking for the method on the instance?). I would say use `type(x).__fspath__`. I'm not aware of any other __dunder__ method that doesn't access the attribute from the type instead of the instance, and I see no point in making this one different. I know there's a line in the Zen about foolish conistencies, but I suspect there's a corollary about foolish inconsistencies. ;) -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On 05/13/2016 06:21 PM, Guido van Rossum wrote: Really, if you want bytes, you should use os.fsencode(); if you want strings, use os.fsencode(); if you want to be polymorphic, use os.fspath() and check the type it returns. Am I severely misunderstanding the API, or did you mean "if you want strings, use os.fsdecode()" here? //arry/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 13.05.2016 17:29, Brett Cannon wrote: Purposeful change. It was what I had in my head after I posted my "groups" breakdown email and what Guido suggested as well independently. This helps alleviate any perf worries as type checks in C are pointer checks that are cheap to make compared to attribute lookup. And even if we do get the community to move on to path objects the check is still cheap enough to not worry. And finally, this is simply how we have almost always coded this kind of special-casing at the C level throughout Python and the stdlib, so it now is more inline with standard coding practices than the original proposal in the PEP. Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect subclass hierarchies such as isinstance(path, str) does? Could not find the definition of it in the source. You said it's just a pointer comparison. To me that implies it could not respect subclasses as isinstance does. If that is the case, the C implementation differs from the Python version in semantics. I'm sorry if I misunderstood the whole implementation here. Best, Sven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Sat, May 14, 2016 at 2:33 AM, Sven R. Kunze wrote:
> On 13.05.2016 17:29, Brett Cannon wrote:
>>
>> Purposeful change. It was what I had in my head after I posted my "groups"
>> breakdown email and what Guido suggested as well independently. This helps
>> alleviate any perf worries as type checks in C are pointer checks that are
>> cheap to make compared to attribute lookup. And even if we do get the
>> community to move on to path objects the check is still cheap enough to not
>> worry. And finally, this is simply how we have almost always coded this kind
>> of special-casing at the C level throughout Python and the stdlib, so it now
>> is more inline with standard coding practices than the original proposal in
>> the PEP.
>
>
> Maybe, we have a misunderstanding here. Does "PyUnicode_Check(path)" respect
> subclass hierarchies such as isinstance(path, str) does? Could not find the
> definition of it in the source.
>
> You said it's just a pointer comparison. To me that implies it could not
> respect subclasses as isinstance does. If that is the case, the C
> implementation differs from the Python version in semantics. I'm sorry if I
> misunderstood the whole implementation here.
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Check
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_CheckExact
"Check" accepts subclasses; "CheckExact" doesn't (it's like "type(x)
is str"). The question is, which one SHOULD be being done? What should
this do:
class TmpPath(str):
def __fspath__(self):
return "/tmp/"+self
x = TmpPath("foo/bar")
open(x, "w")
Does that respect __fspath__, or respect the fact that it's a string?
ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Sat, May 14, 2016 at 2:34 AM, Ethan Furman wrote: > I would say use `type(x).__fspath__`. I'm not aware of any other __dunder__ > method that doesn't access the attribute from the type instead of the > instance, and I see no point in making this one different. > __reduce__ / __reduce_ex__ in pickle.py is accessed with a straight-forward getattr() call. It's the ones that are called from deep within the interpreter core (eg __iter__) that are always looked up on the type. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 13 May 2016 at 17:06, Sven R. Kunze wrote: > As far as I remember, one goal of this proposal was to avoid wallet gardens. > During the lengthy discussion on python-ideas people brought up that some > third-party libs indeed subclass from str. They are currently locked out. Hardly. Libraries that subclass from str will provide their string representation precisely by being (a subclass of) a string. So they don't need the __fspath__ protocol, and can continue to work exactly as now. This PEP merely provides the same capability for libraries that prefer *not* to subclass str. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 05/13/2016 09:06 AM, Sven R. Kunze wrote: On 13.05.2016 11:48, Koos Zevenhoven wrote: >> Sven R Kunze had previously queried: However, the proposed semantics will change if the checks are swapped. So, my actual question is: Is that an intended API inconsistency or a known bug supposed to be resolved later? Taking into account the description (and the drafted type hint), which the documentation will probably reflect, the semantic effects of that are very minor or nonexistent. From your perspective. As far as I remember, one goal of this proposal was to avoid wallet gardens. During the lengthy discussion on python-ideas people brought up that some third-party libs indeed subclass from str. They are currently locked out. Two points: 1) What is a wallet garden? 2) If a third-party path lib subclasses either str or bytes, and the os.* and other stdlib functions accept str or bytes, how are they locked out by not having an __fspath__? Assuming, of course, that those functions use isinstance and not type (if do use type a bug should be filed against them. Indeed. Just one minor note here: str or bytes subclasses *can* implement __fspath__ and currently it will be *ignored*. Sure. And int subclasses can implement __index__, which will likewise be ignored. So no API inconsistency there. API consistency is not defined by docs-matching-implementation but by implementation-matching-expectations. Well, probably both. ;) -- ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On Fri, May 13, 2016 at 7:06 PM, Sven R. Kunze wrote: > On 13.05.2016 11:48, Koos Zevenhoven wrote: >> >> This issue is coupled with the future optimization questions. >> > > AFAIC coupling API design to optimization is called premature optimization. > I suppose so, but currently I consider the API not to make guarantees about what happens if you pass in something other than specified as valid input. I'm not sure what level of precision is usual at this point, but I think this is sufficient for now. I'm sure we will pay attention to this (as we have been doing, both while optimizing prematurely and for other reasons). -- Koos ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 9:33 AM, Larry Hastings wrote: > > > On 05/13/2016 06:21 PM, Guido van Rossum wrote: > > Really, if you want bytes, you should use os.fsencode(); if you want > strings, use os.fsencode(); if you want to be polymorphic, use os.fspath() > and check the type it returns. > > > Am I severely misunderstanding the API, or did you mean "if you want > strings, use os.fsdecode()" here? > encode, decode -- poh-tay-toe, poh-tah-toe. :-) Another slip of the fingers, I did mean os.fsdecode() when you want strings. encode, schmencode... -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 9:34 AM, Ethan Furman wrote: > I would say use `type(x).__fspath__`. I'm not aware of any other > __dunder__ method that doesn't access the attribute from the type instead > of the instance, and I see no point in making this one different. > Agreed. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
On 13 May 2016 at 17:57, Ethan Furman wrote: > 1) What is a wallet garden? I assumed he meant "walled garden" (for people who don't know the phrase, it refers to an area or capability that's only made accessible to "favoured" clients - think of something like Apple's App Store, where you have to be approved to publish there). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 03:43:29PM +, Brett Cannon wrote: > On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote: [...] > > - but os.fspath() will only return str; > > - and os.fspathb() will only return bytes; > I prefer what's in the PEP. I get where you coming from, Steven, but I > don't think it will be common enough to worry about. Think of os.fspath() > like next() where it truly is a very minor convenience function that > happens to special-case str and bytes. Okay, I'm satisfied by the various arguments against this idea. [...] > > I think it is a bit confusing to refer to "path objects", as that seems > > like you are referring only to pathlib.Path objects. It took me far too > > long to realise that here you mean generic path-like objects that obey > > the __fspath__ protocol rather than a specific concrete class. > > > > Since the ABC is called "PathLike", I suggest we refer to "path-like > > objects" rather than "path objects", both in the PEP and in the Python > > docs for this protocol. > > > > I went back and forth with this in my head while writing the PEP. The > problem with making "path-like" mean "objects implementing the PathLike > ABC" becomes how do you refer to an argument of a function that accepts > anything os.fspath() does (i.e. PathLike, str, and bytes)? On further reflection, I think the right language is to use "path-like" for the union of Pathlike, str and bytes. That will, I think, cover the majority of cases: most functions which want to work on a file system path should accept all three. When you want to specify only an object which implements the PathLike ABC, that's called a (virtual) instance of PathLike. I see this as analogous to "iterable". An iterable is anything which can be iterated over. Sometimes that's an iterator. Sometimes its not an iterator. We don't have a special term for "iterable which is not an iterator", mostly because its rare to care about the distinction, but on those rare cases that we do care, we can describe it explicitly. In this case, "path-like" would be equivalent to iterable: anything that can be used as a file system path. Some path-like objects are actual pathlib.Path objects, some are some other PathLike object, and some are strings or bytes. [...] > > > try: > > > return path.__fspath__() > > > > Would I be right in saying that in practice this will actually end up > > being type(path).__fspath__() to match the behaviour of all(?) other > > dunder methods? > > > > I wasn't planning on it because for most types the accessing of the method > directly off of the type for magic methods is because of some special > struct field at the C level that we're pulling from. Since we're not > planning to have an equivalent struct field I don't see any need to do the > extra work of avoiding the instance participating in method lookup. > Obviously if people disagree for some reason then please let me know (maybe > for perf by avoiding the overhead of checking for the method on the > instance?). The reasons I would disagree are: (1) It took me a long time to learn the rule that dunder methods are always called from the class, not the instance, and now I have to learn that there are exceptions? G argggh. (2) If we ever do change to a C struct field, the behaviour will change. Maybe it's better to emulate the same behaviour from the start? (3) If there's a performance speed up, that's a bonus! -- Steven ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, 13 May 2016 at 10:53 Steven D'Aprano wrote: > On Fri, May 13, 2016 at 03:43:29PM +, Brett Cannon wrote: > > On Fri, 13 May 2016 at 04:00 Steven D'Aprano > wrote: > > [...] > > > - but os.fspath() will only return str; > > > - and os.fspathb() will only return bytes; > > > I prefer what's in the PEP. I get where you coming from, Steven, but I > > don't think it will be common enough to worry about. Think of os.fspath() > > like next() where it truly is a very minor convenience function that > > happens to special-case str and bytes. > > Okay, I'm satisfied by the various arguments against this idea. > I'll add a note in the Rejected Ideas section about this. > > > [...] > > > I think it is a bit confusing to refer to "path objects", as that seems > > > like you are referring only to pathlib.Path objects. It took me far too > > > long to realise that here you mean generic path-like objects that obey > > > the __fspath__ protocol rather than a specific concrete class. > > > > > > Since the ABC is called "PathLike", I suggest we refer to "path-like > > > objects" rather than "path objects", both in the PEP and in the Python > > > docs for this protocol. > > > > > > > I went back and forth with this in my head while writing the PEP. The > > problem with making "path-like" mean "objects implementing the PathLike > > ABC" becomes how do you refer to an argument of a function that accepts > > anything os.fspath() does (i.e. PathLike, str, and bytes)? > > On further reflection, I think the right language is to use "path-like" > for the union of Pathlike, str and bytes. That will, I think, cover the > majority of cases: most functions which want to work on a file system > path should accept all three. When you want to specify only an object > which implements the PathLike ABC, that's called a (virtual) instance of > PathLike. > > I see this as analogous to "iterable". An iterable is anything which can > be iterated over. Sometimes that's an iterator. Sometimes its not an > iterator. We don't have a special term for "iterable which is not an > iterator", mostly because its rare to care about the distinction, but on > those rare cases that we do care, we can describe it explicitly. > > In this case, "path-like" would be equivalent to iterable: anything that > can be used as a file system path. Some path-like objects are actual > pathlib.Path objects, some are some other PathLike object, and some are > strings or bytes. > That was my general thinking. I'll add a TODO to add an entry in the glossary for "path-like". > > > [...] > > > > try: > > > > return path.__fspath__() > > > > > > Would I be right in saying that in practice this will actually end up > > > being type(path).__fspath__() to match the behaviour of all(?) other > > > dunder methods? > > > > > > > I wasn't planning on it because for most types the accessing of the > method > > directly off of the type for magic methods is because of some special > > struct field at the C level that we're pulling from. Since we're not > > planning to have an equivalent struct field I don't see any need to do > the > > extra work of avoiding the instance participating in method lookup. > > Obviously if people disagree for some reason then please let me know > (maybe > > for perf by avoiding the overhead of checking for the method on the > > instance?). > > The reasons I would disagree are: > > (1) It took me a long time to learn the rule that dunder methods are > always called from the class, not the instance, and now I have to learn > that there are exceptions? G argggh. > > (2) If we ever do change to a C struct field, the behaviour will change. > Maybe it's better to emulate the same behaviour from the start? > > (3) If there's a performance speed up, that's a bonus! > And Guido agrees with this as well so I'll be updating the PEP shortly after I remember how to do the equivalent from C. :) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] File system path PEP, 3rd draft
Biggest changes since the second draft: 1. Resolve __fspath__() from the type, not the instance (for Guido) 2. Updated the TypeError messages to say "os.PathLike object" instead of "path object" (implicitly for Steven) 3. TODO item to define "path-like" in the glossary (for Steven) 4. Various more things added to Rejected Ideas 5. Added Koos as a co-author (for Koos :) -- PEP: NNN Title: Adding a file system path protocol Version: $Revision$ Last-Modified: $Date$ Author: Brett Cannon , Koos Zevenhoven Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 11-May-2016 Post-History: 11-May-2016, 12-May-2016, 13-May-2016 Abstract This PEP proposes a protocol for classes which represent a file system path to be able to provide a ``str`` or ``bytes`` representation. Changes to Python's standard library are also proposed to utilize this protocol where appropriate to facilitate the use of path objects where historically only ``str`` and/or ``bytes`` file system paths are accepted. The goal is to facilitate the migration of users towards rich path objects while providing an easy way to work with code expecting ``str`` or ``bytes``. Rationale = Historically in Python, file system paths have been represented as strings or bytes. This choice of representation has stemmed from C's own decision to represent file system paths as ``const char *`` [#libc-open]_. While that is a totally serviceable format to use for file system paths, it's not necessarily optimal. At issue is the fact that while all file system paths can be represented as strings or bytes, not all strings or bytes represent a file system path. This can lead to issues where any e.g. string duck-types to a file system path whether it actually represents a path or not. To help elevate the representation of file system paths from their representation as strings and bytes to a richer object representation, the pathlib module [#pathlib]_ was provisionally introduced in Python 3.4 through PEP 428. While considered by some as an improvement over strings and bytes for file system paths, it has suffered from a lack of adoption. Typically the key issue listed for the low adoption rate has been the lack of support in the standard library. This lack of support required users of pathlib to manually convert path objects to strings by calling ``str(path)`` which many found error-prone. One issue in converting path objects to strings comes from the fact that the only generic way to get a string representation of the path was to pass the object to ``str()``. This can pose a problem when done blindly as nearly all Python objects have some string representation whether they are a path or not, e.g. ``str(None)`` will give a result that ``builtins.open()`` [#builtins-open]_ will happily use to create a new file. Exacerbating this whole situation is the ``DirEntry`` object [#os-direntry]_. While path objects have a representation that can be extracted using ``str()``, ``DirEntry`` objects expose a ``path`` attribute instead. Having no common interface between path objects, ``DirEntry``, and any other third-party path library has become an issue. A solution that allows any path-representing object to declare that it is a path and a way to extract a low-level representation that all path objects could support is desired. This PEP then proposes to introduce a new protocol to be followed by objects which represent file system paths. Providing a protocol allows for explicit signaling of what objects represent file system paths as well as a way to extract a lower-level representation that can be used with older APIs which only support strings or bytes. Discussions regarding path objects that led to this PEP can be found in multiple threads on the python-ideas mailing list archive [#python-ideas-archive]_ for the months of March and April 2016 and on the python-dev mailing list archives [#python-dev-archive]_ during April 2016. Proposal This proposal is split into two parts. One part is the proposal of a protocol for objects to declare and provide support for exposing a file system path representation. The other part deals with changes to Python's standard library to support the new protocol. These changes will also lead to the pathlib module dropping its provisional status. Protocol The following abstract base class defines the protocol for an object to be considered a path object:: import abc import typing as t class PathLike(abc.ABC): """Abstract base class for implementing the file system path protocol.""" @abc.abstractmethod def __fspath__(self) -> t.Union[str, bytes]: """Return the file system path representation of the object.""" raise NotImplementedError Objects representing file system paths will implement the ``__fspath__()`` method which will return the ``str`` or ``bytes`` repres
Re: [Python-Dev] File system path PEP, 3rd draft
On May 13, 2016, at 06:37 PM, Brett Cannon wrote: >PEP: NNN >Title: Adding a file system path protocol >Version: $Revision$ >Last-Modified: $Date$ >Author: Brett Cannon , >Koos Zevenhoven >Status: Draft >Type: Standards Track >Content-Type: text/x-rst >Created: 11-May-2016 >Post-History: 11-May-2016, > 12-May-2016, > 13-May-2016 I think this is far enough along that you should self-assign a PEP number and check it into the official repo. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, 3rd draft
On Fri, 13 May 2016 at 12:08 Barry Warsaw wrote: > On May 13, 2016, at 06:37 PM, Brett Cannon wrote: > > >PEP: NNN > >Title: Adding a file system path protocol > >Version: $Revision$ > >Last-Modified: $Date$ > >Author: Brett Cannon , > >Koos Zevenhoven > >Status: Draft > >Type: Standards Track > >Content-Type: text/x-rst > >Created: 11-May-2016 > >Post-History: 11-May-2016, > > 12-May-2016, > > 13-May-2016 > > I think this is far enough along that you should self-assign a PEP number > and > check it into the official repo. > Paraphrasing *someone* from Twitter yesterday, I'm just going to work on GitHub until I absolutely have to check into hg, m'kay? ;) -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, part 2
On Fri, May 13, 2016 at 8:52 PM, Steven D'Aprano wrote: > On Fri, May 13, 2016 at 03:43:29PM +, Brett Cannon wrote: >> On Fri, 13 May 2016 at 04:00 Steven D'Aprano wrote: > > [...] >> > I think it is a bit confusing to refer to "path objects", as that seems >> > like you are referring only to pathlib.Path objects. It took me far too >> > long to realise that here you mean generic path-like objects that obey >> > the __fspath__ protocol rather than a specific concrete class. >> > This terminology is indeed a bit difficult, not least because there are 6 different path classes in pathlib. A couple of months ago, I decided to start to call these pathlib objects and path objects, because I did not know what else to call them without doubling the length. >> > Since the ABC is called "PathLike", I suggest we refer to "path-like >> > objects" rather than "path objects", both in the PEP and in the Python >> > docs for this protocol. >> >> I went back and forth with this in my head while writing the PEP. The >> problem with making "path-like" mean "objects implementing the PathLike >> ABC" becomes how do you refer to an argument of a function that accepts >> anything os.fspath() does (i.e. PathLike, str, and bytes)? > > On further reflection, I think the right language is to use "path-like" > for the union of Pathlike, str and bytes. That will, I think, cover the > majority of cases: most functions which want to work on a file system > path should accept all three. When you want to specify only an object > which implements the PathLike ABC, that's called a (virtual) instance of > PathLike. > As I've told Brett before, I think exactly this reasoning would be a good enough reason not to call the ABC PathLike. [...] >> > Would I be right in saying that in practice this will actually end up >> > being type(path).__fspath__() to match the behaviour of all(?) other >> > dunder methods? >> >> I wasn't planning on it because for most types the accessing of the method >> directly off of the type for magic methods is because of some special >> struct field at the C level that we're pulling from. Since we're not >> planning to have an equivalent struct field I don't see any need to do the >> extra work of avoiding the instance participating in method lookup. >> Obviously if people disagree for some reason then please let me know (maybe >> for perf by avoiding the overhead of checking for the method on the >> instance?). > > The reasons I would disagree are: > > (1) It took me a long time to learn the rule that dunder methods are > always called from the class, not the instance, and now I have to learn > that there are exceptions? G argggh. Who knows, maybe the exception made it take a longer time.. IIRC the docs say that dunder methods are *not guaranteed* to be called on the instance, because they may be called on the class. > (2) If we ever do change to a C struct field, the behaviour will change. > Maybe it's better to emulate the same behaviour from the start? > > (3) If there's a performance speed up, that's a bonus! > > > > -- > Steven > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding Type[C] to PEP 484
I've got another proposed update to PEP 484 ready. There was previously
some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the
typing issue tracker: https://github.com/python/typing/issues/107
I've worked all that into a patch for PEP 484:
https://github.com/python/typing/pull/218
Once we're okay with that I will work on getting the implementation ready
in time for Python 3.5.2.
Here's the added text:
Meta-types
--
A meta-type can be used to indicate a value that is itself a class
object that is a subclass of a given class. It is spelled as
``Type[C]`` where ``C`` is a class. To clarify: while ``C`` (when
used as an annotation) refers to instances of class ``C``, ``Type[C]``
refers to *subclasses* of ``C``. (This is a similar distinction as
between ``object`` and ``type``.)
For example, suppose we have the following classes::
class User: ... # Abstract base for User classes
class BasicUser(User): ...
class ProUser(User): ...
class TeamUser(User): ...
And suppose we have a function that creates an instance of one of
these classes if you pass it a class object::
def new_user(user_class):
user = user_class()
# (Here we could write the user object to a database)
return user
Without ``Type[]`` the best we could do to annotate ``new_user()``
would be::
def new_user(user_class: type) -> User:
...
However using ``Type[]`` and a type variable with an upper bound we
can do much better::
U = TypeVar('U', bound=User)
def new_user(user_class: Type[U]) -> U:
...
Now when we call ``new_user()`` with a specific subclass of ``User`` a
type checker will infer the correct type of the result::
joe = new_user(BasicUser) # Inferred type is BasicUser
At runtime the value corresponding to ``Type[C]`` must be an actual
class object that's a subtype of ``C``, not a special form. IOW, in
the above example calling e.g. ``new_user(Union[BasicUser, ProUser])``
is not allowed.
There are some concerns with this feature: for example when
``new_user()`` calls ``user_class()`` this implies that all subclasses
of ``User`` must support this in their constructor signature. However
this is not unique to ``Type[]``: class methods have similar concerns.
A type checker ought to flag violations of such assumptions, but by
default constructor calls that match the constructor signature in the
indicated base class (``User`` in the example above) should be
allowed. A program containing a complex or extensible class hierarchy
might also handle this by using a factory class method.
Plain ``Type`` without brackets is equivalent to using ``type`` (the
root of Python's metaclass hierarchy). This equivalence also
motivates the name, ``Type``, as opposed to alternatives like
``Class`` or ``SubType``, which were proposed while this feature was
under discussion; this is similar to the relationship between
e.g. ``List`` and ``list``.
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Adding Type[C] to PEP 484
On Fri, 13 May 2016 at 12:38 Guido van Rossum wrote: > I've got another proposed update to PEP 484 ready. There was previously > some discussion on python-ideas (Bcc'ed for closure) and quite a bit on the > typing issue tracker: https://github.com/python/typing/issues/107 > > I've worked all that into a patch for PEP 484: > https://github.com/python/typing/pull/218 > > Once we're okay with that I will work on getting the implementation ready > in time for Python 3.5.2. > +1 from me. -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, 3rd draft
Can't wait until the peps repo is on GitHub! On Fri, May 13, 2016 at 12:24 PM, Brett Cannon wrote: > > > On Fri, 13 May 2016 at 12:08 Barry Warsaw wrote: > >> On May 13, 2016, at 06:37 PM, Brett Cannon wrote: >> >> >PEP: NNN >> >Title: Adding a file system path protocol >> >Version: $Revision$ >> >Last-Modified: $Date$ >> >Author: Brett Cannon , >> >Koos Zevenhoven >> >Status: Draft >> >Type: Standards Track >> >Content-Type: text/x-rst >> >Created: 11-May-2016 >> >Post-History: 11-May-2016, >> > 12-May-2016, >> > 13-May-2016 >> >> I think this is far enough along that you should self-assign a PEP number >> and >> check it into the official repo. >> > > Paraphrasing *someone* from Twitter yesterday, I'm just going to work on > GitHub until I absolutely have to check into hg, m'kay? ;) > > -Brett > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] quick GitHub transition update (was: File system path PEP, 3rd draft)
On Fri, 13 May 2016 at 12:56 Guido van Rossum wrote: > Can't wait until the peps repo is on GitHub! > With devinabox now moved over to GitHub, the peps repo is next on the list to move (and the devguide after that). I don't know if I'm going to be able to make the migration happen before PyCon US, but my goal is to have it either done before the language summit or have every little step lined up by the end of the sprints and then finished up the week following. I got one of the coveted 20 minute timeslots at the language summit to give an update on everything and soliciting feedback on what needs to be in place to migrate the cpython repo so I can have a clear idea of what work is left to do. If people want to prep for this discussion (which will be reported back here so everyone can participate in the end), think about what you *absolutely* must have in place to make working with GitHub for the cpython repo work for you. I don't want any emails on this now (I've got 3 PEPs at various stages plus travel between now and PyCon at this point so I'm busy enough as it is, so any emails discussing this now *will be ignored* as I don't have the mental bandwidth for that discussion prior to PyCon US). Please just reflect on the required parts of your workflow over the next 2 weeks to know how much work you are going to ask of *me* as minimum requirements to make the GitHub migration of the cpython repo happen (i.e. I don't care about "nice to have", only about "without this I flat-out won't contribute to Python on GitHub"). > On Fri, May 13, 2016 at 12:24 PM, Brett Cannon wrote: > >> >> >> On Fri, 13 May 2016 at 12:08 Barry Warsaw wrote: >> >>> On May 13, 2016, at 06:37 PM, Brett Cannon wrote: >>> >>> >PEP: NNN >>> >Title: Adding a file system path protocol >>> >Version: $Revision$ >>> >Last-Modified: $Date$ >>> >Author: Brett Cannon , >>> >Koos Zevenhoven >>> >Status: Draft >>> >Type: Standards Track >>> >Content-Type: text/x-rst >>> >Created: 11-May-2016 >>> >Post-History: 11-May-2016, >>> > 12-May-2016, >>> > 13-May-2016 >>> >>> I think this is far enough along that you should self-assign a PEP >>> number and >>> check it into the official repo. >>> >> >> Paraphrasing *someone* from Twitter yesterday, I'm just going to work on >> GitHub until I absolutely have to check into hg, m'kay? ;) >> >> -Brett >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, 3rd draft
> On May 13, 2016, at 11:37 AM, Brett Cannon wrote:
>
> Biggest changes since the second draft:
> Resolve __fspath__() from the type, not the instance (for Guido)
> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
> path,
> NULL);
_PyObject_LookupSpecial would be preferable then.
--
Philip Jenvey___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, 3rd draft
On Fri, 13 May 2016 at 14:30 Philip Jenvey wrote:
>
> On May 13, 2016, at 11:37 AM, Brett Cannon wrote:
>
> Biggest changes since the second draft:
>
>1. Resolve __fspath__() from the type, not the instance (for Guido)
>
>
> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
> path,
> NULL);
>
>
> _PyObject_LookupSpecial would be preferable then.
>
Yes it would be. :) I'll add it to the PEP.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] File system path PEP, 3rd draft
FYI, I recently sent a couple of emails in my earlier type hinting
thread on -ideas. What I wrote now is about the path ABC regarding
type hints.
-- Koos
On Sat, May 14, 2016 at 12:48 AM, Brett Cannon wrote:
>
>
> On Fri, 13 May 2016 at 14:30 Philip Jenvey wrote:
>>
>>
>> On May 13, 2016, at 11:37 AM, Brett Cannon wrote:
>>
>> Biggest changes since the second draft:
>>
>> Resolve __fspath__() from the type, not the instance (for Guido)
>>
>>
>> if (PyObject_HasAttrString(path->ob_type, "__fspath__")) {
>> return PyObject_CallMethodObjArgs(path->ob_type, "__fspath__",
>> path,
>> NULL);
>>
>>
>> _PyObject_LookupSpecial would be preferable then.
>
>
> Yes it would be. :) I'll add it to the PEP.
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/k7hoven%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] First 3.6.0 alpha release coming up: 2016-05-16 12:00 UTC
Hi everyone! Thank you for all of your work so far on our next feature release cycle, Python 3.6. Code checkins for 3.6 began 12 months ago, after the feature code cutoff and beta phase for Python 3.5.0. We are now about to do 3.6.0 alpha 1, our first of 4 alpha and 4 beta releases planned for the next 6 months. As a reminder, alpha releases are intended to make it easier for the wider community to test the current state of new features and bug fixes for an upcoming Python release as a whole and for us to test the release process. The start of the alpha phase does *not* mean feature development is complete. During the alpha phase, features may be added, modified, or deleted up until the start of the beta phase. Alpha users beware! 3.6.0 beta 1, currently scheduled for 2016-09-07, marks feature code cutoff, at which time new feature development is expected to be complete and no additional features are to be added to 3.6. At the same time, feature development will begin for the following release cycle, 3.7. During the 3.6.0 beta phase, which is planned to end on 2016-12-04 with the first final release candidate, the focus will be on bug fixes and regressions and preparing for the final release. The goal is for the first release candidate to be the same as the final release: that is, no changes will be accepted for 3.6.0 after release candidate 1 without discussion, review, and release manager approval. Please plan accordingly! Although I've been involved behind the scenes in most releases over the past several years, 3.6.0a1 will be my first as a release manager so no doubt there will be bumps along the way. Many thanks to my fellow release managers, particularly Benjamin, Georg, Larry, and Barry, for their help and support. Your understanding, help, and feedback will be greatly appreciated! For this first alpha release at least, the release team (primarily Steve Dower and myself) are going to try a slightly different timetable for producing the release. We are planning for the code cutoff for alpha 1 to be on 2016-05-16 around 12:00 UTC; that allows for code checkins to continue through Sunday almost anywhere on the planet. We will then go off and manufacture the release bits and make them available via the python.org website as soon as possible thereafter, assuming no major glitches, that should be no later than 24 hours thereafter. In the remaining hours until the code cutoff, please get your commits in as soon as possible and please pay attention to the buildbots for possible regressions. Looking ahead, the next alpha release, 3.6.0a2, will follow in about a month on 2016-06-12, which is a week after the end of the development sprints at US PyCon 2016 in Portland, Oregon. Hope to see you there! To summarize: 2016-05-16 ~12:00 UTC: code snapshot for 3.6.0 alpha 1 now to 2016-09-07: Alpha phase (unrestricted feature development) 2016-09-07: 3.6.0 feature code freeze, 3.7.0 feature development begins 2016-09-07 to 2016-12-04: 3.6.0 beta phase (bug and regression fixes, no new features) 2016-12-04 3.6.0 release candidate 1 (3.6.0 code freeze) 2016-12-16 3.6.0 release (3.6.0rc1 plus, if necessary, any dire emergency fixes) Let me know if you have questions or comments. And thanks again for all of your work towards making Python even better! --Ned D AKA The Other Ned https://www.python.org/dev/peps/pep-0494/ https://docs.python.org/devguide/devcycle.html#stages https://en.wikipedia.org/wiki/Software_release_life_cycle#Alpha https://docs.python.org/devguide/buildbots.html -- Ned Deily n...@python.org -- [] Freenode IRC: ned_deily, #python-dev ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
Paul Moore wrote: On 13 May 2016 at 17:57, Ethan Furman wrote: 1) What is a wallet garden? I assumed he meant "walled garden" Works either way -- you'd want a wall around your wallet garden to stop people stealing your wallets. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] file system path protocol PEP
Chris Angelico writes:
> AFAICT, the compatibility layer would simply decode the bytes using
> surrogateescape handling, which should round-trip anything.
By design. See PEP 383. Or rather, the OP should; he has not done
his homework and is confused by his own FUD. This whole subthread is
really python-list territory.
Since a lot of people I respect seem uncertain about the facts, for
the record, let's lay out the (putative) issues remaining for
post-PEP-383 Python vs. str-y path objects.
(0) "Can't work with some POSIX (bytes) paths" is closed by PEP
383, forget it. os.fsdecode(bytespath) as soon as you get one,
os.fsencode(strpath) just before you need one, done. Surrogates
embedded in strpath may need special handling depending on the
application (see (1)).
(1) str.encode(errors='strict') (the default) will blow up on embedded
surrogates. Yes, but that's a *good* thing if you're mixing str
derived from filesystem paths with other text. There's no way to
avoid it. If you're just passing it back to open(), it Just
Works, done.
(2) You're using bytes as text a la 2.x for "efficiency's" sake, and
you're worried that you might pass a str-y Path deep into bytes
territory and it will explode there.
I don't think there is any sympathy left for that use case on
Python dev channels. Define a clear boundary with well-defined
entry and exit gates, and convert there. Then you can get some
sleep. (How-to example: your "compatibility layer".)
(3) You're worried about inefficiency of decoding/encoding to the same
or trivially changed bytes (ie, you didn't need pathlib in the
first place, but you got it anyway) -- this especially matters for
2.7, but is significant for 3.x too, if you're using a bunch of
paths in a tight loop.
I don't have sympathy for that use case, but Brett and Guido do,
and Brett's PEP handles it by making __fspath__ polymorphic in the
usual os.path-y way, with Guido's modification.
This is always a tradeoff. If you know your JPEGs all have
extension '.JPG' and
png_path = jpeg_path[:-4] + b'.png'
is readable enough for you, use that, not pathlib or Antipathy,
and you get your efficiency. (Doing jpeg_path.rindex(b'.') is left
as an exercise for the reader. Part (i): Is it really worth it?)
If you want the readability of a rich path library and the
efficiency of bytes, you *may* have the option of using Ethan's
Antipathy (or whatever).
If you can't use Antipathy, use bytes methods directly, or accept
that it isn't *that* inefficient and use pathlib. At this point,
I think this subcase is just FUD, no real examples were presented
where the efficiency hit of encoding/decoding gets in the way of
getting work done using pathlib.
If you need to stick to stdlib for some reason (eg, to use a
higher-level library that uses pathlib), live with the
"compatibility layer"'s inefficiency. Decoding and encoding are
actually rather low-cost operations at path lengths (PATHMAX=256
was common, not so long ago!). Most high-level libraries will
impose a lot more overhead elsewhere, and calling into pathlib by
itself will add a certain amount of overhead as well.
(4) Lack of transparency/readability for "simple" operations. If
Antipathy is something you can use, I agree it's plausible that
avoiding a few os.fsdecode and os.fsencode calls would look nicer,
but this is really a style question.
My take: I think of paths as human-readable, so presenting them as
str (not bytes) is important to me, important enough that I
advocate that position to other developers. If you do the
conversion at the boundary between a bytes-y module and pathlib
("compatibility layer") I don't see how it affects readability of
the path manipulation code, while data marshaling at boundaries is
a expected fact of software development. YMMV.
(0) is thus a non-issue. (1) is not something that can be addressed
by general principles, let alone language design. (2)-(4) are all
real issues regardless of how I feel they should be resolved :-), but
they're all design trade-offs, not things that can completely block
you from getting some kinds of work done in your own style (eg, the
situation str-minded people were in before PEP 383).
Python 3 is an example of how language design can help alleviate
issues like (2), by discouraging that use case in various ways.
Brett's PEP is an example of how language design can help alleviate
issues like (3) and (4). In particular, it helps us to interface
pathlib to open() and friends in a very natural, readable way, without
explicit conversions that should be unnecessary by the nature of the
operation and its arguments. By contrast, the conversion of bytes to
str is important to do explicitly because they are different
representations of the same thing, and it's import
