[Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

os.scandir() function -- a better and faster directory iterator Version: $Revision$ Last-Modified: $Date$ Author: Ben Hoyt Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-May-2014 Python-Version: 3.5 Abstract This PEP proposes including a new directory iteratio

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

I don't mind iterdir() and would take it :-), but I'll just say why I chose the name scandir() -- though it wasn't my suggestion originally: iterdir() sounds like just an iterator version of listdir(), kinda like keys() and iterkeys() in Python 2. Whereas in actual fact the return values are quite

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> os.listdir() when I worked on "os" module for MicroPython. I essentially > did what your PEP suggests - introduced internal generator function > (ilistdir_ex() in > https://github.com/micropython/micropython-lib/blob/master/os/os/__init__.py#L85 > ), in terms of which both os.listdir() and os.wal

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

>> But the underlying system calls -- ``FindFirstFile`` / >> ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- > > What about FreeBSD, OpenBSD, NetBSD, Solaris, etc. They don't provide readdir? I guess it'd be better to say "Windows" and "Unix-based OSs" throughout the PEP? Because

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Re is_dir etc being properties rather than methods: >> I find this behaviour a bit misleading: using methods and have them >> return cached results. How much (implementation and/or performance >> and/or memory) overhead would incur by using property-like access here? >> I think this would underlin

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> So, here's my alternative proposal: add an "ensure_lstat" flag to > scandir() itself, and don't have *any* methods on DirEntry, only > attributes. > > That would make the DirEntry attributes: > > is_dir: boolean, always populated > is_file: boolean, always populated > is_symlink boole

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> I suppose the exact behavior is still under discussion, as there are only > two or three fields one gets "for free" on Windows (I think...), where as an > os.stat call would get everything available for the platform. No, Windows is nice enough to give you all the same stat_result fields during s

Re: [Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)

Thanks, Victor. I don't have any experience with dir_fd handling, so unfortunately can't really comment here. What advantages does it bring? I notice that even os.listdir() on Python 3.4 doesn't have anything related to file descriptors, so I'd be in favour of not including support. We can always

Re: [Python-Dev] My summary of the scandir (PEP 471)

Thanks for spinning this off to (hopefully) finished the discussion. I agree it's nearly time to update the PEP. > @Ben: it's time to update your PEP to complete it with this > discussion! IMO DirEntry must be as simple as possible and portable: > > - os.scandir(str) > - DirEntry.lstat_result obje

Re: [Python-Dev] My summary of the scandir (PEP 471)

> No need for a microsecond-timed deletion -- a directory with +r but > without +x will allow you to list the entries, but stat calls on the > files will fail with EPERM: Ah -- very good to know, thanks. This definitely points me in the direction of wanting better control over error handling. Spe

Re: [Python-Dev] My summary of the scandir (PEP 471)

> We need per-iteration error handling for the readdir call anyway, so I think > an onerror callback is a better option than dropping the ability to easily > obtain full stat information as part of the iteration. I don't mind the idea of an "onerror" callback, but it's adding complexity. Putting a

Re: [Python-Dev] My summary of the scandir (PEP 471)

Thanks for the effort in your response, Paul. I'm all for KISS, but let's just slow down a bit here. > I think that thin wrapper is needed - even > if the various bells and whistles are useful, they can be built on top > of a low-level version (whereas the converse is not the case). Yes, but API

Re: [Python-Dev] My summary of the scandir (PEP 471)

Thanks for the clarifications and support. > Ah, the wording in the PEP says "Linux, Windows, OS X". Superficially, > that said "everywhere" to me. It might be worth calling out > specifically some examples where it's not available without an extra > system call, just to make the point explicit.

[Python-Dev] Updates to PEP 471, the os.scandir() proposal

in the PEP as well as the relevant python-dev discussion (linked to in the PEP). Thanks, Ben PEP: 471 Title: os.scandir() function -- a better and faster directory iterator Version: $Revision$ Last-Modified: $Date$ Author: Ben Hoyt Status: Draft Type: Standards Track Content-Type: text/x-rst Creat

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> I remember a pending question on python-dev: > > - Martin von Loewis asked if the scandir generator would have send() > and close() methods as any Python generator. I didn't see a reply on > the mailing (nor in the PEP). Good call. Looks like you're referring to this message: https://mail.python

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Only exposing what the OS provides for free will make the API too difficult > to use in the common case. But is there a nice way to expand the API that > will allow the user who is trying to avoid extra expense know what > information is already available? > > Even if the initial version doesn't

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Better to just have the attributes be None if they were not fetched. None > is better than hasattr anyway, at least in the respect of not having to > catch exceptions to function properly. The thing is, is_dir() and lstat() are not attributes (for a good reason). Please read the relevant "Rejec

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> I think you're misunderstanding is_dir() and is_file(), as these don't >> actually call os.stat(). All DirEntry methods either call nothing or >> os.lstat() to get the stat info on the entry itself (not the >> destination of the symlink). > > > Oh. Extract of your PEP: "is_dir(): like os.path.is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> I did better than that -- I read the whole thing! ;) Thanks. :-) > -1 on the PEP's implementation. > > Just like an attribute does not imply a system call, having a > method named 'is_dir' /does/ imply a system call, and not > having one can be just as misleading. Why does a method imply a sy

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> In this case because the names are exactly the same as the os versions which > /do/ make a system call. Fair enough. > So if I'm finally understanding the root problem here: > > - listdir returns a list of strings, one for each filename and one for > each directory, and keeps no other O/S

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Ok, so it means that your example grouping files per type, files and > directories, is also wrong. Or at least, it behaves differently than > os.walk(). You should put symbolic links to directories in the "dirs" > list too. > > if entry.is_dir(): # is_dir() checks os.lstat() > dirs.append(e

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Option 2: > def log_err(exc): > logger.warn("Cannot stat {}".format(exc.filename)) > > def get_tree_size(path): > total = 0 > for entry in os.scandir(path, info='lstat', onerror=log_err): > if entry.is_dir: > total += get_tree_size(entry.full_name) > else:

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> The PEP says that DirEntry should mimic pathlib.Path, so I think that >> DirEntry.is_dir() should work as os.path.isir(): if the entry is a >> symbolic link, you should follow the symlink to get the status of the >> linked file with os.stat(). > > Would this not "break" the tree size script bein

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> One issue with option #2 that I just realized -- does scandir yield the >> entry at all if there's a stat error? It >> can't really, because the caller will expect the .lstat attribute to be >> set (assuming he asked for type='lstat') but >> >> it won't be. Is effectively removing these entries

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

This is just getting way too complex ... further thoughts below. >> This is an interesting idea, but it's just getting more and more >> complex, and I'm guessing that being able to change the attributes of >> DirEntry will make the C implementation more complex. >> >> Also, I'm not sure it's very

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> 1) it has an additional "info" argument, the values of which have to >> be documented ('os', 'type', 'lstat', and what each one means) >> 2) it has an additional "onerror" argument, the signature of which and >> fairly complicated return value is non-obvious and has to be >> documented >> 3) it

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> On a system which did not supply is_dir automatically I would write that as: > > for entry in os.scandir(path): # info defaults to 'os', which is > basically None in this case > if ignore_entry(entry.name): > continue > if os.path.isdir(entry.full_name): > # do

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

I really don't understand why you *want* a worse, much less cross-platform API? > Okay, so using that logic we should head over to the os module and remove: > > ctermid, getenv, getegid... > > Okay, I'm tired of typing, but that list is not even half-way through the os > page, and those are all me

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> DirEntry methods will remain free (no syscall) for directories and >> regular files. One extra syscall will be needed only for symlinks, >> which are more rare than other file types (for example, you wrote " >> Windows typically makes little use of symlinks"). > > The info we want for scandir is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[replying to python-dev this time] >> The "onerror" approach can also deal with readdir failing, which the >> PEP currently glosses over. > > > Do we want this, though? I can see an error handler for individual entries, > but if one of the *dir commands fails that would seem to be fairly > catas

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> Very much agreed that this isn't necessary for just readdir/FindNext >> errors. We've never had this level of detail before -- if listdir() >> fails half way through (very unlikely) it just bombs with OSError and >> you get no entries at all. >> >> If you really really want this (again very unli

[Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

Hi folks, Thanks Victor, Nick, Ethan, and others for continued discussion on the scandir PEP 471 (most recent thread starts at https://mail.python.org/pipermail/python-dev/2014-July/135377.html). Just an aside ... I was reminded again recently why scandir() matters: a scandir user emailed me the

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

First, just to clarify a couple of points. > You forgot one of my argument: we must have exactly the same API than > os.path.is_dir() and pathlib.Path.is_dir(), because it would be very > confusing (source of bugs) to have a different behaviour. Actually, I specifically included that argument. It

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Let's not multiply entities beyond necessity. > > There is well-defined *follow_symlinks* parameter > https://docs.python.org/3/library/os.html#follow-symlinks > e.g., os.access, os.chown, os.link, os.stat, os.utime and many other > functions in os module support follow_symlinks parameter, see >

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Looks doable. Just make sure the cached entries reflect the > 'follow_symlinks' setting -- so a symlink could end up with both an lstat > cached entry and a stat cached entry. Yes, good point -- basically the functions will use the _stat cache if follow_symlinks=True, otherwise the _lstat cache

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Sorry, I don't remember who but someone proposed to add the follow_symlinks > parameter in scandir() directly. If the parameter is added to methods, > there is no such issue. Yeah, I think having the DirEntry methods do different things depending on how scandir() was called is a really bad idea

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> I'd *keep DirEntry.lstat() method* regardless of existence of > .stat(*, follow_symlinks=True) method (despite the slight violation of > DRY principle) for readability. `dir_entry.lstat().st_mode` is more > consice than `dir_entry.stat(follow_symlinks=False).st_mode` and the > meaning of lstat is

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Have you tried modifying importlib's _bootstrap.py to use scandir() instead > of listdir() + stat()? No, I haven't -- I'm not familiar with that code. What does _bootstrap.py do -- does it do a lot of listdir calls and stat-ing of many files? -Ben ___

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Even though there is tangible performance improvement from scandir(), it > would be useful to find out if the API fits well. Got it -- I see where you're coming from now. I'll take a quick look (hopefully later this week). -Ben ___ Python-Dev mailing

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Maybe it is an abuse of the API. A clear_cache() method would be less > ugly :-) But maybe Ben Hoyt does not want to promote keeping DirEntry > for a long time? > > Another question: should we expose DirEntry type directly in the os > namespace? (os.DirEntry) Again, I'

Re: [Python-Dev] PEP 471 "scandir" accepted

> I asked privately Guido van Rossum if I can be the BDFL-delegate for > the PEP 471 and he agreed. I accept the latest version of the PEP: > > http://legacy.python.org/dev/peps/pep-0471/ Thank you! > The PEP also explicitly mentions that os.walk() will be modified to > benefit of the new os.

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> We should mimic os.stat() and os.stat_result: os.stat_result symbol > exists in the os namespace, but the type constructor is not > documented. No need for extra protection like not adding the type in > the os module, or adding a "_" prefix to the name. Yeah, that works for me. > By the way, it

Re: [Python-Dev] PEP 471 "scandir" accepted

> Note: listdir() accepts an integer path (an open file descriptor that > refers to a directory) that is passed to fdopendir() on POSIX [4] i.e., > *you can't use scandir() to replace listdir() in this case* (as I've > already mentioned in [1]). See the corresponding tests from [2]. > > [1] https:/

Re: [Python-Dev] PEP 471 "scandir" accepted

Makes sense, thanks. -Ben On Tue, Jul 22, 2014 at 4:57 PM, Nick Coghlan wrote: > > On 23 Jul 2014 02:18, "Victor Stinner" wrote: >> >> 2014-07-22 17:52 GMT+02:00 Ben Hoyt : >> > However, given that we have to support this for listdir() anyway, I >&g

[Python-Dev] os.walk() is going to be *fast* with scandir

Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that

Re: [Python-Dev] sum(...) limitation

It seems to me this is something of a pointless discussion -- I highly doubt the current situation is going to change, and it works very well. Even if not perfect, sum() is for numbers, sep.join() for strings. However, I will add one comment: I'm overall -1 on trying to change the current situatio

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

> Victor Stinner suggested [1] to allow scandir(fd) but I don't see it > being mentioned in the pep 471 [2]: it neither supports nor rejects the > idea. > > [1] https://mail.python.org/pipermail/python-dev/2014-July/135283.html > [2] http://legacy.python.org/dev/peps/pep-0471/ Yes, listdir() suppo

Re: [Python-Dev] Multiline 'with' statement line continuation

> Even if it weren't a syntax error, the syntax would be ambiguous. How > will you discern the meaning of:: > > with ( > foo, > bar, > baz): > pass > > Is that three separate context managers? Or is it one tuple with three > items? Is it meaningful t

Re: [Python-Dev] Documenting enum types

> The enemy must be documented and exported, since users will encounter them. enum == enemy? Is that you, Raymond? ;-) -Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.

Re: [Python-Dev] Bytes path support

> The official policy is that we want them [support for bytes paths in stdlib > functions] to go away, but reality so far has not budged. We will continue to > hold our breath though. :-) Does that mean that new APIs should explicitly not support bytes? I'm thinking of os.scandir() (PEP 471), wh

Re: [Python-Dev] Bytes path support

>> > The official policy is that we want them [support for bytes paths in >> > stdlib functions] to go away, but reality so far has not budged. We will >> > continue to hold our breath though. :-) >> >> Does that mean that new APIs should explicitly not support bytes? I'm >> thinking of os.scandi

Re: [Python-Dev] Bytes path support

>> If scandir is low-level, and the low-level API's are the ones that should >> support bytes paths, then scandir should support bytes paths. >> >> Is that what you meant to say? > > Yes. The discussions around PEP 471 *deferred* discussions of bytes > and file descriptor support to their own RFEs

[Python-Dev] Adding numbering to PEP 20, the Zen of Python

I was emailing someone today about implementing something (for PEP 471, as it happens) and wanted to link to the Zen of Python [1] and note a particular clause (in this case "If the implementation is hard to explain, it's a bad idea."). However, there are no clause numbers, so you can't refer to sp

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

> > * C implementation: scandir is at least 3.5x faster than listdir, up > > to 44.6x faster on Windows > > * C+Python implementation: scandir is not really faster than listdir, > > between 1.3x and 1.4x faster > > So amusingly, the bottleneck is not so much the cost of system calls, > but the cost

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

> TL;DR: on POSIX, is it useful to know the inode number (st_ino) > without the device number (st_dev)? I can't answer this question (not being a Linux dev and not knowing much about this), but I'm +1 for adding DirEntry.inode(). On Windows, we're exposing all the information FindFirst/FindNext g

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

> Since there are many ways to split this huge file, I agree that it's > just fine to add these 800 lines and *then* think how the huge file > can be splitted. It's a different topic. That's a good idea. Consider adding the new feature (scandir) and the larger issue of refactoring posixmodule as s

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

> +1 we need to provide the inode (we shouldn't be throwing anything from the > underlying directory entry away when possible). But... > > I think the "or None" semantics are a bad idea. It'd be better for this to > raise AttributeError on Windows so that someone can't write the most natural > for

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

>> +1 for inode support. I agree with the above -- it should either raise >> AttributeError on Windows if it's not going to be set ... or it should >> be more like Victor's original proposal where .inode() is a method >> that calls stat on Windows. I don't have strong feelings. > > The whole point

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

>> The whole point of scandir is to expose low-level system calls in a >> cross-platform way. > > Cross-platform is great and preferable, but low-level system facilities > should be made available even when they are unique to a particular OS. Yes, but this can be made cross-platform fairly easily,

[Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on

When merging some changes while working on scandir, I noticed a minor issue with this commit: https://hg.python.org/cpython/rev/4f6f4aa0d80f The definition of "struct win32_stat" has been moved to fileutils.h and renamed to "struct _Py_stat_struct", which is fine -- however, the old "struct win32

Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

part should be fairly straight-forward, as I already have a version available in my GitHub project. -Ben On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner wrote: > Hi, > > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5

Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

Hi Ryan, > ./configure --with-pydebug && make -j7 > > I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got: > > Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree: depth=4, > num_dirs=5, num_files=50 > Using slower ctypes version of scandir > Comparing against

Re: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available

I'm seeing the same issue (though I also get the missing-DLL error dialog when I run python.exe). -Ben On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore wrote: > On 9 March 2015 at 09:34, Larry Hastings wrote: > > On behalf of the Python development community and the Python 3.5 release > > team, I'm t

Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

> > os.walk took 0.061s, scandir.walk took 0.012s -- 5.2x as fast > Great, looks much better. :-) Even a bit better than what I'm seeing -- possibly due to your SSD. -Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/

Re: [Python-Dev] PEP 8 update

> My own preference would be: > > def foo(x): > if x >= 0: > return math.sqrt(x) > return None Kind of getting into the weeds here, but I would always invert this to "return errors early, and keep the normal flow at the main indentation level". Depends a little on w

[Python-Dev] Scandir module's C code updated to Python 3.5 code

Hi folks, With os.scandir() now in the Python 3.5 stdlib, I just thought I'd let folks know that I've released the scandir module version 1.0. So this is now basically a copy-n-paste of the C code that went into CPython 3.5's posixmodule.c with the necessary changes to make it work or Python 2.x (

Re: [Python-Dev] PEP 559 - built-in noop()

I don't think the rationale justifies an entire builtin. You could just use "PYTHONBREAKPOINT=int" to disable, or support "PYTHONBREAKPOINT=0" as I think someone else suggested. I personally can't remember the last time I needed a noop() function. I've more often needed an identity() function, and

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

I've read the examples you wrote here, but I'm struggling to see what the real-life use cases are for this. When would you care about *both* very long-running servers (104 days+) and nanosecond precision? I'm not saying it could never happen, but would want to see real "experience reports" of when

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

"very stable server"? -Ben On Mon, Oct 16, 2017 at 11:58 AM, Guido van Rossum wrote: > On Mon, Oct 16, 2017 at 8:37 AM, Ben Hoyt wrote: > >> I've read the examples you wrote here, but I'm struggling to see what the >> real-life use cases are for this.

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

Makes sense, thanks. -Ben On Mon, Oct 16, 2017 at 12:28 PM, Victor Stinner wrote: > 2017-10-16 18:14 GMT+02:00 Ben Hoyt : > > Got it -- fair enough. > > > > We deploy so often where I work (a couple of times a week at least) that > 104 > > days seems like an etern

[Python-Dev] Enable access to the AST for Python code

Hi Python devs, Enabling access to the AST for compiled code would make some cool things possible (C# LINQ-style ORMs, for example), and not knowing too much about this part of Python internals, I'm wondering how possible and practical this would be. Context: PonyORM (http://ponyorm.com/) allows

Re: [Python-Dev] Enable access to the AST for Python code

nitely cool though! (Using movie poster > style quotes you can turn this into a ringing endorsement: "definitely > cool" -- The BDFL. :-) > > On Wed, May 20, 2015 at 7:26 PM, Ben Hoyt wrote: > >> Hi Python devs, >> >> Enabling access to the AST for comp

Re: [Python-Dev] Enable access to the AST for Python code

Heh, thanks. :-) On Thu, May 21, 2015 at 2:17 PM, Guido van Rossum wrote: > Dang it. :-) I just want to encourage you to continue pursuing this idea, > one way or another. > > On Thu, May 21, 2015 at 7:01 AM, Ben Hoyt wrote: > >> Thanks. Good point about python-ideas -

Re: [Python-Dev] Enable access to the AST for Python code

ike this is that ideally you don't want to support > this just for CPython. It's definitely cool though! (Using movie poster > style quotes you can turn this into a ringing endorsement: "definitely cool" > -- The BDFL. :-) > > On Wed, May 20, 2015 at 7:26 PM, Be

Re: [Python-Dev] [RELEASED] Python 3.5.0b3 is now available

Thanks! Looking forward to trying this. I'm not sure where these descriptions come from, or whether they're carried over from b2 to b3 etc, but one small note on this bullet point: * PEP 471, os.scandir(), a faster alternative to os.walk() This isn't quite correct. os.scandir() is actually an al

Re: [Python-Dev] [RELEASED] Python 3.5.1 and 3.4.4rc1 are now available

Great, thank you! Small note, not sure if it's related to the release or not: the downloads menu on python.org seems to be broken. The 2.7 download button is showing for me, but the 3.x download button is kind of kaput. See screenshot: http://i.imgur.com/ji1LCnn.png -Ben On Mon, Dec 7, 2015 at

[Python-Dev] Improving docs for len() of set

Hi folks, Just a suggestion for a documentation tweak. Currently the docs for len() on a set say this: .. describe:: len(s) Return the cardinality of set *s*. I'm a relatively seasoned programmer, but I don't really have a maths background, and I didn't know what "cardinality" meant. I

Re: [Python-Dev] Improving docs for len() of set

Thanks! Commit ref: https://hg.python.org/cpython/rev/a67fda8e33b0 -Ben On Mon, Feb 8, 2016 at 1:00 PM, Gregory P. Smith wrote: > > > On Mon, Feb 8, 2016 at 8:24 AM Ben Hoyt wrote: > >> Hi folks, >> >> Just a suggestion for a documentation tweak. Currently the

[Python-Dev] Scandir module seeking new maintainer

Hi folks, I'm not sure if this is the right place to ask, but seeing I was fairly active here when developing scandir and getting it into Python 3.5, so I thought I'd start here. I'm the author and current maintainer of the scandir module (Python 3.5's os.sc

Re: [Python-Dev] Scandir module seeking new maintainer

> > So I'm looking for someone who wants to take over the maintenance of the >> scandir PyPI module. It wouldn't be much work, as there are only a >> couple of fairly small issues to fix. Or it might be a good way to start >> for someone who wants to contribute to a small but useful open source >>

Re: [Python-Dev] Scandir module seeking new maintainer

> FYI I just finished (I hope!) to bug on os.walk(bytes) on Windows with > the "emulated" scandir (since os.scandir() only works on Unicode on > Python 3.5+): > http://bugs.python.org/issue25911 > > Since I helped you to write your PEP 471 (scandir) as the > BDFL-delegate, I think that I now unders

Re: [Python-Dev] Scandir module seeking new maintainer

FYI to the group: I'll discuss details with Victor privately. -Ben On Tue, Mar 29, 2016 at 8:12 AM, Ben Hoyt wrote: > > >> FYI I just finished (I hope!) to bug on os.walk(bytes) on Windows with >> the "emulated" scandir (since os.scandir() only works on Un

Re: [Python-Dev] Scandir module seeking new maintainer

, Ben Hoyt wrote: > FYI to the group: I'll discuss details with Victor privately. -Ben > > > On Tue, Mar 29, 2016 at 8:12 AM, Ben Hoyt wrote: > >> >> >>> FYI I just finished (I hope!) to bug on os.walk(bytes) on Windows with >>> the "emulate

[Python-Dev] PEP 428 - pathlib API questions

PEP 428 looks nice. Thanks, Antoine! I have a couple of questions about the module name and API. I think I've read through most of the previous discussion, but may have missed some, so please point me to the right place if there have already been discussions about these things. 1) Someone on redd

[Python-Dev] pathlib and issue 11406 (a directory iterator returning stat-like info)

Hi folks, I decided to start another thread for my thoughts on the interaction between pathlib (Antoine's new PEP 428), issue 11406 (proposal for a directory iterator returning stat-like info), and my own scandir library, which implements something along the lines of issue 11406. My scandir libra

Re: [Python-Dev] PEP 428 - pathlib API questions

> Well, "path" is much too common already, and it's an obvious variable > name for a filesystem path, so "pathlib" is better to avoid name > clashes. Yep, that makes total sense, thanks. >> However, it seems there was no further discussion about why not >> "extension" and "extensions"? I have nev

Re: [Python-Dev] pathlib and issue 11406 (a directory iterator returning stat-like info)

> Right now, pathlib doesn't cache. Guido decided it was safer to start > off like that, and perhaps later we can add some optional caching. > > One reason caching didn't go in is that it's not clear which API is > best. Working on pluggin scandir() into pathlib would actually help > choosing a sta

Re: [Python-Dev] PEP 428 - pathlib API questions

>> However, it seems there was no further discussion about why not >> "extension" and "extensions"? I have never heard a filename extension >> being called a "suffix". > > > You can't have read many unix man pages, then! Huh, no I haven't! Certainly not regularly, as I'm almost exclusively a Windo

Re: [Python-Dev] pathlib and issue 11406 (a directory iterator returning stat-like info)

Antoine's class-global flag seems like a bad idea. > A global string (or Path) keyed cache (rather than a per-object cache) would > actually be a safer option, since it would ensure distinct path objects > always gave the same answer. That's the approach I will likely pursue at > some point in wal

Re: [Python-Dev] pathlib and issue 11406 (a directory iterator returning stat-like info)

> I think we should think hard and deep about all the consequences. I was > initially in favor of stat caching, but during offline review of PEP 428 > Nick pointed out that there are too many different ways to do stat caching, > and convinced me that it would be wrong to rush it. Now that beta 1 is

Re: [Python-Dev] PEP 428 - pathlib API questions

> Using "**" for directory spanning globs is also another case of us borrowing > a reasonably common idiom from *nix systems that may not be familiar to > Windows users. Okay, *nix wins then. :-) Python's stdlib is already fairly *nix-oriented (even when it's being cross-platform), so I guess it's

Re: [Python-Dev] pathlib and issue 11406 (a directory iterator returning stat-like info)

> The idea of the rich stat result object is that has all that info > prepopulated, based on an initial stat call. "Caching" it amounts to "keep a > reference to it". > > It is suggested that it would be a subset of the pathlib.Path API: > http://bugs.python.org/issue19725 > > If it's also a supers

Re: [Python-Dev] pathlib and issue 11406 (a directory iterator returning stat-like info)

> OK, so I'm a Windows dev, but my understanding is that d_ino is useful > to tell if two files are identical - hard links to the same physical > file have the same d_ino value. I don't believe it's possible to do > this on Windows at all. > > I've seen it used in tools like diff, to short-circuit

[Python-Dev] Returning Windows file attribute information via os.stat()

Hi folks, As pointed out to me recently in an issue report [1] on my scandir module, Python's os.stat() simply discards most of the file attribute information fetched via the Win32 system calls. On Windows, os.stat() calls CreateFile to open the file and get the dwFileAttributes value, but it thro

Re: [Python-Dev] Returning Windows file attribute information via os.stat()

> > FILE_ATTRIBUTE_HIDDEN = 2 # constant defined in Windows.h > > > > if hasattr(st, 'st_winattrs') and st.st_winattrs & > > FILE_ATTRIBUTE_HIDDEN: > > I don't like such API, it requires to import constants, use masks, etc. > > I would prefer something like: > >if st.win_hidden: ... > > O

Re: [Python-Dev] Returning Windows file attribute information via os.stat()

>> if hasattr(st, 'st_winattrs') and st.st_winattrs & >> FILE_ATTRIBUTE_HIDDEN: > > That could be written more succinctly as: > > if getattr(st, 'st_winattrs', 0) & FILE_ATTRIBUTE_HIDDEN: > >> return True >> return False Yes, good call. Or one further: return getattr(

Re: [Python-Dev] Returning Windows file attribute information via os.stat()

> The stat module exposes a load of constants - why not add the > (currently known) ones there? Finding the values of Windows constants > if you don't have access to the C headers can be a pain, so having > them defined *somewhere* as named values is useful. So stat.FILE_ATTRIBUTES_HIDDEN and the

Re: [Python-Dev] Returning Windows file attribute information via os.stat()

>> To solve this problem, what do people think about adding an >> "st_winattrs" attribute to the object returned by os.stat() on >> Windows? > > +1 to the idea, whatever the exact implementation. Cool. I think we should add a st_winattrs integer attribute (on Windows) and then also add the FILE_A

Re: [Python-Dev] Returning Windows file attribute information via os.stat()

>> What would be the next steps to get this to happen? Open an issue on >> bugs.python.org and submit a patch with tests? > > Yep! Okay, I've done step one (opened an issue on bugs.python.org), and hope to provide a patch in the next few weeks if no-one else does (I've never compiled CPython on Wi

[Python-Dev] mimetypes broken on Windows

Hi folks, The built-in mimetypes module is broken on Windows, and it has been since Python 2.7 alpha 1. On all Windows systems I've tried, guess_type() returns the wrong mime type for common types like .png and .jpg. For example (on Python 2.7.4 and 3.3.1): >>> import mimetypes >>> mimetypes.gues

  1   2   >