Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

Makes sense, thanks. -Ben On Mon, Oct 16, 2017 at 12:28 PM, Victor Stinner wrote: > 2017-10-16 18:14 GMT+02:00 Ben Hoyt : > > Got it -- fair enough. > > > > We deploy so often where I work (a couple of times a week at least) that > 104 > > days seems like an etern

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

"very stable server"? -Ben On Mon, Oct 16, 2017 at 11:58 AM, Guido van Rossum wrote: > On Mon, Oct 16, 2017 at 8:37 AM, Ben Hoyt wrote: > >> I've read the examples you wrote here, but I'm struggling to see what the >> real-life use cases are for this.

Re: [Python-Dev] PEP 564: Add new time functions with nanosecond resolution

I've read the examples you wrote here, but I'm struggling to see what the real-life use cases are for this. When would you care about *both* very long-running servers (104 days+) and nanosecond precision? I'm not saying it could never happen, but would want to see real "experience reports" of when

Re: [Python-Dev] PEP 559 - built-in noop()

I don't think the rationale justifies an entire builtin. You could just use "PYTHONBREAKPOINT=int" to disable, or support "PYTHONBREAKPOINT=0" as I think someone else suggested. I personally can't remember the last time I needed a noop() function. I've more often needed an identity() function, and

Re: [Python-Dev] PEP 548: More Flexible Loop Control

I think Serhiy's response is excellent and agree with it. My gut reaction is "this looks like Perl" (and not in a good way), but more specifically it makes the control flow almost invisible. So I'm definitely -1 on this. The current while True ... break idiom is not pretty, but it's also very clea

Re: [Python-Dev] Appending a link back to bugs.python.org in GitHub PRs

With the linking back and forth, I'm curious why there wasn't a switch to use GitHub's issue tracker when we switched to GitHub. I'm sure there was previous discussion about this and good reasons not to, but couldn't find those quickly (PEP 512, Google search, etc) -- can someone point me in the ri

Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

er > > > > -Original Message- > From: Python-Dev [mailto:python-dev-bounces+peter.xihong.wang=intel.com@ > python.org] On Behalf Of Nathaniel Smith > Sent: Tuesday, July 18, 2017 7:00 PM > To: Ben Hoyt > Cc: Python-Dev > Subject: Re: [Python-Dev] Program r

Re: [Python-Dev] for...else

This is more of a python-ideas discussion, and Steven's answer is good. I'll just add one thing. Maybe it's obvious to others, but I've liked for...else since I found a kind of mnemonic to help me remember when the "else" part happens: I think of it not as "for ... else" but as "break ... else" --

Re: [Python-Dev] Python startup time

Yes, agreed that startup time matters for scripting. I was talking to someone on the Google Cloud SDK (CLI) team recently, and they said startup time is a big deal for them ... it's especially problematic for shell tab completion helpers, because every time you press tab the shell has to load your

Re: [Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

; On 19 July 2017 at 02:18, Antoine Pitrou wrote: > > On Tue, 18 Jul 2017 12:03:36 -0400 > > Ben Hoyt wrote: > >> The program is a pentomino puzzle solver, and it works via code > generation, > >> generating a ton of nested "if" statements, so I be

[Python-Dev] Program runs in 12s on Python 2.7, but 5s on Python 3.5 -- why so much difference?

Hi folks, (Not entirely sure this is the right place for this question, but hopefully it's of interest to several folks.) A few days ago I posted a note in response to Victor Stinner's articles on his CPython contributions, noting that I wrote a program that ran in 11.7 seconds on Python 2.7, but

Re: [Python-Dev] Articles on my contributions to CPython during 2017 Q1 and Q2

erpreter.) I'll post details in the next week or so. -Ben On Fri, Jul 14, 2017 at 9:33 AM, Victor Stinner wrote: > 2017-07-14 13:37 GMT+02:00 Ben Hoyt : > > Wow, amazing work. The Stinnerbot strikes again! > > Thanks. > > > A lot of great optimizations and b

Re: [Python-Dev] Articles on my contributions to CPython during 2017 Q1 and Q2

Wow, amazing work. The Stinnerbot strikes again! A lot of great optimizations and bugfixes. Speaking of optimizations, I just wrote some code which takes 12s on Python 2.7 and 5s on Python 3.5. so we're doing something right! I might post about it shortly. -Ben On Jul 13, 2017 11:34 AM, "Victor

Re: [Python-Dev] Star the CPython GitHub project if you like Python!

Nice! I also posted it on reddit.com/r/Python, where it got a bit of traction: https://www.reddit.com/r/Python/comments/6kg4w0/cpython_recently_moved_to_github_star_the_project/ -Ben On Tue, Jul 4, 2017 at 9:15 AM, Victor Stinner wrote: > 4 days later, we got +2,389 new stars, thank you! (8,53

Re: [Python-Dev] Micro-optimizations by adding special-case bytecodes?

> I don't want to discourage you, but this is unlikely to produce > significant speedups on real-world code. The simple reason is that > comparing to None is not a bottleneck for any real application -- > comparing to None is probably damn fast compared to everything else > the real application do

Re: [Python-Dev] Micro-optimizations by adding special-case bytecodes?

Thanks for the pointers -- I wasn't aware of perf timeit. I'll try to get some more stable benchmark results. -Ben On Wed, Jun 28, 2017 at 10:20 AM, Victor Stinner wrote: > (Victor wears his benchmark hat.) > > 2017-06-28 15:50 GMT+02:00 Ben Hoyt : >> ../test_none.

Re: [Python-Dev] Micro-optimizations by adding special-case bytecodes?

-+--+ Not significant (15): chaos; go; json_loads; nbody; pickle_pure_python; pidigits; python_startup_no_site; raytrace; scimark_lu; scimark_sor; sympy_expand; sympy_sum; sympy_str; unpickle_pure_python; xml_etree_generate On Thu, May 25, 2017 at 10:28 AM, Ben Hoyt wrote

Re: [Python-Dev] "Micro-optimisations can speed up CPython"

That's very interesting -- thanks for sharing, Serhiy and Antoine. Very relevant! On Mon, May 29, 2017 at 10:15 AM, Serhiy Storchaka wrote: > 29.05.17 15:13, Antoine Pitrou пише: > >> I hope readers won't get bothered by what is mostly a link to blogpost >> (well, two of them :-)), but I suspect

Re: [Python-Dev] Micro-optimizations by adding special-case bytecodes?

ment more optmisations and > reduce the number of instructions. In my experience, less instructions = > faster code. > > http://faster-cpython.readthedocs.io/registervm.html > > Mark's bytecode uses registers but also a stack. > > Victor > > Le 24 mai 2017 8:09 PM, &q

Re: [Python-Dev] Micro-optimizations by adding special-case bytecodes?

hole.c. -Ben On Wed, May 24, 2017 at 4:14 PM, Erik wrote: > Hi Ben, > > On 24/05/17 19:07, Ben Hoyt wrote: > >> I'm not proposing to do this yet, as I'd need to benchmark to see how >> much of a gain (if any) it would amount to, but I'm just wonderi

[Python-Dev] Micro-optimizations by adding special-case bytecodes?

Hi folks, I was looking at some `dis` output today, and I was wondering if anyone has investigated optimizing Python (slightly) by adding special-case bytecodes for common expressions or statements involving constants? For example, I (and, based on a quick grep of the stdlib, many others) write "

Re: [Python-Dev] cgi.FieldStorage with multipart/form-data tries to decode binary file as UTF-8 if "filename=" not specified

hen file_input.read().encode('utf-8', 'surrogateescape') ... pretty hacky, but at least you can get the bytes back. -Ben On Wed, Feb 15, 2017 at 12:35 PM, Brett Cannon wrote: > > > On Wed, 15 Feb 2017 at 08:14 Ben Hoyt wrote: > >> I posted this on StackOverf

[Python-Dev] cgi.FieldStorage with multipart/form-data tries to decode binary file as UTF-8 if "filename=" not specified

I posted this on StackOverflow [1], but I'm posting it here as well, as I believe this is a bug (or at least quirk) in cgi.FieldStorage where you can't access a file upload properly if "filename=" is not present in the MIME part's Content-Disposition header. There are a couple of related bugs open

Re: [Python-Dev] Can CPython on GitHub use the "Merge" button on pull requests (now that they support "squash and merge")?

Great, and thanks for the info! -Ben On Sep 13, 2016 5:31 PM, "Brett Cannon" wrote: > > > On Tue, 13 Sep 2016 at 13:56 Ben Hoyt wrote: > >> I noticed in [PEP 512 - Document steps to commit a pull request]( >> https://www.python.org/dev/peps/pep-0512/#document-

[Python-Dev] Can CPython on GitHub use the "Merge" button on pull requests (now that they support "squash and merge")?

I noticed in [PEP 512 - Document steps to commit a pull request]( https://www.python.org/dev/peps/pep-0512/#document-steps-to-commit-a-pull-request) it says that CPython on GitHub won't be able to use GitHub's "Merge" button on pull requests, because we want a linear history with one commit per cha

Re: [Python-Dev] Scandir module seeking new maintainer

, Ben Hoyt wrote: > FYI to the group: I'll discuss details with Victor privately. -Ben > > > On Tue, Mar 29, 2016 at 8:12 AM, Ben Hoyt wrote: > >> >> >>> FYI I just finished (I hope!) to bug on os.walk(bytes) on Windows with >>> the "emulate

Re: [Python-Dev] Scandir module seeking new maintainer

FYI to the group: I'll discuss details with Victor privately. -Ben On Tue, Mar 29, 2016 at 8:12 AM, Ben Hoyt wrote: > > >> FYI I just finished (I hope!) to bug on os.walk(bytes) on Windows with >> the "emulated" scandir (since os.scandir() only works on Un

Re: [Python-Dev] Scandir module seeking new maintainer

> FYI I just finished (I hope!) to bug on os.walk(bytes) on Windows with > the "emulated" scandir (since os.scandir() only works on Unicode on > Python 3.5+): > http://bugs.python.org/issue25911 > > Since I helped you to write your PEP 471 (scandir) as the > BDFL-delegate, I think that I now unders

Re: [Python-Dev] Scandir module seeking new maintainer

> > So I'm looking for someone who wants to take over the maintenance of the >> scandir PyPI module. It wouldn't be much work, as there are only a >> couple of fairly small issues to fix. Or it might be a good way to start >> for someone who wants to contribute to a small but useful open source >>

[Python-Dev] Scandir module seeking new maintainer

Hi folks, I'm not sure if this is the right place to ask, but seeing I was fairly active here when developing scandir and getting it into Python 3.5, so I thought I'd start here. I'm the author and current maintainer of the scandir module (Python 3.5's os.sc

Re: [Python-Dev] Improving docs for len() of set

Thanks! Commit ref: https://hg.python.org/cpython/rev/a67fda8e33b0 -Ben On Mon, Feb 8, 2016 at 1:00 PM, Gregory P. Smith wrote: > > > On Mon, Feb 8, 2016 at 8:24 AM Ben Hoyt wrote: > >> Hi folks, >> >> Just a suggestion for a documentation tweak. Currently the

[Python-Dev] Improving docs for len() of set

Hi folks, Just a suggestion for a documentation tweak. Currently the docs for len() on a set say this: .. describe:: len(s) Return the cardinality of set *s*. I'm a relatively seasoned programmer, but I don't really have a maths background, and I didn't know what "cardinality" meant. I

Re: [Python-Dev] [RELEASED] Python 3.5.1 and 3.4.4rc1 are now available

Great, thank you! Small note, not sure if it's related to the release or not: the downloads menu on python.org seems to be broken. The 2.7 download button is showing for me, but the 3.x download button is kind of kaput. See screenshot: http://i.imgur.com/ji1LCnn.png -Ben On Mon, Dec 7, 2015 at

Re: [Python-Dev] [RELEASED] Python 3.5.0b3 is now available

Thanks! Looking forward to trying this. I'm not sure where these descriptions come from, or whether they're carried over from b2 to b3 etc, but one small note on this bullet point: * PEP 471, os.scandir(), a faster alternative to os.walk() This isn't quite correct. os.scandir() is actually an al

Re: [Python-Dev] Enable access to the AST for Python code

ike this is that ideally you don't want to support > this just for CPython. It's definitely cool though! (Using movie poster > style quotes you can turn this into a ringing endorsement: "definitely cool" > -- The BDFL. :-) > > On Wed, May 20, 2015 at 7:26 PM, Be

Re: [Python-Dev] Enable access to the AST for Python code

Heh, thanks. :-) On Thu, May 21, 2015 at 2:17 PM, Guido van Rossum wrote: > Dang it. :-) I just want to encourage you to continue pursuing this idea, > one way or another. > > On Thu, May 21, 2015 at 7:01 AM, Ben Hoyt wrote: > >> Thanks. Good point about python-ideas -

Re: [Python-Dev] Enable access to the AST for Python code

nitely cool though! (Using movie poster > style quotes you can turn this into a ringing endorsement: "definitely > cool" -- The BDFL. :-) > > On Wed, May 20, 2015 at 7:26 PM, Ben Hoyt wrote: > >> Hi Python devs, >> >> Enabling access to the AST for comp

[Python-Dev] Enable access to the AST for Python code

Hi Python devs, Enabling access to the AST for compiled code would make some cool things possible (C# LINQ-style ORMs, for example), and not knowing too much about this part of Python internals, I'm wondering how possible and practical this would be. Context: PonyORM (http://ponyorm.com/) allows

[Python-Dev] Scandir module's C code updated to Python 3.5 code

Hi folks, With os.scandir() now in the Python 3.5 stdlib, I just thought I'd let folks know that I've released the scandir module version 1.0. So this is now basically a copy-n-paste of the C code that went into CPython 3.5's posixmodule.c with the necessary changes to make it work or Python 2.x (

Re: [Python-Dev] PEP 8 update

> My own preference would be: > > def foo(x): > if x >= 0: > return math.sqrt(x) > return None Kind of getting into the weeds here, but I would always invert this to "return errors early, and keep the normal flow at the main indentation level". Depends a little on w

Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

> > os.walk took 0.061s, scandir.walk took 0.012s -- 5.2x as fast > Great, looks much better. :-) Even a bit better than what I'm seeing -- possibly due to your SSD. -Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/

Re: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available

I'm seeing the same issue (though I also get the missing-DLL error dialog when I run python.exe). -Ben On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore wrote: > On 9 March 2015 at 09:34, Larry Hastings wrote: > > On behalf of the Python development community and the Python 3.5 release > > team, I'm t

Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

Hi Ryan, > ./configure --with-pydebug && make -j7 > > I then ran ./python.exe ~/Workspace/python/scandir/benchmark.py and I got: > > Creating tree at /Users/rstuart/Workspace/python/scandir/benchtree: depth=4, > num_dirs=5, num_files=50 > Using slower ctypes version of scandir > Comparing against

Re: [Python-Dev] PEP 471 Final: os.scandir() merged into Python 3.5

part should be fairly straight-forward, as I already have a version available in my GitHub project. -Ben On Sat, Mar 7, 2015 at 9:13 PM, Victor Stinner wrote: > Hi, > > FYI I commited the implementation of os.scandir() written by Ben Hoyt. > I hope that it will be part of Python 3.5

[Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on

When merging some changes while working on scandir, I noticed a minor issue with this commit: https://hg.python.org/cpython/rev/4f6f4aa0d80f The definition of "struct win32_stat" has been moved to fileutils.h and renamed to "struct _Py_stat_struct", which is fine -- however, the old "struct win32

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

>> The whole point of scandir is to expose low-level system calls in a >> cross-platform way. > > Cross-platform is great and preferable, but low-level system facilities > should be made available even when they are unique to a particular OS. Yes, but this can be made cross-platform fairly easily,

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

>> +1 for inode support. I agree with the above -- it should either raise >> AttributeError on Windows if it's not going to be set ... or it should >> be more like Victor's original proposal where .inode() is a method >> that calls stat on Windows. I don't have strong feelings. > > The whole point

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

> +1 we need to provide the inode (we shouldn't be throwing anything from the > underlying directory entry away when possible). But... > > I think the "or None" semantics are a bad idea. It'd be better for this to > raise AttributeError on Windows so that someone can't write the most natural > for

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

> Since there are many ways to split this huge file, I agree that it's > just fine to add these 800 lines and *then* think how the huge file > can be splitted. It's a different topic. That's a good idea. Consider adding the new feature (scandir) and the larger issue of refactoring posixmodule as s

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

> TL;DR: on POSIX, is it useful to know the inode number (st_ino) > without the device number (st_dev)? I can't answer this question (not being a Linux dev and not knowing much about this), but I'm +1 for adding DirEntry.inode(). On Windows, we're exposing all the information FindFirst/FindNext g

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

> > * C implementation: scandir is at least 3.5x faster than listdir, up > > to 44.6x faster on Windows > > * C+Python implementation: scandir is not really faster than listdir, > > between 1.3x and 1.4x faster > > So amusingly, the bottleneck is not so much the cost of system calls, > but the cost

[Python-Dev] Adding numbering to PEP 20, the Zen of Python

I was emailing someone today about implementing something (for PEP 471, as it happens) and wanted to link to the Zen of Python [1] and note a particular clause (in this case "If the implementation is hard to explain, it's a bad idea."). However, there are no clause numbers, so you can't refer to sp

Re: [Python-Dev] Bytes path support

>> If scandir is low-level, and the low-level API's are the ones that should >> support bytes paths, then scandir should support bytes paths. >> >> Is that what you meant to say? > > Yes. The discussions around PEP 471 *deferred* discussions of bytes > and file descriptor support to their own RFEs

Re: [Python-Dev] Bytes path support

>> > The official policy is that we want them [support for bytes paths in >> > stdlib functions] to go away, but reality so far has not budged. We will >> > continue to hold our breath though. :-) >> >> Does that mean that new APIs should explicitly not support bytes? I'm >> thinking of os.scandi

Re: [Python-Dev] Bytes path support

> The official policy is that we want them [support for bytes paths in stdlib > functions] to go away, but reality so far has not budged. We will continue to > hold our breath though. :-) Does that mean that new APIs should explicitly not support bytes? I'm thinking of os.scandir() (PEP 471), wh

Re: [Python-Dev] Documenting enum types

> The enemy must be documented and exported, since users will encounter them. enum == enemy? Is that you, Raymond? ;-) -Ben ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.

Re: [Python-Dev] Multiline 'with' statement line continuation

> Even if it weren't a syntax error, the syntax would be ambiguous. How > will you discern the meaning of:: > > with ( > foo, > bar, > baz): > pass > > Is that three separate context managers? Or is it one tuple with three > items? Is it meaningful t

Re: [Python-Dev] os.walk() is going to be *fast* with scandir

> Victor Stinner suggested [1] to allow scandir(fd) but I don't see it > being mentioned in the pep 471 [2]: it neither supports nor rejects the > idea. > > [1] https://mail.python.org/pipermail/python-dev/2014-July/135283.html > [2] http://legacy.python.org/dev/peps/pep-0471/ Yes, listdir() suppo

Re: [Python-Dev] sum(...) limitation

It seems to me this is something of a pointless discussion -- I highly doubt the current situation is going to change, and it works very well. Even if not perfect, sum() is for numbers, sep.join() for strings. However, I will add one comment: I'm overall -1 on trying to change the current situatio

[Python-Dev] os.walk() is going to be *fast* with scandir

Just thought I'd share some of my excitement about how fast the all-C version [1] of os.scandir() is turning out to be. Below are the results of my scandir / walk benchmark run with three different versions. I'm using an SSD, which seems to make it especially faster than listdir / walk. Note that

Re: [Python-Dev] PEP 471 "scandir" accepted

Makes sense, thanks. -Ben On Tue, Jul 22, 2014 at 4:57 PM, Nick Coghlan wrote: > > On 23 Jul 2014 02:18, "Victor Stinner" wrote: >> >> 2014-07-22 17:52 GMT+02:00 Ben Hoyt : >> > However, given that we have to support this for listdir() anyway, I >&g

Re: [Python-Dev] PEP 471 "scandir" accepted

> Note: listdir() accepts an integer path (an open file descriptor that > refers to a directory) that is passed to fdopendir() on POSIX [4] i.e., > *you can't use scandir() to replace listdir() in this case* (as I've > already mentioned in [1]). See the corresponding tests from [2]. > > [1] https:/

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> We should mimic os.stat() and os.stat_result: os.stat_result symbol > exists in the os namespace, but the type constructor is not > documented. No need for extra protection like not adding the type in > the os module, or adding a "_" prefix to the name. Yeah, that works for me. > By the way, it

Re: [Python-Dev] PEP 471 "scandir" accepted

> I asked privately Guido van Rossum if I can be the BDFL-delegate for > the PEP 471 and he agreed. I accept the latest version of the PEP: > > http://legacy.python.org/dev/peps/pep-0471/ Thank you! > The PEP also explicitly mentions that os.walk() will be modified to > benefit of the new os.

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Maybe it is an abuse of the API. A clear_cache() method would be less > ugly :-) But maybe Ben Hoyt does not want to promote keeping DirEntry > for a long time? > > Another question: should we expose DirEntry type directly in the os > namespace? (os.DirEntry) Again, I'

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Even though there is tangible performance improvement from scandir(), it > would be useful to find out if the API fits well. Got it -- I see where you're coming from now. I'll take a quick look (hopefully later this week). -Ben ___ Python-Dev mailing

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Have you tried modifying importlib's _bootstrap.py to use scandir() instead > of listdir() + stat()? No, I haven't -- I'm not familiar with that code. What does _bootstrap.py do -- does it do a lot of listdir calls and stat-ing of many files? -Ben ___

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> I'd *keep DirEntry.lstat() method* regardless of existence of > .stat(*, follow_symlinks=True) method (despite the slight violation of > DRY principle) for readability. `dir_entry.lstat().st_mode` is more > consice than `dir_entry.stat(follow_symlinks=False).st_mode` and the > meaning of lstat is

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Sorry, I don't remember who but someone proposed to add the follow_symlinks > parameter in scandir() directly. If the parameter is added to methods, > there is no such issue. Yeah, I think having the DirEntry methods do different things depending on how scandir() was called is a really bad idea

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Looks doable. Just make sure the cached entries reflect the > 'follow_symlinks' setting -- so a symlink could end up with both an lstat > cached entry and a stat cached entry. Yes, good point -- basically the functions will use the _stat cache if follow_symlinks=True, otherwise the _lstat cache

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

> Let's not multiply entities beyond necessity. > > There is well-defined *follow_symlinks* parameter > https://docs.python.org/3/library/os.html#follow-symlinks > e.g., os.access, os.chown, os.link, os.stat, os.utime and many other > functions in os module support follow_symlinks parameter, see >

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

First, just to clarify a couple of points. > You forgot one of my argument: we must have exactly the same API than > os.path.is_dir() and pathlib.Path.is_dir(), because it would be very > confusing (source of bugs) to have a different behaviour. Actually, I specifically included that argument. It

[Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

Hi folks, Thanks Victor, Nick, Ethan, and others for continued discussion on the scandir PEP 471 (most recent thread starts at https://mail.python.org/pipermail/python-dev/2014-July/135377.html). Just an aside ... I was reminded again recently why scandir() matters: a scandir user emailed me the

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> Very much agreed that this isn't necessary for just readdir/FindNext >> errors. We've never had this level of detail before -- if listdir() >> fails half way through (very unlikely) it just bombs with OSError and >> you get no entries at all. >> >> If you really really want this (again very unli

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

[replying to python-dev this time] >> The "onerror" approach can also deal with readdir failing, which the >> PEP currently glosses over. > > > Do we want this, though? I can see an error handler for individual entries, > but if one of the *dir commands fails that would seem to be fairly > catas

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> DirEntry methods will remain free (no syscall) for directories and >> regular files. One extra syscall will be needed only for symlinks, >> which are more rare than other file types (for example, you wrote " >> Windows typically makes little use of symlinks"). > > The info we want for scandir is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

I really don't understand why you *want* a worse, much less cross-platform API? > Okay, so using that logic we should head over to the os module and remove: > > ctermid, getenv, getegid... > > Okay, I'm tired of typing, but that list is not even half-way through the os > page, and those are all me

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> On a system which did not supply is_dir automatically I would write that as: > > for entry in os.scandir(path): # info defaults to 'os', which is > basically None in this case > if ignore_entry(entry.name): > continue > if os.path.isdir(entry.full_name): > # do

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> 1) it has an additional "info" argument, the values of which have to >> be documented ('os', 'type', 'lstat', and what each one means) >> 2) it has an additional "onerror" argument, the signature of which and >> fairly complicated return value is non-obvious and has to be >> documented >> 3) it

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

This is just getting way too complex ... further thoughts below. >> This is an interesting idea, but it's just getting more and more >> complex, and I'm guessing that being able to change the attributes of >> DirEntry will make the C implementation more complex. >> >> Also, I'm not sure it's very

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> One issue with option #2 that I just realized -- does scandir yield the >> entry at all if there's a stat error? It >> can't really, because the caller will expect the .lstat attribute to be >> set (assuming he asked for type='lstat') but >> >> it won't be. Is effectively removing these entries

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> The PEP says that DirEntry should mimic pathlib.Path, so I think that >> DirEntry.is_dir() should work as os.path.isir(): if the entry is a >> symbolic link, you should follow the symlink to get the status of the >> linked file with os.stat(). > > Would this not "break" the tree size script bein

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Option 2: > def log_err(exc): > logger.warn("Cannot stat {}".format(exc.filename)) > > def get_tree_size(path): > total = 0 > for entry in os.scandir(path, info='lstat', onerror=log_err): > if entry.is_dir: > total += get_tree_size(entry.full_name) > else:

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Ok, so it means that your example grouping files per type, files and > directories, is also wrong. Or at least, it behaves differently than > os.walk(). You should put symbolic links to directories in the "dirs" > list too. > > if entry.is_dir(): # is_dir() checks os.lstat() > dirs.append(e

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> In this case because the names are exactly the same as the os versions which > /do/ make a system call. Fair enough. > So if I'm finally understanding the root problem here: > > - listdir returns a list of strings, one for each filename and one for > each directory, and keeps no other O/S

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> I did better than that -- I read the whole thing! ;) Thanks. :-) > -1 on the PEP's implementation. > > Just like an attribute does not imply a system call, having a > method named 'is_dir' /does/ imply a system call, and not > having one can be just as misleading. Why does a method imply a sy

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

>> I think you're misunderstanding is_dir() and is_file(), as these don't >> actually call os.stat(). All DirEntry methods either call nothing or >> os.lstat() to get the stat info on the entry itself (not the >> destination of the symlink). > > > Oh. Extract of your PEP: "is_dir(): like os.path.is

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Better to just have the attributes be None if they were not fetched. None > is better than hasattr anyway, at least in the respect of not having to > catch exceptions to function properly. The thing is, is_dir() and lstat() are not attributes (for a good reason). Please read the relevant "Rejec

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> Only exposing what the OS provides for free will make the API too difficult > to use in the common case. But is there a nice way to expand the API that > will allow the user who is trying to avoid extra expense know what > information is already available? > > Even if the initial version doesn't

Re: [Python-Dev] Updates to PEP 471, the os.scandir() proposal

> I remember a pending question on python-dev: > > - Martin von Loewis asked if the scandir generator would have send() > and close() methods as any Python generator. I didn't see a reply on > the mailing (nor in the PEP). Good call. Looks like you're referring to this message: https://mail.python

[Python-Dev] Updates to PEP 471, the os.scandir() proposal

in the PEP as well as the relevant python-dev discussion (linked to in the PEP). Thanks, Ben PEP: 471 Title: os.scandir() function -- a better and faster directory iterator Version: $Revision$ Last-Modified: $Date$ Author: Ben Hoyt Status: Draft Type: Standards Track Content-Type: text/x-rst Creat

Re: [Python-Dev] My summary of the scandir (PEP 471)

Thanks for the clarifications and support. > Ah, the wording in the PEP says "Linux, Windows, OS X". Superficially, > that said "everywhere" to me. It might be worth calling out > specifically some examples where it's not available without an extra > system call, just to make the point explicit.

Re: [Python-Dev] My summary of the scandir (PEP 471)

Thanks for the effort in your response, Paul. I'm all for KISS, but let's just slow down a bit here. > I think that thin wrapper is needed - even > if the various bells and whistles are useful, they can be built on top > of a low-level version (whereas the converse is not the case). Yes, but API

Re: [Python-Dev] My summary of the scandir (PEP 471)

> We need per-iteration error handling for the readdir call anyway, so I think > an onerror callback is a better option than dropping the ability to easily > obtain full stat information as part of the iteration. I don't mind the idea of an "onerror" callback, but it's adding complexity. Putting a

Re: [Python-Dev] My summary of the scandir (PEP 471)

> No need for a microsecond-timed deletion -- a directory with +r but > without +x will allow you to list the entries, but stat calls on the > files will fail with EPERM: Ah -- very good to know, thanks. This definitely points me in the direction of wanting better control over error handling. Spe

Re: [Python-Dev] My summary of the scandir (PEP 471)

Thanks for spinning this off to (hopefully) finished the discussion. I agree it's nearly time to update the PEP. > @Ben: it's time to update your PEP to complete it with this > discussion! IMO DirEntry must be as simple as possible and portable: > > - os.scandir(str) > - DirEntry.lstat_result obje

Re: [Python-Dev] PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)

Thanks, Victor. I don't have any experience with dir_fd handling, so unfortunately can't really comment here. What advantages does it bring? I notice that even os.listdir() on Python 3.4 doesn't have anything related to file descriptors, so I'd be in favour of not including support. We can always

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> I suppose the exact behavior is still under discussion, as there are only > two or three fields one gets "for free" on Windows (I think...), where as an > os.stat call would get everything available for the platform. No, Windows is nice enough to give you all the same stat_result fields during s

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

> So, here's my alternative proposal: add an "ensure_lstat" flag to > scandir() itself, and don't have *any* methods on DirEntry, only > attributes. > > That would make the DirEntry attributes: > > is_dir: boolean, always populated > is_file: boolean, always populated > is_symlink boole

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

Re is_dir etc being properties rather than methods: >> I find this behaviour a bit misleading: using methods and have them >> return cached results. How much (implementation and/or performance >> and/or memory) overhead would incur by using property-like access here? >> I think this would underlin

  1   2   >