Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Paul Moore]

On 13 February 2015 at 00:15, Vinay Sajip  wrote:
> I remembered that I added a feature (with the help of Pawel Jasinski) to 
> allow e.g. py -ipy to pick
> a command 'ipy' configured in the .ini file (this is used to launch 
> IronPython, but it could be used
> for other things too).

At the time the launcher was first released I *really* wanted that
feature, but couldn't figure out how to cleanly add it. I'd love to
see that go into the main repo :-)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Paul Moore]

On 13 February 2015 at 06:59, Thomas Heller  wrote:
> To make it clear: My suggestion is (or was, maybe it isn't a good idea)
> to have some way to start 'something' that is in the Scripts
> directory of a Python installation (which is usually a python script
> or an exe-wrapper for it), without typing in the full path.  And without
> changing the PATH envrionment variable.

People using the old-style "scripts" keyword to setup() can put pretty
much anything into the Scripts directory - I've seen .py files, Python
files with no suffix, bat files, even Unix shell files (not much use
on Windows...) in there. I don't think it's reasonable for the
launcher to try to run all of those things.

Maybe restricting it to running ".py" files or ".exe" files would be
reasonable. That covers entry points (which should be the norm for
newer projects) and Python scripts (which are the most likely things
to be portable).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Paul Moore]

On 13 February 2015 at 07:25, Nick Coghlan  wrote:
> PEP 441 was aimed at giving the feature more visibility, in addition
> to making appropriately designed archives easier to create (IIRC, my
> main request for that PEP was to change the proposed module name to
> the more prosaic, but also more self-explanatory, "pyzapp", although
> it looks like I may have suggested that to Daniel offline rather than
> through the mailing list).
>
> Regardless, the PEP's not controversial as far as I am aware, so it
> would be nice if someone had the time to get it cleaned up and
> formally proposed for 3.5 (assuming Daniel was OK with that).

I'd be willing to give that a go. The PEPs URL is giving me a "Bad
Gateway" error at the moment, but I think the only coding needed in
there is writing the "pyzaa/pyzapp" module. (If Steve would be willing
to add the file associations - I'm not sure I want to dive into the
new installer technology along with all the *other* things I'm
volunteering for right now :-))

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Jonas Wielicki]

If I may humbly chime in this, with a hint...

On 13.02.2015 05:01, Guido van Rossum wrote:
> On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman  wrote:
>> [snip]
>> 2) always use the type of self when creating new instances
>>
>>pros:
>>  - subclasses automatically maintain type
>>  - much less code in the simple cases [1]
>>
>>cons:
>>  - if constructor signatures change, must override all methods which
>>create new objects
>>
>> Unless there are powerful reasons against number 2 (such as performance,
>> or the effort to affect the change), it sure
>> seems like the nicer way to go.
>>
>> So back to my original question: what other concerns are there, and has
>> anybody done any benchmarks?
>>
> 
> Con for #2 is a showstopper. Forget about it.

I would like to mention that there is another language out there which
knows about virtual constructors (virtual like in virtual methods, with
signature match requirements and such), which is FreePascal (and Delphi,
and I think original Object Pascal too).

It is actually a feature I liked about these languages, compared to
C++03 and others, that constructors could be virtual and that classes
were first-class citizens.

Of course, Python cannot check the signature at compile time. But I
think as long as it is documented, there should be no reason not to
allow and support it. It really is analogous to other methods which need
to have a matching signature.

just my two cents,
jwi






signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Victor Stinner]

Hi,

TL;DR: on POSIX, is it useful to know the inode number (st_ino)
without the device number (st_dev)?

While reading feedback on the Python 3.5 alpha 1 release, I saw a
comment saying that the current design of os.scandir() (PEP 471)
doesn't fit a very specific usecase where the inode number is needed:

"Ah, turns out we needed even more optimizations than that is able to
give us; in particular, the underlying system readdir call gives us
the inode number, which we need to compare against a cache of hard
links, in order to avoid having to stat the underlying files if we've
already done so on another hard link. It looks like the DirEntry API
used here only includes the path and name, not the inode number,
without invoking another stat call, and we needed to optimize out that
extra stat call."
https://www.reddit.com/r/Python/comments/2synry/so_8_peps_are_currently_being_proposed_for_python/cnvnz1w

Since the C function readdir() provides the inode number (d_ino field
of the dirent structure), I propose add a new DirEntry.inode() method.


*** Now the real question: is it useful to know the inode number
(st_ino) without the device number (st_dev)? ***

On POSIX, you can still get the st_dev from DirEntry.stat(), but it
always require a system call. So you loose the whole purpose of
DirEntry (no extra syscall).

I wrote a script script check_stdev.py, attached to this email, to
check if all entries of a directory have the same st_dev value than
the directory itself:

- same for /usr/bin, /usr/lib, /tmp, /proc, ...
- different for /dev

What about "union" file systems like UnionFS or thinks like "mount -o
bind"? Can someone test? Does anyone have some information?

So the answer looks to be: it's useful for all directories except of
/dev. Example:
---
/dev/hugepages st_dev is different: 35 vs 5
/dev/mqueue st_dev is different: 13 vs 5
/dev/pts st_dev is different: 11 vs 5
/dev/shm st_dev is different: 17 vs 5
---


On POSIX, DirEntry.inode() just exposes the d_ino value from readdir().

On Windows, FirstFindFileW/FindFindFileW returns almost a full
stat_result structure, except of st_ino, st_dev and st_nlink fields
which are set to 0.

So DirEntry.inode() has to call os.lstat() to read the inode number.
The inode number will be cached by DirEntry.inode() in the DirEntry
object, but the os.lstat() result is dropped.

On Windows, I don't want to cache the full os.lstat() result from
DirEntry.inode() into DirEntry to replace the previous incomplete
stat_result from FirstFindFileW/FindFindFileW, because DirEntry.stat()
would return a different result (st_ino, st_dev, st_nlink fields set
or not) depending if the inode() methode was called or not.

Note: scandir-6.patch of http://bugs.python.org/issue22524 contains an
implementation of os.scandir() with DirEntry.inode(), if you want to
play.

Victor
import sys, os

path = sys.argv[1]
names = os.listdir(path)
dev = os.lstat(path).st_dev
for name in names:
entry_path = os.path.join(path, name)
entry_dev = os.lstat(entry_path).st_dev
if entry_dev != dev:
print("%s st_dev is different: %s vs %s" % (entry_path, entry_dev, dev))

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Vinay Sajip]

- Original Message -

From: Paul Moore 


> I'd also appreciate your views on the spin-off thread and PEP 486
> ("Make the Python Launcher aware of virtual environments").


I just thought of something ... as far as I know, we've enabled searching for a 
command on the path - since when a venv is active in the sense of VIRTUAL_ENV 
being defined, PATH is also set up so that the venv's Scripts folder is on it - 
so we would just need to find "python.exe" on the PATH and invoke that - no 
need to even use the value of VIRTUAL_ENV, making implementation simpler. What 
do you think?

Regards,

Vinay
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Paul Moore]

On 13 February 2015 at 09:47, Vinay Sajip  wrote:
> I just thought of something ... as far as I know, we've enabled searching for 
> a command on the path - since when a venv is active in the sense of 
> VIRTUAL_ENV being defined, PATH is also set up so that the venv's Scripts 
> folder is on it - so we would just need to find "python.exe" on the PATH and 
> invoke that - no need to even use the value of VIRTUAL_ENV, making 
> implementation simpler. What do you think?

Two comments:

1. I'm not sure (but I've not dug into the implementation yet) that
there's much practical difference between locating python.exe via a
PATH search, or locating it via concatenating VIRTUAL_ENV and the
relative path.
2. It would be a nice, although extremely obscure, feature to be able
to allow people who want to keep Python off their path to set
VIRTUAL_ENV but *not* set PATH, so that py.exe does the right thing
but there's *still* no python.exe on PATH. Limit the dependency on a
single environment variable rather than two, in other words.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

With Python's cooperative inheritance, I think you want to do everything
through one constructor sending keyword arguments up the chain.  The
keyword arguments are popped off as needed.  With this setup I don't think
you need "overloaded constructors".

Best,
Neil

On Fri, Feb 13, 2015 at 4:44 AM, Jonas Wielicki 
wrote:

> If I may humbly chime in this, with a hint...
>
> On 13.02.2015 05:01, Guido van Rossum wrote:
> > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman 
> wrote:
> >> [snip]
> >> 2) always use the type of self when creating new instances
> >>
> >>pros:
> >>  - subclasses automatically maintain type
> >>  - much less code in the simple cases [1]
> >>
> >>cons:
> >>  - if constructor signatures change, must override all methods which
> >>create new objects
> >>
> >> Unless there are powerful reasons against number 2 (such as performance,
> >> or the effort to affect the change), it sure
> >> seems like the nicer way to go.
> >>
> >> So back to my original question: what other concerns are there, and has
> >> anybody done any benchmarks?
> >>
> >
> > Con for #2 is a showstopper. Forget about it.
>
> I would like to mention that there is another language out there which
> knows about virtual constructors (virtual like in virtual methods, with
> signature match requirements and such), which is FreePascal (and Delphi,
> and I think original Object Pascal too).
>
> It is actually a feature I liked about these languages, compared to
> C++03 and others, that constructors could be virtual and that classes
> were first-class citizens.
>
> Of course, Python cannot check the signature at compile time. But I
> think as long as it is documented, there should be no reason not to
> allow and support it. It really is analogous to other methods which need
> to have a matching signature.
>
> just my two cents,
> jwi
>
>
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Victor Stinner]

Hi,

TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x
faster than os.listdir() when the file type is checked?

I accepted the PEP 471 (os.scandir) a few months ago, but it is not
implement yet in Python 3.5, because I didn't make a choice on the
implementation.

Ben Hoyt wrote different implementations:
- full C: os.scandir() and DirEntry are written in C (no change on os.py)
- C+Python: os._scandir() (wrapper for opendir/readdir and
FindFirstFileW/FindNextFileW) in C, DirEntry in Python
- ctypes: os.scandir() and DirEntry fully implemented in Python

I'm not interested by the ctypes implementation. It's useful for a
third party project hosted at PyPI, but for CPython I prefer to wrap C
functions using C code.


In short, the C implementation is faster than the C+Python implementation.

The issue #22524 (*) is full of benchmark numbers. IMO the most
interesting benchmark is to compare os.listdir() + os.stat() versus
os.scandir() + Direntry.is_dir(). Let me try to summarize results of
this benchmark:

* C implementation: scandir is at least 3.5x faster than listdir, up
to 44.6x faster on Windows
* C+Python implementation: scandir is not really faster than listdir,
between 1.3x and 1.4x faster

(*) http://bugs.python.org/issue22524


Ben Hoyt reminded me that os.scandir() (PEP 471) doesn't add any new
feature: pathlib already provides a nice API on top of os and os.path
modules. (You may even notice that DirEntry a much fewer methods ;-))
The main (only?) purpose of the PEP is performance.

If os.scandir() is "only" 1.4x faster, I don't think that it is
interesting to use os.scandir() in an application. I guess that all
applications/libraries will want to keep compatibility with Python 3.4
and older and so will anyway have to duplicate the code to use
os.listdir() + os.stat(). So is it worth to duplicate code for such
small speedup?

Now I see 3 choices:

- take the full C implementation, because it's much faster (at least
3.4x faster!)
- reject the whole PEP 471 (not nice), because it adds too much code
for a minor speedup (not true on Windows: up to 44x faster!)
- take the C+Python implementation, because maintenance matters more
than performances (only 1.3x faster, sorry)

=> IMO the best option is to take the C implementation. What do you think?


I'm concerned by the length of the C code: the full C implementations
adds ~800 lines of C code to posixmodule.c. This file is already the
longest C file in CPython. I don't want to make it longer, but I'm not
motived to start to split it. Last time I proposed to split a file
(unicodeobject.c), some developers complained that it makes search
harder. I don't understand this, there are so many tools to navigate
in C code. But it was enough for me to give up on this idea.

A alternative is to add a new _scandir.c module to host the new C
code, and share some code with posixmodule.c: remove "static" keyword
from required C functions (functions to convert Windows attributes to
a os.stat_result object). That's a reasonable choice. What do you
think?


FYI I ran the benchmark on different hardware (SSD, HDD, tmpfs), file
systems (ext4, tmpfs, NFS/ext4), operating systems (Linux, Windows).

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Paul Moore]

On 13 February 2015 at 10:07, Victor Stinner  wrote:
> => IMO the best option is to take the C implementation. What do you think?

FWIW (as I'm not a core dev) I agree. The Windows speedup is huge, and
well worth adding the code. I'm assuming that the majority of the C
code is cross-platform, so we're not adding a big chunk of code
needing *both* Windows and C skills to maintain (any dev with C skills
could handle it).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Glenn Linderman]

On 2/13/2015 1:13 AM, Paul Moore wrote:

On 13 February 2015 at 06:59, Thomas Heller  wrote:

To make it clear: My suggestion is (or was, maybe it isn't a good idea)
to have some way to start 'something' that is in the Scripts
directory of a Python installation (which is usually a python script
or an exe-wrapper for it), without typing in the full path.  And without
changing the PATH envrionment variable.

People using the old-style "scripts" keyword to setup() can put pretty
much anything into the Scripts directory - I've seen .py files, Python
files with no suffix, bat files, even Unix shell files (not much use
on Windows...) in there. I don't think it's reasonable for the
launcher to try to run all of those things.

Maybe restricting it to running ".py" files or ".exe" files would be
reasonable. That covers entry points (which should be the norm for
newer projects) and Python scripts (which are the most likely things
to be portable).
The WINDOWS py launcher hardly needs to be portable.  Batch/CMD files 
also seem useful on WINDOWS. And Powershell?


If a launcher is developed for or enhanced for other systems, shell 
scripts may be useful, and Batch/CMD not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Victor Stinner]

2015-02-13 11:19 GMT+01:00 Paul Moore :
> On 13 February 2015 at 10:07, Victor Stinner  wrote:
>> => IMO the best option is to take the C implementation. What do you think?
>
> FWIW (as I'm not a core dev) I agree. The Windows speedup is huge, and
> well worth adding the code. I'm assuming that the majority of the C
> code is cross-platform, so we're not adding a big chunk of code
> needing *both* Windows and C skills to maintain (any dev with C skills
> could handle it).
>
> Paul

The patch can be read here:
http://bugs.python.org/file36963/scandir-2.patch

Or using Rietveld:
http://bugs.python.org/review/22524/#ps13104

The C code is quite simple. It takes 800 lines because C code is more
"verbose" than Python code. Manipulate strings, manage memory, take
care of the reference counter, etc. just takes more lines.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Serhiy Storchaka]

On 13.02.15 12:07, Victor Stinner wrote:

TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x
faster than os.listdir() when the file type is checked?


You can try to make Python implementation faster if

1) Don't set attributes to None in constructor.

2) Implement scandir as:

def scandir(path):
return map(partial(DirEntry, path), _scandir(path)).

3) Or pass DirEntry to _scandir:

def scandir(path):
yield from _scandir(path, DirEntry)


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Paul Moore]

On 13 February 2015 at 10:15, Glenn Linderman  wrote:
> Maybe restricting it to running ".py" files or ".exe" files would be
> reasonable. That covers entry points (which should be the norm for
> newer projects) and Python scripts (which are the most likely things
> to be portable).
>
> The WINDOWS py launcher hardly needs to be portable.  Batch/CMD files also
> seem useful on WINDOWS. And Powershell?
>
> If a launcher is developed for or enhanced for other systems, shell scripts
> may be useful, and Batch/CMD not.

By "portable" in this sense I meant "not written by a Unix developer
who hadn't thought about Windows much". I was comparing Python files
with a .py extension and extensionless Python files, specifically (my
wording didn't make that clear, sorry). Yes, files with extensions of
.bat or .cmd or .ps1 are probably written for Windows only. But I
think it's getting out of scope of the launcher to decide to use
cmd.exe to launch .bat and .cmd files, and powershell to launch .ps1
files.

Regardless, my real point was that there can be all sorts of stuff in
the Scripts directory, and I'm not convinced that this is a direction
the launcher should be going in. Someone *could* make a case for
launching certain types of file that might be in there, but I'm not
going to be that person.

One thought - it's not hard using pkg_resources to enumerate all
available console_script entry points, pick one and run it. So you
could pretty simply write a script (that itself could be run with the
launcher) to run anything in Scripts that's built via entry points
(which is the recommended way these days).

# launch.py
import sys
import pkg_resources

# Get the script name
script = sys.argv[1]
# Remove our name from sys.argv
del sys.argv[0]

# There's probably a better API you can use
for ep in pkg_resources.working_set.iter_entry_points('console_scripts'):
if ep.name == script:
fn = ep.load()
sys.exit(fn())

# Couldn't find the entry point
print("No entry point called {} available".format(script), file=sys.stderr)
sys.exit(1)

Now you can do "py launch.py pip --help" and it will run pip for you.

If a project isn't exposing its scripts as entry points, ask them to :-)

Paul

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Serhiy Storchaka]

On 13.02.15 12:07, Victor Stinner wrote:

* C implementation: scandir is at least 3.5x faster than listdir, up
to 44.6x faster on Windows


Results on Windows was obtained in the becnhmark that doesn't drop disk 
caches and runs listdir before scandir.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Victor Stinner]

2015-02-13 12:27 GMT+01:00 Serhiy Storchaka :
> On 13.02.15 12:07, Victor Stinner wrote:
>>
>> * C implementation: scandir is at least 3.5x faster than listdir, up
>> to 44.6x faster on Windows
>
>
> Results on Windows was obtained in the becnhmark that doesn't drop disk
> caches and runs listdir before scandir.

The benchmark code is here:
http://bugs.python.org/file38120/bench_scandir2.py

Eeach test is repeated 5 times, I compared the duration of the fastest
call. A test calls the function 5 times in a loop.

Anyway, it's not the first call to listdir() which fills the disk
cache, but the call to count_entries() (which is implemented with
os.scandir).

So is there any issue in the benchmark script or not?

On Linux, you can use "bench_nocache" (and "bench_nostat_nocache")
which flushs the cache: it writes "3" into /proc/sys/vm/drop_caches.
scandir is always faster when the disk cache is flushed.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Nick Coghlan]

On 13 February 2015 at 20:33, Victor Stinner  wrote:
> 2015-02-13 11:19 GMT+01:00 Paul Moore :
>> On 13 February 2015 at 10:07, Victor Stinner  
>> wrote:
>>> => IMO the best option is to take the C implementation. What do you think?
>>
>> FWIW (as I'm not a core dev) I agree. The Windows speedup is huge, and
>> well worth adding the code. I'm assuming that the majority of the C
>> code is cross-platform, so we're not adding a big chunk of code
>> needing *both* Windows and C skills to maintain (any dev with C skills
>> could handle it).
>>
>> Paul
>
> The patch can be read here:
> http://bugs.python.org/file36963/scandir-2.patch
>
> Or using Rietveld:
> http://bugs.python.org/review/22524/#ps13104
>
> The C code is quite simple. It takes 800 lines because C code is more
> "verbose" than Python code. Manipulate strings, manage memory, take
> care of the reference counter, etc. just takes more lines.

This isn't code I'd expect us to have to change very often, so the
maintenance risks associated with the pure C implementation seem low.
Having it in a separate file rather than making the main
implementation file for os even larger does seem like an attractive
structural option though.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Victor Stinner]

2015-02-13 11:52 GMT+01:00 Serhiy Storchaka :
> You can try to make Python implementation faster if
>
> 1) Don't set attributes to None in constructor.

The class uses __slots__. Setting attributes in the class body is
forbidden when __slots__ is used.

> 3) Or pass DirEntry to _scandir:
>
> def scandir(path):
> yield from _scandir(path, DirEntry)

I implemented that and there is no major change (1.3x faster => 1.5x,
it's still far from 3.5x faster seen with the C implementation).


I analyzed numbers (on my desktop PC, HDD, ext4):

- readdir: 380 ns
- os.stat:1500 ns
- DirEntry(C): 100 ns
- DirEntry (Py):   530 ns (5.3x slower)
- is_dir(C):75 ns
- is_dir (Py): 260 ns (3.5x slower)

listdir+stat benchmarks takes (readdir + stat) nanoseconds
scandir+is_dir takes (readdir + DirEntry + is_dir) nanoseconds
=> scandir+is_dir is faster than list+stat if (DirEntry+is_dir) is
faster than (stat).

Callig os.stat takes 1500 ns, while readdir() only provides
informations required by the benchmark. So if DirEntry +
DirEntry.is_dir is faster than 1500 ns, we won :-)

The Python implementation takes 790 ns, but the C implementation takes
only 175 ns! (4.5x faster)

I don't think that any Python performance trick can reduce the Python
overhead to make the C+Python implementation interesting compared to
os.listdir+os.stat. We are talking about nanoseconds, Python cannot
beat C at this resolution.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Antoine Pitrou]

On Fri, 13 Feb 2015 13:27:04 +0200
Serhiy Storchaka  wrote:

> On 13.02.15 12:07, Victor Stinner wrote:
> > * C implementation: scandir is at least 3.5x faster than listdir, up
> > to 44.6x faster on Windows
> 
> Results on Windows was obtained in the becnhmark that doesn't drop disk 
> caches and runs listdir before scandir.

Well, that's the point. The objective here is to speed up Python, not
the hard drive.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Antoine Pitrou]

On Fri, 13 Feb 2015 11:07:03 +0100
Victor Stinner  wrote:
> 
> * C implementation: scandir is at least 3.5x faster than listdir, up
> to 44.6x faster on Windows
> * C+Python implementation: scandir is not really faster than listdir,
> between 1.3x and 1.4x faster

So amusingly, the bottleneck is not so much the cost of system calls,
but the cost of Python wrappers around system calls.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Ben Hoyt]

> > * C implementation: scandir is at least 3.5x faster than listdir, up
> > to 44.6x faster on Windows
> > * C+Python implementation: scandir is not really faster than listdir,
> > between 1.3x and 1.4x faster
>
> So amusingly, the bottleneck is not so much the cost of system calls,
> but the cost of Python wrappers around system calls.

Yes, that's basically right. Or put another way, the cost of the extra
system calls is dwarfed by the cost of wrapping things in Python.

Victor's given a great summary of the issues at the top of this
thread, and I'm definitely for the all-C version -- otherwise we gain
a bunch of speed by not calling stat(), but then lose most of it again
with the Python wrapping. As Victor noted, the rationale for PEP 471
has always been about performance, and if we don't have much of that
(especially on Linux), it's not nearly as worthwhile.

Re maintenance of the C code: yes, the pure C version is about twice
as many lines as the half Python version (~800 vs ~400), but I think
Nick makes a good point here: "This isn't code I'd expect us to have
to change very often, so the maintenance risks associated with the
pure C implementation seem low." We have to vet this code thoroughly
basically once, now. :-)

If we go ahead with the all C approach, I'd be in favour of
refactoring a little and putting the new scandir code into a separate
C file. There are two ways to do this: a) sticking with a single
Python module and just referencing the non-static functions in
scandir.c from posixmodule.c, or b) sharing some functions but making
_scandir.c its own importable module. Option (a) is somewhat simpler
as there's not module setup stuff twice, but I don't know if there's a
precedent for that way of doing things.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Ben Hoyt]

> TL;DR: on POSIX, is it useful to know the inode number (st_ino)
> without the device number (st_dev)?

I can't answer this question (not being a Linux dev and not knowing
much about this), but I'm +1 for adding DirEntry.inode().

On Windows, we're exposing all the information FindFirst/FindNext give
us, but on Linux we expose everything useful from readdir except
d_ino, which is easy to add, and according to that reddit comment, may
make scandir useful in more real scenarios.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Antoine Pitrou]

On Fri, 13 Feb 2015 08:35:00 -0500
Ben Hoyt  wrote:
> 
> If we go ahead with the all C approach, I'd be in favour of
> refactoring a little and putting the new scandir code into a separate
> C file. There are two ways to do this: a) sticking with a single
> Python module and just referencing the non-static functions in
> scandir.c from posixmodule.c, or b) sharing some functions but making
> _scandir.c its own importable module. Option (a) is somewhat simpler
> as there's not module setup stuff twice, but I don't know if there's a
> precedent for that way of doing things.

The _io module already does things that way (the (a) option, I mean).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Steve Dower]

I think posixmodule is a great candidate for splitting up by platform rather 
than function, as the whole file is packed with ifdef. It's really only lacking 
a volunteer to do it, but we could start here (ie. make posixmodule_nt.c for 
the Windows impl, etc.) and progressively move function implementations out 
over time? All the module setup and probably most of the Python layer can stay 
where it is.

More likely we're going to get bogged down discussing it again though, so if 
that happens my vote is to just make posixmodule.c 800 lines longer.

Cheers,
Steve

Top-posted from my Windows Phone

From: Antoine Pitrou
Sent: ‎2/‎13/‎2015 5:44
To: python-dev@python.org
Subject: Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation 
(full C or C+Python)

On Fri, 13 Feb 2015 08:35:00 -0500
Ben Hoyt  wrote:
>
> If we go ahead with the all C approach, I'd be in favour of
> refactoring a little and putting the new scandir code into a separate
> C file. There are two ways to do this: a) sticking with a single
> Python module and just referencing the non-static functions in
> scandir.c from posixmodule.c, or b) sharing some functions but making
> _scandir.c its own importable module. Option (a) is somewhat simpler
> as there's not module setup stuff twice, but I don't know if there's a
> precedent for that way of doing things.

The _io module already does things that way (the (a) option, I mean).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

Launching non-Python scripts from py.exe is not going to work as well as you 
may hope. There will be numerous subtle bugs due to the cmd->py.exe->cmd 
wrapping (Powershell users would have seen this with some console apps or batch 
files). I think having a global command to add a particular Python to PATH for 
the current session will work best here, and yes, I'm working on it.

If py.exe starts defaulting to whatever is on PATH then I don't see the point 
of it. Knowing that you'll get the latest 2.x version by default is quite handy 
(I'd be quite okay changing that to 3.x, though it is specified in the original 
PEP). For me, the point of py.exe is to be able to ignore PATH completely.

Adding more commands to the launcher args (eg. -ipy) will make the parsing far 
more complex, especially when deciding which args to consume and which to 
forward. I think you can make a copy of the launcher called ipy.exe and add 
ipy.ini with the path to IronPython in it as default.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎2/‎13/‎2015 3:20
To: Glenn Linderman
Cc: Python Dev
Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

On 13 February 2015 at 10:15, Glenn Linderman  wrote:
> Maybe restricting it to running ".py" files or ".exe" files would be
> reasonable. That covers entry points (which should be the norm for
> newer projects) and Python scripts (which are the most likely things
> to be portable).
>
> The WINDOWS py launcher hardly needs to be portable.  Batch/CMD files also
> seem useful on WINDOWS. And Powershell?
>
> If a launcher is developed for or enhanced for other systems, shell scripts
> may be useful, and Batch/CMD not.

By "portable" in this sense I meant "not written by a Unix developer
who hadn't thought about Windows much". I was comparing Python files
with a .py extension and extensionless Python files, specifically (my
wording didn't make that clear, sorry). Yes, files with extensions of
.bat or .cmd or .ps1 are probably written for Windows only. But I
think it's getting out of scope of the launcher to decide to use
cmd.exe to launch .bat and .cmd files, and powershell to launch .ps1
files.

Regardless, my real point was that there can be all sorts of stuff in
the Scripts directory, and I'm not convinced that this is a direction
the launcher should be going in. Someone *could* make a case for
launching certain types of file that might be in there, but I'm not
going to be that person.

One thought - it's not hard using pkg_resources to enumerate all
available console_script entry points, pick one and run it. So you
could pretty simply write a script (that itself could be run with the
launcher) to run anything in Scripts that's built via entry points
(which is the recommended way these days).

# launch.py
import sys
import pkg_resources

# Get the script name
script = sys.argv[1]
# Remove our name from sys.argv
del sys.argv[0]

# There's probably a better API you can use
for ep in pkg_resources.working_set.iter_entry_points('console_scripts'):
if ep.name == script:
fn = ep.load()
sys.exit(fn())

# Couldn't find the entry point
print("No entry point called {} available".format(script), file=sys.stderr)
sys.exit(1)

Now you can do "py launch.py pip --help" and it will run pip for you.

If a project isn't exposing its scripts as entry points, ask them to :-)

Paul

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Victor Stinner]

2015-02-13 15:36 GMT+01:00 Steve Dower :
> I think posixmodule is a great candidate for splitting up by platform rather
> than function, as the whole file is packed with ifdef. It's really only
> lacking a volunteer to do it, but we could start here (ie. make
> posixmodule_nt.c for the Windows impl, etc.) and progressively move function
> implementations out over time? All the module setup and probably most of the
> Python layer can stay where it is.
>
> More likely we're going to get bogged down discussing it again though, so if
> that happens my vote is to just make posixmodule.c 800 lines longer.

Since there are many ways to split this huge file, I agree that it's
just fine to add these 800 lines and *then* think how the huge file
can be splitted. It's a different topic.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Jonas Wielicki]

If I may humbly chime in this, with a hint...

On 13.02.2015 05:01, Guido van Rossum wrote:
> On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman  wrote:
>> [snip]
>> 2) always use the type of self when creating new instances
>>
>>pros:
>>  - subclasses automatically maintain type
>>  - much less code in the simple cases [1]
>>
>>cons:
>>  - if constructor signatures change, must override all methods which
>>create new objects
>>
>> Unless there are powerful reasons against number 2 (such as performance,
>> or the effort to affect the change), it sure
>> seems like the nicer way to go.
>>
>> So back to my original question: what other concerns are there, and has
>> anybody done any benchmarks?
>>
> 
> Con for #2 is a showstopper. Forget about it.

I would like to mention that there is another language out there which
knows about virtual constructors (virtual like in virtual methods, with
signature match requirements and such), which is FreePascal (and Delphi,
and I think original Object Pascal too).

It is actually a feature I liked about these languages, compared to
C++03 and others, that constructors could be virtual and that classes
were first-class citizens.

Of course, Python cannot check the signature at compile time. But I
think as long as it is documented, there should be no reason not to
allow and support it. It really is analogous to other methods which need
to have a matching signature.

just my two cents,
jwi




signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Paul Moore]

On 13 February 2015 at 14:27, Steve Dower  wrote:
> If py.exe starts defaulting to whatever is on PATH then I don't see the
> point of it. Knowing that you'll get the latest 2.x version by default is
> quite handy (I'd be quite okay changing that to 3.x, though it is specified
> in the original PEP). For me, the point of py.exe is to be able to ignore
> PATH completely.

It does seem a bit bizarre to me that a launcher shipped with Python 3
defaults to using Python 2 if it's available. I'd like to see that
change, but as it's specified in the original PEP I guess it might
meet with some pushback :-(

> Adding more commands to the launcher args (eg. -ipy) will make the parsing
> far more complex, especially when deciding which args to consume and which
> to forward. I think you can make a copy of the launcher called ipy.exe and
> add ipy.ini with the path to IronPython in it as default.

Well, Vinay has this functionality already available in his fork, so
it's not impossible. But whether or not it's added I do think that all
distributed versions of py.exe should have the same functionality, so
either the CPython version should add it or Vinay should remove it (or
rename his version and describe it as an *enhanced* version of the
py.exe launcher, rather than as a *standalone* version...)

I'm +0 on the functionality personally - I quite like the idea, but
I've no real need for it and I've lived without it long enough now.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Ben Hoyt]

> Since there are many ways to split this huge file, I agree that it's
> just fine to add these 800 lines and *then* think how the huge file
> can be splitted. It's a different topic.

That's a good idea. Consider adding the new feature (scandir) and the
larger issue of refactoring posixmodule as separate issues, separate
commits, etc.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Ethan Furman]

On 02/13/2015 02:07 AM, Victor Stinner wrote:
> Hi,
> 
> TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x
> faster than os.listdir() when the file type is checked?

+1 for the all-C version.

--
~Ethan~



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Ethan Furman]

On 02/13/2015 01:57 AM, Paul Moore wrote:

> 2. It would be a nice, although extremely obscure, feature to be able
> to allow people who want to keep Python off their path to set
> VIRTUAL_ENV but *not* set PATH, so that py.exe does the right thing
> but there's *still* no python.exe on PATH. Limit the dependency on a
> single environment variable rather than two, in other words.

+1

--
~Ethan~



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Thu, Feb 12, 2015 at 11:01 PM, Guido van Rossum  wrote:

>
>> 2) always use the type of self when creating new instances
>> ..
>>cons:
>>  - if constructor signatures change, must override all methods which
>>create new objects
>>
>
> Con for #2 is a showstopper. Forget about it.
>

Sorry if I am missing something obvious, but I still don't understand why
the same logic does not apply to class methods that create new instances:

>>> from datetime import *
>>> date.today()
datetime.date(2015, 2, 13)
>>> datetime.today()
datetime.datetime(2015, 2, 13, 11, 37, 23, 678680)
>>> class Date(date):
... pass
...
>>> Date.today()
Date(2015, 2, 13)

(I actually find datetime.today() returning a datetime rather than a date a
questionable design decision, but probably the datetime type should not
have been a subclass of the date to begin with.)

Are there any date subclasses in the wild that don't accept year, month,
day in the constructor?  If you create such a class, wouldn't you want to
override __add__ and friends anyways?  We already know that you will have
to override today().
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Summary of Python tracker Issues

[Python tracker]

ACTIVITY SUMMARY (2015-02-06 - 2015-02-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open4790 (+25)
  closed 30430 (+32)
  total  35220 (+57)

Open issues with patches: 2254 


Issues opened (47)
==

#21793: httplib client/server status refactor
http://bugs.python.org/issue21793  reopened by serhiy.storchaka

#23402: dynload_shlib does not close ldl handles when the interpreter 
http://bugs.python.org/issue23402  opened by Adam

#23403: Use pickle protocol 4 by default?
http://bugs.python.org/issue23403  opened by serhiy.storchaka

#23404: 'make touch' does not work with git clones of the source repos
http://bugs.python.org/issue23404  opened by vlee

#23405: Tools/freeze "make" gets missing file error with unix shared b
http://bugs.python.org/issue23405  opened by ned.deily

#23406: interning and list comprehension leads to unexpected behavior
http://bugs.python.org/issue23406  opened by Abraham.Smith

#23407: os.walk always follows Windows junctions
http://bugs.python.org/issue23407  opened by craigh

#23408: Pickle tests use incorrect test data
http://bugs.python.org/issue23408  opened by serhiy.storchaka

#23410: Document more BaseHTTPRequestHandler attributes
http://bugs.python.org/issue23410  opened by vadmium

#23411: Update urllib.parse.__all__
http://bugs.python.org/issue23411  opened by vadmium

#23414: seek(count, whence) accepts bogus whence on windows, python2.7
http://bugs.python.org/issue23414  opened by mattip

#23415: add-to-pydotorg does not support .exe installers for Windows
http://bugs.python.org/issue23415  opened by larry

#23416: Make urllib.parse.SplitResult etc arguments optional
http://bugs.python.org/issue23416  opened by vadmium

#23418: Keep http.server.__all__ up to date
http://bugs.python.org/issue23418  opened by vadmium

#23419: Faster default __reduce__ for classes without __init__
http://bugs.python.org/issue23419  opened by serhiy.storchaka

#23420: python -m cProfile -s fails with non informative message
http://bugs.python.org/issue23420  opened by rkuska

#23422: Clarify docs for importlib.import_module()
http://bugs.python.org/issue23422  opened by brett.cannon

#23423: XPath Support in ElementTree doc omission
http://bugs.python.org/issue23423  opened by mbakeranalecta

#23424: Unicode character ends interactive session
http://bugs.python.org/issue23424  opened by AGrzes

#23425: Windows getlocale unix-like with french, german, portuguese, s
http://bugs.python.org/issue23425  opened by fo...@yahoo.com

#23426: run_setup is broken in distutils
http://bugs.python.org/issue23426  opened by belopolsky

#23427: Python should expose command when invoked with -c
http://bugs.python.org/issue23427  opened by jgehrcke

#23428: Use the monotonic clock for thread conditions on POSIX platfor
http://bugs.python.org/issue23428  opened by haypo

#23430: socketserver.BaseServer.handle_error() should not catch exitin
http://bugs.python.org/issue23430  opened by vadmium

#23431: Idle Application Not Responding
http://bugs.python.org/issue23431  opened by ww0115

#23432: Duplicate content in SystemExit documentation
http://bugs.python.org/issue23432  opened by berker.peksag

#23434: RFC6266 support (Content-Disposition for HTTP)
http://bugs.python.org/issue23434  opened by Myroslav.Opyr

#23436: xml.dom.minidom.Element.ownerDocument is hiden
http://bugs.python.org/issue23436  opened by krocard

#23437: Make user scripts directory versioned on Windows
http://bugs.python.org/issue23437  opened by pmoore

#23439: Fixed http.client.__all__ and added a test
http://bugs.python.org/issue23439  opened by vadmium

#23440: Extend http.server.SimpleHTTPRequestHandler testing
http://bugs.python.org/issue23440  opened by vadmium

#23441: rlcompleter: tab on empty prefix => insert spaces
http://bugs.python.org/issue23441  opened by arigo

#23442: http.client.REQUEST_HEADER_FIELDS_TOO_LARGE renamed in 3.5
http://bugs.python.org/issue23442  opened by vadmium

#23443: XMLRPCLIB Exception uses str not class or instance
http://bugs.python.org/issue23443  opened by kmarsh

#23444: HCI Bluetooth socket bind error on an arm crosscompiled enviro
http://bugs.python.org/issue23444  opened by Thomas.Chiroux

#23446: Use PyMem_New instead of PyMem_Malloc
http://bugs.python.org/issue23446  opened by serhiy.storchaka

#23448: urllib2 needs to remove scope from IPv6 address when creating 
http://bugs.python.org/issue23448  opened by ngierman

#23449: Fatal errors rebuilding 3.5 from Visual Studio Windows 8.1 64 
http://bugs.python.org/issue23449  opened by BreamoreBoy

#23450: Possible loss of data warnings building 3.5 Visual Studio Wind
http://bugs.python.org/issue23450  opened by BreamoreBoy

#23451: Deprecation warnings building 3.5 Visual Studio Windows 8.1 64
http://bugs.python.org/issue23451  opened by BreamoreBoy

#23452: Build errors using VS Express 2

Re: [Python-Dev] subclassing builtin data structures

[Ionel Cristian Mărieș]

Can we at least make it use the constructor (if there's a custom one)?
Seems like a reasonable compromise to me (let whoever implements a custom
__new__ deal with argument variance).

Eg, make it use a __new__ like this:

>>> class FancyInt(int):
... def __new__(self, value):
... return int.__new__(FancyInt, value)
...
... def __repr__(self):
... return "FancyInt(%s)" % super().__repr__()
...
>>> x = FancyInt(1)
>>>
>>> x
FancyInt(1)
>>> x += 1
>>> x   # it should be FancyInt(2)
2



Thanks,
-- Ionel Cristian Mărieș, blog.ionelmc.ro

On Fri, Feb 13, 2015 at 6:01 AM, Guido van Rossum  wrote:

> On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman  wrote:
>
>> On 02/12/2015 06:39 PM, Alexander Belopolsky wrote:
>>
>> > In my view, a constructor is no different from any other method.  If
>> the designers of the subclass decided to change the
>> > signature in an incompatible way, they should either override all
>> methods that create new objects or live with tracebacks.
>>
>> > On the other hand, if all I want in my Date class is a better
>> __format__ method, I am forced to override all operators
>> > or have my objects silently degrade [...]
>>
>> So there are basically two choices:
>>
>> 1) always use the type of the most-base class when creating new instances
>>
>>pros:
>>  - easy
>>  - speedy code
>>  - no possible tracebacks on new object instantiation
>>
>>cons:
>>  - a subclass that needs/wants to maintain itself must override all
>>methods that create new instances, even if the only change is to
>>the type of object returned
>>
>> 2) always use the type of self when creating new instances
>>
>>pros:
>>  - subclasses automatically maintain type
>>  - much less code in the simple cases [1]
>>
>>cons:
>>  - if constructor signatures change, must override all methods which
>>create new objects
>>
>> Unless there are powerful reasons against number 2 (such as performance,
>> or the effort to affect the change), it sure
>> seems like the nicer way to go.
>>
>> So back to my original question: what other concerns are there, and has
>> anybody done any benchmarks?
>>
>
> Con for #2 is a showstopper. Forget about it.
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/contact%40ionelmc.ro
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Guido van Rossum]

I vote for the C implementation.

On Fri, Feb 13, 2015 at 2:07 AM, Victor Stinner 
wrote:

> Hi,
>
> TL,DR: are you ok to add 800 lines of C code for os.scandir(), 4x
> faster than os.listdir() when the file type is checked?
>
> I accepted the PEP 471 (os.scandir) a few months ago, but it is not
> implement yet in Python 3.5, because I didn't make a choice on the
> implementation.
>
> Ben Hoyt wrote different implementations:
> - full C: os.scandir() and DirEntry are written in C (no change on os.py)
> - C+Python: os._scandir() (wrapper for opendir/readdir and
> FindFirstFileW/FindNextFileW) in C, DirEntry in Python
> - ctypes: os.scandir() and DirEntry fully implemented in Python
>
> I'm not interested by the ctypes implementation. It's useful for a
> third party project hosted at PyPI, but for CPython I prefer to wrap C
> functions using C code.
>
>
> In short, the C implementation is faster than the C+Python implementation.
>
> The issue #22524 (*) is full of benchmark numbers. IMO the most
> interesting benchmark is to compare os.listdir() + os.stat() versus
> os.scandir() + Direntry.is_dir(). Let me try to summarize results of
> this benchmark:
>
> * C implementation: scandir is at least 3.5x faster than listdir, up
> to 44.6x faster on Windows
> * C+Python implementation: scandir is not really faster than listdir,
> between 1.3x and 1.4x faster
>
> (*) http://bugs.python.org/issue22524
>
>
> Ben Hoyt reminded me that os.scandir() (PEP 471) doesn't add any new
> feature: pathlib already provides a nice API on top of os and os.path
> modules. (You may even notice that DirEntry a much fewer methods ;-))
> The main (only?) purpose of the PEP is performance.
>
> If os.scandir() is "only" 1.4x faster, I don't think that it is
> interesting to use os.scandir() in an application. I guess that all
> applications/libraries will want to keep compatibility with Python 3.4
> and older and so will anyway have to duplicate the code to use
> os.listdir() + os.stat(). So is it worth to duplicate code for such
> small speedup?
>
> Now I see 3 choices:
>
> - take the full C implementation, because it's much faster (at least
> 3.4x faster!)
> - reject the whole PEP 471 (not nice), because it adds too much code
> for a minor speedup (not true on Windows: up to 44x faster!)
> - take the C+Python implementation, because maintenance matters more
> than performances (only 1.3x faster, sorry)
>
> => IMO the best option is to take the C implementation. What do you think?
>
>
> I'm concerned by the length of the C code: the full C implementations
> adds ~800 lines of C code to posixmodule.c. This file is already the
> longest C file in CPython. I don't want to make it longer, but I'm not
> motived to start to split it. Last time I proposed to split a file
> (unicodeobject.c), some developers complained that it makes search
> harder. I don't understand this, there are so many tools to navigate
> in C code. But it was enough for me to give up on this idea.
>
> A alternative is to add a new _scandir.c module to host the new C
> code, and share some code with posixmodule.c: remove "static" keyword
> from required C functions (functions to convert Windows attributes to
> a os.stat_result object). That's a reasonable choice. What do you
> think?
>
>
> FYI I ran the benchmark on different hardware (SSD, HDD, tmpfs), file
> systems (ext4, tmpfs, NFS/ext4), operating systems (Linux, Windows).
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Guido van Rossum]

On Thu, Feb 12, 2015 at 8:58 PM, Ethan Furman  wrote:

> On 02/12/2015 08:01 PM, Guido van Rossum wrote:
> > On Thu, Feb 12, 2015 at 7:41 PM, Ethan Furman wrote:
> >>
> >> 2) always use the type of self when creating new instances
> >>
> >>cons:
> >>  - if constructor signatures change, must override all methods
> which
> >>create new objects
> >
> > Con for #2 is a showstopper. Forget about it.
>
> Happy to, but can you explain why requiring the programmer to override the
> necessary methods, or get tracebacks, is a
> showstopper?  Is there a previous email thread I can read that discusses
> it?
>

IIUC you're proposing that the base class should *try* to construct an
instance of the subclass by calling the type with an argument, and fail if
it doesn't work. But that makes the whole thing brittle in the light of
changes to the subclass constructor. Also, what should the argument be? The
only answer I can think of is an instance of the base class. Finally, this
would require more special-casing in every built-in class (at least every
built-in class that sometimes returns instances of itself).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Nathaniel Smith]

On 13 Feb 2015 02:09, "Victor Stinner"  wrote:
>
> A alternative is to add a new _scandir.c module to host the new C
> code, and share some code with posixmodule.c: remove "static" keyword
> from required C functions (functions to convert Windows attributes to
> a os.stat_result object).

Hopefully not too annoying question from an outsider: has cpython's build
system added the necessary bits to do this on a safe, portable,
non-symbol-namespace polluting way? E.g. using -fvisibility=hidden on Linux?

(I'm partially wondering because until very recently numpy was built by
concatenating all the different c files together and compiling that,
because that was the only portable way to let different files share access
to symbols without also exporting those symbols publicly from the resulting
module shared objects. And numpy supports a lot fewer platforms than
cpython proper...)

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Fri, Feb 13, 2015 at 12:35 PM, Guido van Rossum  wrote:

> IIUC you're proposing that the base class should *try* to construct an
> instance of the subclass by calling the type with an argument, and fail if
> it doesn't work. But that makes the whole thing brittle in the light of
> changes to the subclass constructor. Also, what should the argument be? The
> only answer I can think of is an instance of the base class.


No.  The arguments should be whatever arguments are appropriate for the
baseclass's __init__ or __new__.  In the case of datetime.date that would
be year, month, day.

Note that the original pure python prototype of the datetime module had
date.__add__ and friends call self.__class__(year, month, day).
Unfortunately, it looks like the original sandbox did not survive the the
hg conversion, so I cannot provide a link to the relevant history.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Guido van Rossum]

On Fri, Feb 13, 2015 at 10:02 AM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

>
> On Fri, Feb 13, 2015 at 12:35 PM, Guido van Rossum 
> wrote:
>
>> IIUC you're proposing that the base class should *try* to construct an
>> instance of the subclass by calling the type with an argument, and fail if
>> it doesn't work. But that makes the whole thing brittle in the light of
>> changes to the subclass constructor. Also, what should the argument be? The
>> only answer I can think of is an instance of the base class.
>
>
> No.  The arguments should be whatever arguments are appropriate for the
> baseclass's __init__ or __new__.  In the case of datetime.date that would
> be year, month, day.
>

Agreed. (I was thinking of the case that Ethan brought up, which used int
as an example.)


> Note that the original pure python prototype of the datetime module had
> date.__add__ and friends call self.__class__(year, month, day).
> Unfortunately, it looks like the original sandbox did not survive the the
> hg conversion, so I cannot provide a link to the relevant history.
>

FWIW you're wrong when you claim that "a constructor is no different from
any other method". Someone else should probably explain this (it's an old
argument that's been thoroughly settled).

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Fri, Feb 13, 2015 at 1:11 PM, Guido van Rossum  wrote:

>
>
>> Note that the original pure python prototype of the datetime module had
>> date.__add__ and friends call self.__class__(year, month, day).
>> Unfortunately, it looks like the original sandbox did not survive the the
>> hg conversion, so I cannot provide a link to the relevant history.
>>
>
> FWIW you're wrong when you claim that "a constructor is no different from
> any other method". Someone else should probably explain this (it's an old
> argument that's been thoroughly settled).
>

Well, the best answer I've got in the past [1] was "ask on python-dev since
Guido called the operator overriding expectation." :-)

[1] http://bugs.python.org/issue2267#msg108060
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:
>>
>> FWIW you're wrong when you claim that "a constructor is no different
from any other method". Someone else should probably explain this (it's an
old argument that's been thoroughly settled).
>
>
> Well, the best answer I've got in the past [1] was "ask on python-dev
since Guido called the operator overriding expectation." :-)


And let me repost this bit of history [1]:

Here is the annotated pre-r82065 code:

 39876 gvanrossum def __add__(self, other):
 39876 gvanrossum if isinstance(other, timedelta):
 39928 gvanrossum return self.__class__(self.__days +
other.__days,
 39876 gvanrossum   self.__seconds +
other.__seconds,
 39876 gvanrossum   self.__microseconds +
other.__microseconds)
 40207tim_one return NotImplemented
 39876 gvanrossum


[1] http://bugs.python.org/issue2267#msg125979
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Guido van Rossum]

Are you willing to wait 10 days for an answer? I'm out of round tuits for a
while.

On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

>
> On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky <
> alexander.belopol...@gmail.com> wrote:
> >>
> >> FWIW you're wrong when you claim that "a constructor is no different
> from any other method". Someone else should probably explain this (it's an
> old argument that's been thoroughly settled).
> >
> >
> > Well, the best answer I've got in the past [1] was "ask on python-dev
> since Guido called the operator overriding expectation." :-)
>
>
> And let me repost this bit of history [1]:
>
> Here is the annotated pre-r82065 code:
>
>  39876 gvanrossum def __add__(self, other):
>  39876 gvanrossum if isinstance(other, timedelta):
>  39928 gvanrossum return self.__class__(self.__days +
> other.__days,
>  39876 gvanrossum   self.__seconds +
> other.__seconds,
>  39876 gvanrossum   self.__microseconds +
> other.__microseconds)
>  40207tim_one return NotImplemented
>  39876 gvanrossum
>
>
> [1] http://bugs.python.org/issue2267#msg125979
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Alexander Belopolsky]

On Fri, Feb 13, 2015 at 5:07 AM, Victor Stinner 
wrote:

> Now I see 3 choices:
>
> - take the full C implementation, because it's much faster (at least
> 3.4x faster!)
> - reject the whole PEP 471 (not nice), because it adds too much code
> for a minor speedup (not true on Windows: up to 44x faster!)
> - take the C+Python implementation, because maintenance matters more
> than performances (only 1.3x faster, sorry)
>
> => IMO the best option is to take the C implementation. What do you think?
>

+1
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments

[Paul Moore]

On 12 February 2015 at 19:44, Paul Moore  wrote:
> Impact on Script Launching
> ==

Now that I'm looking into the details of the code for the launcher,
I've noticed that a shebang line of "#!/usr/bin/env python" will first
of all search PATH for a python executable, before falling back to the
default Python. This is *not* specified in PEP 397, although it makes
sense (as it parallels the behaviour in Unix).

Given that this behaviour exists, doing an explicit check for
VIRTUAL_ENV during shebang processing is unnecessary. The same effect
can be gained just by using "#!/usr/bin/env python" (it won't cover
the case of someone setting VIRTUAL_ENV but *not* adding it to PATH,
but that's no great loss).

I propose simply dropping this section of the PEP. I'll replace it
with a note explaining why it was dropped, and explaining the
behaviour of /usr/bin/env. Is that sufficient?

(By the way, on a procedural note, how do I update a PEP? Do I just
send an updated version to p...@python.org, or is there a better way?)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Isaac Schwabacher]

On 15-02-13, Guido van Rossum  wrote:
> Are you willing to wait 10 days for an answer? I'm out of round tuits for 
> a while.

IIUC, the argument is that the Liskov Substitution Principle is a statement 
about how objects of a subtype behave relative to objects of a supertype, and 
it doesn't apply to constructors because they aren't behaviors of existing 
objects. So other overriding methods *should* be able to handle the same inputs 
that the respective overridden methods do, but constructors don't need to. Even 
though __init__ is written as an instance method, it seems like it's "morally" 
a part of the class method __new__ that's only split off for convenience.

If this message is unclear, it's because I don't really understand this myself 
and I'm trying to articulate my best understanding of what's been said on this 
thread and those it links to.

ijs

> On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky 
>  wrote:
> 
> > 
> > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky 
> >  wrote:
> > >>
> > >> FWIW you're wrong when you claim that "a constructor is no different 
> > >> from any other method". Someone else should probably explain this 
> > >> (it's an old argument that's been thoroughly settled).
> > >
> > >
> > > Well, the best answer I've got in the past [1] was "ask on python-dev 
> > > since Guido called the operator overriding expectation." :-)
> > 
> > 
> > And let me repost this bit of history [1]:
> > 
> > Here is the annotated pre-r82065 code:
> > 
> > 39876 gvanrossum def __add__(self, other):
> > 39876 gvanrossum if isinstance(other, timedelta):
> > 39928 gvanrossum return self.__class__(self.__days + other.__days,
> > 39876 gvanrossum self.__seconds + other.__seconds,
> > 39876 gvanrossum self.__microseconds + other.__microseconds)
> > 40207 tim_one return NotImplemented
> > 39876 gvanrossum
> > 
> > 
> > 
> > [1] http://bugs.python.org/issue2267#msg125979
> > 
> > 
> 
> 
> 
> 
> -- 
> --Guido van Rossum (python.org/~guido(http://python.org/~guido))
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

Interesting:
http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle

On Fri, Feb 13, 2015 at 3:37 PM, Isaac Schwabacher 
wrote:

> On 15-02-13, Guido van Rossum  wrote:
> > Are you willing to wait 10 days for an answer? I'm out of round
> tuits for a while.
>
> IIUC, the argument is that the Liskov Substitution Principle is a
> statement about how objects of a subtype behave relative to objects of a
> supertype, and it doesn't apply to constructors because they aren't
> behaviors of existing objects. So other overriding methods *should* be able
> to handle the same inputs that the respective overridden methods do, but
> constructors don't need to. Even though __init__ is written as an instance
> method, it seems like it's "morally" a part of the class method __new__
> that's only split off for convenience.
>
> If this message is unclear, it's because I don't really understand this
> myself and I'm trying to articulate my best understanding of what's been
> said on this thread and those it links to.
>
> ijs
>
> > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky <
> alexander.belopol...@gmail.com(javascript:main.compose()> wrote:
> >
> > >
> > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky <
> alexander.belopol...@gmail.com(javascript:main.compose()> wrote:
> > > >>
> > > >> FWIW you're wrong when you claim that "a constructor is no
> different from any other method". Someone else should probably explain this
> (it's an old argument that's been thoroughly settled).
> > > >
> > > >
> > > > Well, the best answer I've got in the past [1] was "ask on
> python-dev since Guido called the operator overriding expectation." :-)
> > >
> > >
> > > And let me repost this bit of history [1]:
> > >
> > > Here is the annotated pre-r82065 code:
> > >
> > > 39876 gvanrossum def __add__(self, other):
> > > 39876 gvanrossum if isinstance(other, timedelta):
> > > 39928 gvanrossum return self.__class__(self.__days + other.__days,
> > > 39876 gvanrossum self.__seconds + other.__seconds,
> > > 39876 gvanrossum self.__microseconds + other.__microseconds)
> > > 40207 tim_one return NotImplemented
> > > 39876 gvanrossum
> > >
> > >
> > >
> > > [1] http://bugs.python.org/issue2267#msg125979
> > >
> > >
> >
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido(http://python.org/~guido))
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Isaac Schwabacher]

On 15-02-13, Neil Girdhar  wrote:
> Interesting: 
> http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle

You'd better believe I read that thread not 20 minutes ago. :)

> On Fri, Feb 13, 2015 at 3:37 PM, Isaac Schwabacher 
>  target="1">ischwabac...@wisc.edu> wrote:
> 
> > On 15-02-13, Guido van Rossum wrote:
> > > Are you willing to wait 10 days for an answer? I'm out of round tuits 
> > > for a while.
> > 
> > IIUC, the argument is that the Liskov Substitution Principle is a statement 
> > about how objects of a subtype behave relative to objects of a supertype, 
> > and it doesn't apply to constructors because they aren't behaviors 
> > of existing objects. So other overriding methods *should* be able to handle 
> > the same inputs that the respective overridden methods do, but constructors 
> > don't need to. Even though __init__ is written as an instance method, 
> > it seems like it's "morally" a part of the class method __new__ 
> > that's only split off for convenience.
> > 
> > If this message is unclear, it's because I don't really understand 
> > this myself and I'm trying to articulate my best understanding of 
> > what's been said on this thread and those it links to.
> > 
> > ijs
> > 
> > > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky 
> > >  > >  't=alexander.belopol...@gmail.com>(java_script:main.compose()> wrote:
> > >
> > > >
> > > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky 
> > > >  > > > (java_script:main.compose()> wrote:
> > > > >>
> > > > >> FWIW you're wrong when you claim that "a constructor is no 
> > > > >> different from any other method". Someone else should probably 
> > > > >> explain this (it's an old argument that's been thoroughly 
> > > > >> settled).
> > > > >
> > > > >
> > > > > Well, the best answer I've got in the past [1] was "ask on 
> > > > > python-dev since Guido called the operator overriding expectation." 
> > > > > :-)
> > > >
> > > >
> > > > And let me repost this bit of history [1]:
> > > >
> > > > Here is the annotated pre-r82065 code:
> > > >
> > > > 39876 gvanrossum def __add__(self, other):
> > > > 39876 gvanrossum if isinstance(other, timedelta):
> > > > 39928 gvanrossum return self.__class__(self.__days + other.__days,
> > > > 39876 gvanrossum self.__seconds + other.__seconds,
> > > > 39876 gvanrossum self.__microseconds + other.__microseconds)
> > > > 40207 tim_one return NotImplemented
> > > > 39876 gvanrossum
> > > >
> > > >
> > > >
> > > > [1] > --Guido van Rossum 
> > > > (python.org/~guido(http://python.org/~guido)(http://python.org/~guido(http://python.org/~guido)))
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org 
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe: 
> > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
> >
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (no subject)

[Guido van Rossum]

I am still in favor of this PEP but have run out of time to review it and
the feedback. I'm going on vacation for a week or so, maybe I'll find time,
if not I'll start reviewing this around Feb 23.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Building on Windows - importlib.h changed

[Paul Moore]

I'm working on a patch for the Python launcher. I built Python
(current tip, on MS Windows, with VS 2015), and I've just noticed that
hg status shows:

>>hg status -mard
M Doc\using\windows.rst
M PC\launcher.c
M Python\importlib.h

I didn't change importlib.h, and I don't see that the changes I did
make would affect importlib. I presume I'm OK *not* including the
importlib.h change in my patch?

Paul

PS By the way, the Windows build process is beautifully clean these
days. Many thanks to Steve Dower and whoever else has worked on
streamlining it!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Serhiy Storchaka]

On 13.02.15 05:41, Ethan Furman wrote:

So there are basically two choices:

1) always use the type of the most-base class when creating new instances

pros:
  - easy
  - speedy code
  - no possible tracebacks on new object instantiation

cons:
  - a subclass that needs/wants to maintain itself must override all
methods that create new instances, even if the only change is to
the type of object returned

2) always use the type of self when creating new instances

pros:
  - subclasses automatically maintain type
  - much less code in the simple cases [1]

cons:
  - if constructor signatures change, must override all methods which
create new objects


And switching to (2) would break existing code which uses subclasses 
with constructors with different signature (e.g. defaultdict).


The third choice is to use different specially designed constructor.

class A(int):


class A(int):
... def __add__(self, other): 

... return self.__make_me__(int(self) + int(other)) 



... def __repr__(self): 


... return 'A(%d)' % self
...

A.__make_me__ = A
A(2) + 3

A(5)

class B(A):

... def __repr__(self):
... return 'B(%d)' % self
...

B.__make_me__ = B
B(2) + 3

B(5)

We can add special attribute used to creating results of operations to 
all basic classes. By default it would be equal to the base class 
constructor.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building on Windows - importlib.h changed

[Zachary Ware]

On Fri, Feb 13, 2015 at 4:09 PM, Paul Moore  wrote:
> I'm working on a patch for the Python launcher. I built Python
> (current tip, on MS Windows, with VS 2015), and I've just noticed that
> hg status shows:
>
>>>hg status -mard
> M Doc\using\windows.rst
> M PC\launcher.c
> M Python\importlib.h
>
> I didn't change importlib.h, and I don't see that the changes I did
> make would affect importlib. I presume I'm OK *not* including the
> importlib.h change in my patch?

Yes, importlib.h changes should never be included in a patch (it would
make the patch extremely huge for no benefit).  It's strange that
you're getting modifications, though; I can't reproduce on Linux (and
don't have quick access to Windows right now).  Make sure you're
definitely at tip, and if it's still changing please open an issue.

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar  wrote:

> Interesting:
> http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle
>

Let me humbly conjecture that the people who wrote the top answers have
background in less capable languages than Python.

Not every language allows you to call self.__class__().  In the languages
that don't you can get away with incompatible constructor signatures.

However, let me try to focus the discussion on a specific issue before we
go deep into OOP theory.

With python's standard datetime.date we have:

>>> from datetime import *
>>> class Date(date):
... pass
...
>>> Date.today()
Date(2015, 2, 13)
>>> Date.fromordinal(1)
Date(1, 1, 1)

Both .today() and .fromordinal(1) will break in a subclass that redefines
__new__ as follows:

>>> class Date2(date):
... def __new__(cls, ymd):
... return date.__new__(cls, *ymd)
...
>>> Date2.today()
Traceback (most recent call last):
  File "", line 1, in 
TypeError: __new__() takes 2 positional arguments but 4 were given
>>> Date2.fromordinal(1)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: __new__() takes 2 positional arguments but 4 were given

Why is this acceptable, but we have to sacrifice the convenience of having
Date + timedelta
return Date to make it work  with Date2:

>>> Date2((1,1,1)) + timedelta(1)
datetime.date(1, 1, 2)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building on Windows - importlib.h changed

[Eric Snow]

On Fri, Feb 13, 2015 at 3:49 PM, Zachary Ware
 wrote:
> On Fri, Feb 13, 2015 at 4:09 PM, Paul Moore  wrote:
>> I'm working on a patch for the Python launcher. I built Python
>> (current tip, on MS Windows, with VS 2015), and I've just noticed that
>> hg status shows:
>>
hg status -mard
>> M Doc\using\windows.rst
>> M PC\launcher.c
>> M Python\importlib.h
>>
>> I didn't change importlib.h, and I don't see that the changes I did
>> make would affect importlib. I presume I'm OK *not* including the
>> importlib.h change in my patch?
>
> Yes, importlib.h changes should never be included in a patch (it would

Unless they should. :)  E.g. you modified importlib/_bootstrap.py, the
marshal format, bytecodes, etc.

-eric
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

I personally don't think this is a big enough issue to warrant any changes,
but I think Serhiy's solution would be the ideal best with one additional
parameter: the caller's type.  Something like

def __make_me__(self, cls, *args, **kwargs)

and the idea is that any time you want to construct a type, instead of

self.__class__(assumed arguments…)

where you are not sure that the derived class' constructor knows the right
argument types, you do

def SomeCls:
 def some_method(self, ...):
   return self.__make_me__(SomeCls, assumed arguments…)

Now the derived class knows who is asking for a copy.  In the case of
defaultdict, for example, he can implement __make_me__ as follows:

def __make_me__(self, cls, *args, **kwargs):
if cls is dict: return default_dict(self.default_factory, *args,
**kwargs)
return default_dict(*args, **kwargs)

essentially the caller is identifying himself so that the receiver knows
how to interpret the arguments.

Best,

Neil

On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com> wrote:

>
> On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar 
> wrote:
>
>> Interesting:
>> http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle
>>
>
> Let me humbly conjecture that the people who wrote the top answers have
> background in less capable languages than Python.
>
> Not every language allows you to call self.__class__().  In the languages
> that don't you can get away with incompatible constructor signatures.
>
> However, let me try to focus the discussion on a specific issue before we
> go deep into OOP theory.
>
> With python's standard datetime.date we have:
>
> >>> from datetime import *
> >>> class Date(date):
> ... pass
> ...
> >>> Date.today()
> Date(2015, 2, 13)
> >>> Date.fromordinal(1)
> Date(1, 1, 1)
>
> Both .today() and .fromordinal(1) will break in a subclass that redefines
> __new__ as follows:
>
> >>> class Date2(date):
> ... def __new__(cls, ymd):
> ... return date.__new__(cls, *ymd)
> ...
> >>> Date2.today()
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: __new__() takes 2 positional arguments but 4 were given
> >>> Date2.fromordinal(1)
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: __new__() takes 2 positional arguments but 4 were given
>
> Why is this acceptable, but we have to sacrifice the convenience of having
> Date + timedelta
> return Date to make it work  with Date2:
>
> >>> Date2((1,1,1)) + timedelta(1)
> datetime.date(1, 1, 2)
>
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building on Windows - importlib.h changed

[Zachary Ware]

On Fri, Feb 13, 2015 at 4:56 PM, Eric Snow  wrote:
> On Fri, Feb 13, 2015 at 3:49 PM, Zachary Ware
>  wrote:
>> Yes, importlib.h changes should never be included in a patch (it would
>
> Unless they should. :)  E.g. you modified importlib/_bootstrap.py, the
> marshal format, bytecodes, etc.

If I'm not mistaken, the devguide suggests leaving generated files out
of patches, though I can't find where to link to it.  If not, it
probably should :).  Generated files ought to be generated by the
build process the same way on any system, so leaving them in the patch
amounts to unnecessary noise, especially in the case of importlib.h
which is not human-readable (and huge).  In the case of a generated
file being affected by a patch, a friendly reminder that files will
need to be regenerated should be included when posting the patch,
though.

-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Building on Windows - importlib.h changed

[Paul Moore]

On 13 February 2015 at 22:49, Zachary Ware  wrote:
> It's strange that
> you're getting modifications, though; I can't reproduce on Linux (and
> don't have quick access to Windows right now).  Make sure you're
> definitely at tip, and if it's still changing please open an issue.

Definitely at tip. I just redid the build and confirmed it's
reproducible. There is a message generated during the build that says
importlib.h has changed.

Reported at http://bugs.python.org/issue23461

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

Unlike a regular method, you would never need to call super since you
should know everyone that could be calling you.  Typically, when you call
super, you have something like this:

A < B, C

B < D

so you end up with

mro: A, B, C, D

And then when A calls super and B calls super it gets C which it doesn't
know about.

But in the case of make_me, it's someone like C who  is calling make_me.
If it gets a method in B, then that's a straight-up bug.  make_me needs to
be reimplemented in A as well, and A would never delegate up since other
classes in the mro chain (like B) might not know about C.

Best,
Neil

On Fri, Feb 13, 2015 at 7:00 PM, Isaac Schwabacher 
wrote:

> On 15-02-13, Neil Girdhar  wrote:
> > I personally don't think this is a big enough issue to warrant any
> changes, but I think Serhiy's solution would be the ideal best with one
> additional parameter: the caller's type. Something like
> >
> > def __make_me__(self, cls, *args, **kwargs)
> >
> >
> > and the idea is that any time you want to construct a type, instead of
> >
> >
> > self.__class__(assumed arguments…)
> >
> >
> > where you are not sure that the derived class' constructor knows the
> right argument types, you do
> >
> >
> > def SomeCls:
> > def some_method(self, ...):
> > return self.__make_me__(SomeCls, assumed arguments…)
> >
> >
> > Now the derived class knows who is asking for a copy. In the case of
> defaultdict, for example, he can implement __make_me__ as follows:
> >
> >
> > def __make_me__(self, cls, *args, **kwargs):
> > if cls is dict: return default_dict(self.default_factory, *args,
> **kwargs)
> > return default_dict(*args, **kwargs)
> >
> >
> > essentially the caller is identifying himself so that the receiver knows
> how to interpret the arguments.
> >
> >
> > Best,
> >
> >
> > Neil
>
> Such a method necessarily involves explicit switching on classes... ew.
> Also, to make this work, a class needs to have a relationship with its
> superclass's superclasses. So in order for DefaultDict's subclasses not to
> need to know about dict, it would need to look like this:
>
> class DefaultDict(dict):
> @classmethod # instance method doesn't make sense here
> def __make_me__(cls, base, *args, **kwargs): # make something like
> base(*args, **kwargs)
> # when we get here, nothing in cls.__mro__ above DefaultDict knows
> how to construct an equivalent to base(*args, **kwargs) using its own
> constructor
> if base is DefaultDict:
> return DefaultDict(*args, **kwargs) # if DefaultDict is the
> best we can do, do it
> elif base is dict:
> return cls.__make_me__(DefaultDict, None, *args, **kwargs) #
> subclasses that know about DefaultDict but not dict will intercept this
> else:
> super(DefaultDict, cls).__make_me__(base, *args, **kwargs) #
> we don't know how to make an equivalent to base.__new__(*args, **kwargs),
> so keep looking
>
> I don't even think this is guaranteed to construct an object of class cls
> corresponding to a base(*args, **kwargs) even if it were possible, since
> multiple inheritance can screw things up. You might need to have an
> explicit list of "these are the superclasses whose constructors I can
> imitate", and have the interpreter find an optimal path for you.
>
> > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky <
> alexander.belopol...@gmail.com(javascript:main.compose()> wrote:
> >
> > >
> > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar 
> > > 
> wrote:
> > >
> > > > Interesting:
> http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle
> > > >
> > >
> > >
> > > Let me humbly conjecture that the people who wrote the top answers
> have background in less capable languages than Python.
> > >
> > >
> > > Not every language allows you to call self.__class__(). In the
> languages that don't you can get away with incompatible constructor
> signatures.
> > >
> > >
> > > However, let me try to focus the discussion on a specific issue before
> we go deep into OOP theory.
> > >
> > >
> > > With python's standard datetime.date we have:
> > >
> > >
> > > >>> from datetime import *
> > > >>> class Date(date):
> > > ... pass
> > > ...
> > > >>> Date.today()
> > > Date(2015, 2, 13)
> > > >>> Date.fromordinal(1)
> > > Date(1, 1, 1)
> > >
> > >
> > > Both .today() and .fromordinal(1) will break in a subclass that
> redefines __new__ as follows:
> > >
> > >
> > > >>> class Date2(date):
> > > ... def __new__(cls, ymd):
> > > ... return date.__new__(cls, *ymd)
> > > ...
> > > >>> Date2.today()
> > > Traceback (most recent call last):
> > > File "", line 1, in 
> > > TypeError: __new__() takes 2 positional arguments but 4 were given
> > > >>> Date2.fromordinal(1)
> > > Traceback (most recent call last):
> > > File "", line 1, in 
> > > TypeError: __new__() takes 2 positional arguments but 4 were given
> > >
> > >
> > >
> > >
> > > Why is this acceptable, but we have to sacrifice the conv

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Nick Coghlan]

On 14 Feb 2015 01:47, "Paul Moore"  wrote:
>
> On 13 February 2015 at 14:27, Steve Dower 
wrote:
> > If py.exe starts defaulting to whatever is on PATH then I don't see the
> > point of it. Knowing that you'll get the latest 2.x version by default
is
> > quite handy (I'd be quite okay changing that to 3.x, though it is
specified
> > in the original PEP). For me, the point of py.exe is to be able to
ignore
> > PATH completely.
>
> It does seem a bit bizarre to me that a launcher shipped with Python 3
> defaults to using Python 2 if it's available. I'd like to see that
> change, but as it's specified in the original PEP I guess it might
> meet with some pushback :-(

If there's no Python 2 on the system it doesn't matter, and if they're both
there, we opted to remain consistent with the Linux distro world.

OTOH, it may be time to reconsider our recommendation to distros as well,
suggesting that for Python 3.5+, we will consider it appropriate to have
the "python" symlink refer to "python3".

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Greg Ewing]

Isaac Schwabacher wrote:

IIUC, the argument is that the Liskov Substitution Principle is a statement
about how objects of a subtype behave relative to objects of a supertype, and
it doesn't apply to constructors because they aren't behaviors of existing
objects.


Another way to say that is that constructors are class
methods, not instance methods.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Isaac Schwabacher]

On 15-02-13, Neil Girdhar  wrote:
> Unlike a regular method, you would never need to call super since you should 
> know everyone that could be calling you. Typically, when you call super, you 
> have something like this:
> 
> A < B, C
> 
> 
> B < D
> 
> 
> so you end up with 
> 
> 
> mro: A, B, C, D
> 
> 
> And then when A calls super and B calls super it gets C which it doesn't 
> know about.

But C calls super and gets D. The scenario I'm concerned with is that A knows 
how to mimic B's constructor and B knows how to mimic D's, but A doesn't know 
about D. So D asks A if it knows how to mimic D's constructor, and it says no. 
Via super, B gets a shot, and it does know, so it translates the arguments to 
D's constructor into arguments to B's constructor, and again asks A if it knows 
how to handle them. Then A says yes, translates the args, and constructs an A. 
If C ever gets consulted, it responds "I don't know a thing" and calls super.

> But in the case of make_me, it's someone like C who is calling make_me. 
> If it gets a method in B, then that's a straight-up bug. make_me needs to 
> be reimplemented in A as well, and A would never delegate up since other 
> classes in the mro chain (like B) might not know about C.

This scheme (as I've written it) depends strongly on all the classes in the MRO 
having __make_me__ methods with this very precisely defined structure: test 
base against yourself, then any superclasses you care to mimic, then call 
super. Any antisocial superclass ruins everyone's party.

> Best,
> Neil
> 
> 
> On Fri, Feb 13, 2015 at 7:00 PM, Isaac Schwabacher 
>  target="1">ischwabac...@wisc.edu> wrote:
> 
> > On 15-02-13, Neil Girdhar wrote:
> > > I personally don't think this is a big enough issue to warrant any 
> > > changes, but I think Serhiy's solution would be the ideal best with 
> > > one additional parameter: the caller's type. Something like
> > >
> > > def __make_me__(self, cls, *args, **kwargs)
> > >
> > >
> > > and the idea is that any time you want to construct a type, instead of
> > >
> > >
> > > self.__class__(assumed arguments…)
> > >
> > >
> > > where you are not sure that the derived class' constructor knows the 
> > > right argument types, you do
> > >
> > >
> > > def SomeCls:
> > > def some_method(self, ...):
> > > return self.__make_me__(SomeCls, assumed arguments…)
> > >
> > >
> > > Now the derived class knows who is asking for a copy. In the case of 
> > > defaultdict, for example, he can implement __make_me__ as follows:
> > >
> > >
> > > def __make_me__(self, cls, *args, **kwargs):
> > > if cls is dict: return default_dict(self.default_factory, *args, **kwargs)
> > > return default_dict(*args, **kwargs)
> > >
> > >
> > > essentially the caller is identifying himself so that the receiver knows 
> > > how to interpret the arguments.
> > >
> > >
> > > Best,
> > >
> > >
> > > Neil
> > 
> > Such a method necessarily involves explicit switching on classes... ew.
> > Also, to make this work, a class needs to have a relationship with its 
> > superclass's superclasses. So in order for DefaultDict's subclasses 
> > not to need to know about dict, it would need to look like this:
> > 
> > class DefaultDict(dict):
> > @classmethod # instance method doesn't make sense here
> > def __make_me__(cls, base, *args, **kwargs): # make something like 
> > base(*args, **kwargs)
> > # when we get here, nothing in cls.__mro__ above DefaultDict knows 
> > how to construct an equivalent to base(*args, **kwargs) using its own 
> > constructor
> > if base is DefaultDict:
> > return DefaultDict(*args, **kwargs) # if DefaultDict is the 
> > best we can do, do it
> > elif base is dict:
> > return cls.__make_me__(DefaultDict, None, *args, **kwargs) # 
> > subclasses that know about DefaultDict but not dict will intercept this
> > else:
> > super(DefaultDict, cls).__make_me__(base, *args, **kwargs) # we 
> > don't know how to make an equivalent to base.__new__(*args, **kwargs), 
> > so keep looking
> > 
> > I don't even think this is guaranteed to construct an object of class 
> > cls corresponding to a base(*args, **kwargs) even if it were possible, 
> > since multiple inheritance can screw things up. You might need to have an 
> > explicit list of "these are the superclasses whose constructors I can 
> > imitate", and have the interpreter find an optimal path for you.
> > 
> > > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky 
> > >  > >  't=alexander.belopol...@gmail.com>(java_script:main.compose()> wrote:
> > >
> > > >
> > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar  > > > (java_script:main.compose()> wrote:
> > > >
> > > > > Interesting: > > Not every language allows you to call 
> > > > > self.__class__(). In the languages that don't you can get away 
> >

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

You're right.

On Fri, Feb 13, 2015 at 7:55 PM, Isaac Schwabacher 
wrote:

> On 15-02-13, Neil Girdhar  wrote:
> > Unlike a regular method, you would never need to call super since you
> should know everyone that could be calling you. Typically, when you call
> super, you have something like this:
> >
> > A < B, C
> >
> >
> > B < D
> >
> >
> > so you end up with
> >
> >
> > mro: A, B, C, D
> >
> >
> > And then when A calls super and B calls super it gets C which it
> doesn't know about.
>
> But C calls super and gets D. The scenario I'm concerned with is that A
> knows how to mimic B's constructor and B knows how to mimic D's, but A
> doesn't know about D. So D asks A if it knows how to mimic D's constructor,
> and it says no. Via super, B gets a shot, and it does know, so it
> translates the arguments to D's constructor into arguments to B's
> constructor, and again asks A if it knows how to handle them. Then A says
> yes, translates the args, and constructs an A. If C ever gets consulted, it
> responds "I don't know a thing" and calls super.
>
> > But in the case of make_me, it's someone like C who is calling
> make_me. If it gets a method in B, then that's a straight-up bug.
> make_me needs to be reimplemented in A as well, and A would never delegate
> up since other classes in the mro chain (like B) might not know about C.
>
> This scheme (as I've written it) depends strongly on all the classes in
> the MRO having __make_me__ methods with this very precisely defined
> structure: test base against yourself, then any superclasses you care to
> mimic, then call super. Any antisocial superclass ruins everyone's party.
>
> > Best,
> > Neil
> >
> >
> > On Fri, Feb 13, 2015 at 7:00 PM, Isaac Schwabacher <
> alexander.belopol...@gmail.com 
> ischwabac...@wisc.edu> wrote:
> >
> > > On 15-02-13, Neil Girdhar wrote:
> > > > I personally don't think this is a big enough issue to warrant
> any changes, but I think Serhiy's solution would be the ideal best with
> one additional parameter: the caller's type. Something like
> > > >
> > > > def __make_me__(self, cls, *args, **kwargs)
> > > >
> > > >
> > > > and the idea is that any time you want to construct a type, instead
> of
> > > >
> > > >
> > > > self.__class__(assumed arguments…)
> > > >
> > > >
> > > > where you are not sure that the derived class' constructor knows
> the right argument types, you do
> > > >
> > > >
> > > > def SomeCls:
> > > > def some_method(self, ...):
> > > > return self.__make_me__(SomeCls, assumed arguments…)
> > > >
> > > >
> > > > Now the derived class knows who is asking for a copy. In the case of
> defaultdict, for example, he can implement __make_me__ as follows:
> > > >
> > > >
> > > > def __make_me__(self, cls, *args, **kwargs):
> > > > if cls is dict: return default_dict(self.default_factory, *args,
> **kwargs)
> > > > return default_dict(*args, **kwargs)
> > > >
> > > >
> > > > essentially the caller is identifying himself so that the receiver
> knows how to interpret the arguments.
> > > >
> > > >
> > > > Best,
> > > >
> > > >
> > > > Neil
> > >
> > > Such a method necessarily involves explicit switching on classes... ew.
> > > Also, to make this work, a class needs to have a relationship with its
> superclass's superclasses. So in order for DefaultDict's subclasses
> not to need to know about dict, it would need to look like this:
> > >
> > > class DefaultDict(dict):
> > > @classmethod # instance method doesn't make sense here
> > > def __make_me__(cls, base, *args, **kwargs): # make something like
> base(*args, **kwargs)
> > > # when we get here, nothing in cls.__mro__ above DefaultDict
> knows how to construct an equivalent to base(*args, **kwargs) using its own
> constructor
> > > if base is DefaultDict:
> > > return DefaultDict(*args, **kwargs) # if DefaultDict is
> the best we can do, do it
> > > elif base is dict:
> > > return cls.__make_me__(DefaultDict, None, *args, **kwargs)
> # subclasses that know about DefaultDict but not dict will intercept this
> > > else:
> > > super(DefaultDict, cls).__make_me__(base, *args, **kwargs)
> # we don't know how to make an equivalent to base.__new__(*args,
> **kwargs), so keep looking
> > >
> > > I don't even think this is guaranteed to construct an object of
> class cls corresponding to a base(*args, **kwargs) even if it were
> possible, since multiple inheritance can screw things up. You might need to
> have an explicit list of "these are the superclasses whose constructors I
> can imitate", and have the interpreter find an optimal path for you.
> > >
> > > > On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky <
> http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle(javascript:main.compose('new',
> 't=alexander.belopol...@gmail.com>(java_script:main.compose()> wrote:
> > > >
> > > > >
> > > > > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar <
> misters

Re: [Python-Dev] subclassing builtin data structures

[Ethan Furman]

On 02/13/2015 02:31 PM, Serhiy Storchaka wrote:
> On 13.02.15 05:41, Ethan Furman wrote:
>> So there are basically two choices:
>>
>> 1) always use the type of the most-base class when creating new instances
>>
>> pros:
>>   - easy
>>   - speedy code
>>   - no possible tracebacks on new object instantiation
>>
>> cons:
>>   - a subclass that needs/wants to maintain itself must override all
>> methods that create new instances, even if the only change is to
>> the type of object returned
>>
>> 2) always use the type of self when creating new instances
>>
>> pros:
>>   - subclasses automatically maintain type
>>   - much less code in the simple cases [1]
>>
>> cons:
>>   - if constructor signatures change, must override all methods which
>> create new objects
> 
> And switching to (2) would break existing code which uses subclasses with 
> constructors with different signature (e.g.
> defaultdict).

I don't think defaultdict is a good example -- I don't see any methods on it 
that return a new dict, default or
otherwise. So if this change happened, defaultdict would have to have its own 
__add__ and not rely on dict's __add__.


> The third choice is to use different specially designed constructor.
> 
> class A(int):
> 
> --> class A(int):
> ... def __add__(self, other):
> ... return self.__make_me__(int(self) + int(other))
> 
> ... def __repr__(self):
> ... return 'A(%d)' % self

How would this help in the case of defaultdict?  __make_me__ is a class method, 
but it needs instance info to properly
create a new dict with the same default factory.

--
~Ethan~



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

I think it works as Isaac explained if __make_me__ is an instance method
that also accepts the calling class type.

On Fri, Feb 13, 2015 at 8:12 PM, Ethan Furman  wrote:

> On 02/13/2015 02:31 PM, Serhiy Storchaka wrote:
> > On 13.02.15 05:41, Ethan Furman wrote:
> >> So there are basically two choices:
> >>
> >> 1) always use the type of the most-base class when creating new
> instances
> >>
> >> pros:
> >>   - easy
> >>   - speedy code
> >>   - no possible tracebacks on new object instantiation
> >>
> >> cons:
> >>   - a subclass that needs/wants to maintain itself must override all
> >> methods that create new instances, even if the only change is to
> >> the type of object returned
> >>
> >> 2) always use the type of self when creating new instances
> >>
> >> pros:
> >>   - subclasses automatically maintain type
> >>   - much less code in the simple cases [1]
> >>
> >> cons:
> >>   - if constructor signatures change, must override all methods
> which
> >> create new objects
> >
> > And switching to (2) would break existing code which uses subclasses
> with constructors with different signature (e.g.
> > defaultdict).
>
> I don't think defaultdict is a good example -- I don't see any methods on
> it that return a new dict, default or
> otherwise. So if this change happened, defaultdict would have to have its
> own __add__ and not rely on dict's __add__.
>
>
> > The third choice is to use different specially designed constructor.
> >
> > class A(int):
> >
> > --> class A(int):
> > ... def __add__(self, other):
> > ... return self.__make_me__(int(self) + int(other))
> >
> > ... def __repr__(self):
> > ... return 'A(%d)' % self
>
> How would this help in the case of defaultdict?  __make_me__ is a class
> method, but it needs instance info to properly
> create a new dict with the same default factory.
>
> --
> ~Ethan~
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Isaac Schwabacher]

On 15-02-13, Neil Girdhar  wrote:
> I personally don't think this is a big enough issue to warrant any 
> changes, but I think Serhiy's solution would be the ideal best with one 
> additional parameter: the caller's type. Something like
> 
> def __make_me__(self, cls, *args, **kwargs)
> 
> 
> and the idea is that any time you want to construct a type, instead of 
> 
> 
> self.__class__(assumed arguments…)
> 
> 
> where you are not sure that the derived class' constructor knows the 
> right argument types, you do
> 
> 
> def SomeCls:
> def some_method(self, ...):
> return self.__make_me__(SomeCls, assumed arguments…)
> 
> 
> Now the derived class knows who is asking for a copy. In the case of 
> defaultdict, for example, he can implement __make_me__ as follows:
> 
> 
> def __make_me__(self, cls, *args, **kwargs):
> if cls is dict: return default_dict(self.default_factory, *args, **kwargs)
> return default_dict(*args, **kwargs)
> 
> 
> essentially the caller is identifying himself so that the receiver knows how 
> to interpret the arguments.
> 
> 
> Best,
> 
> 
> Neil

Such a method necessarily involves explicit switching on classes... ew.
Also, to make this work, a class needs to have a relationship with its 
superclass's superclasses. So in order for DefaultDict's subclasses not to need 
to know about dict, it would need to look like this:

class DefaultDict(dict):
@classmethod # instance method doesn't make sense here
def __make_me__(cls, base, *args, **kwargs): # make something like 
base(*args, **kwargs)
# when we get here, nothing in cls.__mro__ above DefaultDict knows how 
to construct an equivalent to base(*args, **kwargs) using its own constructor
if base is DefaultDict:
return DefaultDict(*args, **kwargs) # if DefaultDict is the best we 
can do, do it
elif base is dict:
return cls.__make_me__(DefaultDict, None, *args, **kwargs) # 
subclasses that know about DefaultDict but not dict will intercept this
else:
super(DefaultDict, cls).__make_me__(base, *args, **kwargs) # we 
don't know how to make an equivalent to base.__new__(*args, **kwargs), so keep 
looking

I don't even think this is guaranteed to construct an object of class cls 
corresponding to a base(*args, **kwargs) even if it were possible, since 
multiple inheritance can screw things up. You might need to have an explicit 
list of "these are the superclasses whose constructors I can imitate", and have 
the interpreter find an optimal path for you.

> On Fri, Feb 13, 2015 at 5:55 PM, Alexander Belopolsky 
>  wrote:
> 
> > 
> > On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar 
> >  wrote:
> > 
> > > Interesting: 
> > > http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle
> > > 
> > 
> > 
> > Let me humbly conjecture that the people who wrote the top answers have 
> > background in less capable languages than Python.
> > 
> > 
> > Not every language allows you to call self.__class__(). In the languages 
> > that don't you can get away with incompatible constructor signatures.
> > 
> > 
> > However, let me try to focus the discussion on a specific issue before we 
> > go deep into OOP theory.
> > 
> > 
> > With python's standard datetime.date we have:
> > 
> > 
> > >>> from datetime import *
> > >>> class Date(date):
> > ... pass
> > ...
> > >>> Date.today()
> > Date(2015, 2, 13)
> > >>> Date.fromordinal(1)
> > Date(1, 1, 1)
> > 
> > 
> > Both .today() and .fromordinal(1) will break in a subclass that redefines 
> > __new__ as follows:
> > 
> > 
> > >>> class Date2(date):
> > ... def __new__(cls, ymd):
> > ... return date.__new__(cls, *ymd)
> > ...
> > >>> Date2.today()
> > Traceback (most recent call last):
> > File "", line 1, in 
> > TypeError: __new__() takes 2 positional arguments but 4 were given
> > >>> Date2.fromordinal(1)
> > Traceback (most recent call last):
> > File "", line 1, in 
> > TypeError: __new__() takes 2 positional arguments but 4 were given
> > 
> > 
> > 
> > 
> > Why is this acceptable, but we have to sacrifice the convenience of having 
> > Date + timedelta
> > return Date to make it work with Date2:
> > 
> > 
> > >>> Date2((1,1,1)) + timedelta(1)
> > datetime.date(1, 1, 2)
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Isaac Schwabacher]

Hmm... super *is* a problem, because if we super, we can end up asking 
superclasses that we then won't know how to mimic. So what we really want is to 
move the superclasses we can mimic to the front of the MRO. If none of those 
can indirectly mimic the base class, then we try the other classes in the MRO 
to see if we can get a partial upgrade.

On 15-02-13, Neil Girdhar  wrote:
> I think it works as Isaac explained if __make_me__ is an instance method that 
> also accepts the calling class type.
> 
> On Fri, Feb 13, 2015 at 8:12 PM, Ethan Furman  et...@stoneleaf.us> wrote:
> 
> > On 02/13/2015 02:31 PM, Serhiy Storchaka wrote:
> > > On 13.02.15 05:41, Ethan Furman wrote:
> > >> So there are basically two choices:
> > >>
> > >> 1) always use the type of the most-base class when creating new instances
> > >>
> > >> pros:
> > >> - easy
> > >> - speedy code
> > >> - no possible tracebacks on new object instantiation
> > >>
> > >> cons:
> > >> - a subclass that needs/wants to maintain itself must override all
> > >> methods that create new instances, even if the only change is to
> > >> the type of object returned
> > >>
> > >> 2) always use the type of self when creating new instances
> > >>
> > >> pros:
> > >> - subclasses automatically maintain type
> > >> - much less code in the simple cases [1]
> > >>
> > >> cons:
> > >> - if constructor signatures change, must override all methods which
> > >> create new objects
> > >
> > > And switching to (2) would break existing code which uses subclasses with 
> > > constructors with different signature (e.g.
> > > defaultdict).
> > 
> > I don't think defaultdict is a good example -- I don't see any 
> > methods on it that return a new dict, default or
> > otherwise. So if this change happened, defaultdict would have to have its 
> > own __add__ and not rely on dict's __add__.
> > 
> > 
> > > The third choice is to use different specially designed constructor.
> > >
> > > class A(int):
> > >
> > > --> class A(int):
> > > ... def __add__(self, other):
> > > ... return self.__make_me__(int(self) + int(other))
> > >
> > > ... def __repr__(self):
> > > ... return 'A(%d)' % self
> > 
> > How would this help in the case of defaultdict? __make_me__ is a class 
> > method, but it needs instance info to properly
> > create a new dict with the same default factory.
> > 
> > --
> > ~Ethan~
> > 
> > 
> > ___
> > Python-Dev mailing list
> > https://mail.python.org/mailman/listinfo/python-dev(javascript:main.compose('new',
> >  't=Python-Dev@python.org>
> >  > Unsubscribe: 
> > https://mail.python.org/mailman/options/python-dev/mistersheik%40gmail.com
> > 
> >
import inspect

class A:
def __init__(self, bar):
self.bar = bar
def foo(self):
return self.__class__.__make_me__(A, self.bar)
@classmethod
def __make_me__(cls, base, *args, **kwargs):
if base is A:
return A(*args, **kwargs)
else:
return super(A, cls).__make_me__(base, *args, **kwargs)

class B(A):
def __init__(self, bar, baz):
self.bar = bar
self.baz = baz
@classmethod
def __make_me__(cls, base, *args, **kwargs):
if base is B:
return B(*args, **kwargs)
elif base is A:
bound = inspect.signature(A).bind(*args, **kwargs)
return cls.__make_me__(B, bound.arguments['bar'], None)
else:
super(A, cls).__make_me__(base, *args, **kwargs)

class C(A):
def __init__(self):
pass
@classmethod
def __make_me__(cls, base, *args, **kwargs):
if base is C:
return C(*args, **kwargs)
elif base is A:
bound = inspect.signature(A).bind(*args, **kwargs)
return cls.__make_me__(C)
else:
return super(C, cls).__make_me__(base, *args, **kwargs)

class D(C, B):
def __init__(self, bar, baz, spam):
self.bar = bar
self.baz = baz
self.spam = spam
@classmethod
def __make_me__(cls, base, *args, **kwargs):
if base is D:
return D(*args, **kwargs)
elif base is B:
bound = inspect.signature(B).bind(*args, **kwargs)
return cls.__make_me__(D, bound.arguments['bar'],
   bound.arguments['baz'], 'hello')
else:
return super(D, cls).__make_me__(base, *args, **kwargs)

if __name__ == '__main__':
assert D('a', 'b', 'c').foo().__class__ is D
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Stephen J. Turnbull]

Victor Stinner writes:

 > *** Now the real question: is it useful to know the inode number
 > (st_ino) without the device number (st_dev)? ***
 > 
 > On POSIX, you can still get the st_dev from DirEntry.stat(), but it
 > always require a system call. So you loose the whole purpose of
 > DirEntry (no extra syscall).

True, but I suppose in many cases the user will know that all file
system objects handled are on the same device, or will be willing to
risk an occasional anomoly.

IMO: Document the limitation (if no extra syscall) or inefficiency
(with the syscall), and let the user choose.

The remaining issue is whether to provide a convenience function for
the device number, with appropriately loud warnings about how
inefficient it is, or to deter the user with the need to call .stat()
and extract the device number.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Cameron Simpson]

On 14Feb2015 11:35, Stephen J. Turnbull  wrote:

Victor Stinner writes:
> *** Now the real question: is it useful to know the inode number
> (st_ino) without the device number (st_dev)? ***
>
> On POSIX, you can still get the st_dev from DirEntry.stat(), but it
> always require a system call. So you loose the whole purpose of
> DirEntry (no extra syscall).

True, but I suppose in many cases the user will know that all file
system objects handled are on the same device, or will be willing to
risk an occasional anomoly.


In POSIX, all filsystem objects named by a directory are on the same device 
unless one is a mount point. (And in that case, d_ino from stat won't match 
d_ino from scandir; I expect.)



IMO: Document the limitation (if no extra syscall) or inefficiency
(with the syscall), and let the user choose.


+1 on .inode(): d_ino has been available in the directory data on POSIX since 
at least V7 UNIX (1970s), almost certainly earlier. Agree the limitation should 
be mentioned.



The remaining issue is whether to provide a convenience function for
the device number, with appropriately loud warnings about how
inefficient it is, or to deter the user with the need to call .stat()
and extract the device number.


-1 on that. People will use it! Given the doco above, it should be obvious 
under what circumstances one might choose to call stat, and making that stat 
overt means it is less likely to be called unwisely.


Since scandir is all about efficiency, providing a very costly convenience 
function seems to go against the grain.


Regarding usefulness: Victor, you've got the typical use case in another post 
(i.e. useful as in "advantageous"), and your own tests show that st_dev of the 
dir matches st_dev of a dir's entries in all normal/regular filesystems (i.e.  
useful as in "meaningful/consistent").  Special filesystems like /dev may be 
weird, but people relying on this should be aware of the constraint anyway.  
Since a directory at the low level is essentially a mapping of names to inodes 
within the directory's filesystem, this is to be expected.


Cheers,
Cameron Simpson 

Uh, this is only temporary...unless it works.   - Red Green
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Serhiy Storchaka]

On 14.02.15 03:12, Ethan Furman wrote:

The third choice is to use different specially designed constructor.

class A(int):

--> class A(int):
... def __add__(self, other):
... return self.__make_me__(int(self) + int(other))

... def __repr__(self):
... return 'A(%d)' % self


How would this help in the case of defaultdict?  __make_me__ is a class method, 
but it needs instance info to properly
create a new dict with the same default factory.


In case of defaultdict (when dict would have to have __add__ and like) 
either __make_me__ == dict (then defaultdict's methods will return 
dicts) or it will be instance method.


def __make_me__(self, other):
return defaultdict(self.default_factory, other)


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Serhiy Storchaka]

On 14.02.15 01:03, Neil Girdhar wrote:

Now the derived class knows who is asking for a copy.  In the case of
defaultdict, for example, he can implement __make_me__ as follows:

def __make_me__(self, cls, *args, **kwargs):
 if cls is dict: return default_dict(self.default_factory, *args,
**kwargs)
 return default_dict(*args, **kwargs)

essentially the caller is identifying himself so that the receiver knows
how to interpret the arguments.


No, my idea was that __make_me__ has the same signature in all 
subclasses. It takes exactly one argument and creates an instance of 
concrete class, so it never fails. If you want to create an instance of 
different class in the derived class, you should explicitly override 
__make_me__.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com