Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Nick Coghlan]

On 14 Feb 2015 03:43, "Nathaniel Smith"  wrote:
>
> On 13 Feb 2015 02:09, "Victor Stinner"  wrote:
> >
> > A alternative is to add a new _scandir.c module to host the new C
> > code, and share some code with posixmodule.c: remove "static" keyword
> > from required C functions (functions to convert Windows attributes to
> > a os.stat_result object).
>
> Hopefully not too annoying question from an outsider: has cpython's build
system added the necessary bits to do this on a safe, portable,
non-symbol-namespace polluting way? E.g. using -fvisibility=hidden on Linux?

We just add a "_Py" prefix on the things that we're making available to the
linker solely for own use and don't provide any backwards compatibility
guarantees for other people that decide to use them directly despite the
leading underscore and lack of documentation.

Cheers,
Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 486: Make the Python Launcher aware of virtual environments

[Nick Coghlan]

On 14 Feb 2015 07:31, "Paul Moore"  wrote:
>
> (By the way, on a procedural note, how do I update a PEP? Do I just
> send an updated version to p...@python.org, or is there a better way?)

If you're happy to handle the PEP editor responsibilities described in PEP
1 yourself, you can also ask for commit access to the PEPs repo to update
it directly.

Cheers,
Nick.

>
> Paul
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Nick Coghlan]

On 14 Feb 2015 13:17, "Cameron Simpson"  wrote:
>
> -1 on that. People will use it! Given the doco above, it should be
obvious under what circumstances one might choose to call stat, and making
that stat overt means it is less likely to be called unwisely.
>
> Since scandir is all about efficiency, providing a very costly
convenience function seems to go against the grain.
>
> Regarding usefulness: Victor, you've got the typical use case in another
post (i.e. useful as in "advantageous"), and your own tests show that
st_dev of the dir matches st_dev of a dir's entries in all normal/regular
filesystems (i.e.  useful as in "meaningful/consistent").  Special
filesystems like /dev may be weird, but people relying on this should be
aware of the constraint anyway.  Since a directory at the low level is
essentially a mapping of names to inodes within the directory's filesystem,
this is to be expected.

+1 from me for Cameron's perspective & rationale - it's useful for
detecting hardlinks, it will usually work, and the cases where it isn't
sufficient on its own are filesystem handling edge cases in more ways than
one.

Cheers,
Nick.

>
> Cheers,
> Cameron Simpson 
>
> Uh, this is only temporary...unless it works.   - Red Green
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Nick Coghlan]

On 14 Feb 2015 07:39, "Isaac Schwabacher"  wrote:
>
> On 15-02-13, Guido van Rossum  wrote:
> > Are you willing to wait 10 days for an answer? I'm out of round
tuits for a while.
>
> IIUC, the argument is that the Liskov Substitution Principle is a
statement about how objects of a subtype behave relative to objects of a
supertype, and it doesn't apply to constructors because they aren't
behaviors of existing objects. So other overriding methods *should* be able
to handle the same inputs that the respective overridden methods do, but
constructors don't need to. Even though __init__ is written as an instance
method, it seems like it's "morally" a part of the class method __new__
that's only split off for convenience.

A potentially helpful example is to consider a type system where Square is
a subclass of Rectangle.

To specify a Rectangle takes a height and a width, but a Square only needs
the length of one side, and letting the height and width be specified
independently would be outright wrong.

Many possible operations on a Square also *should* formally return a
Rectangle, as doing something like doubling the height gives you a result
that isn't a square any more. (That's also why shapes must be immutable for
this particular type hierarchy to make any sense)

(You can construct similar examples for Circle & Ellipse, and Python's
numeric hierarchy is a real-world example of some of the complexity that
can arise)

There's simply no sensible default behaviour other than the status quo in
the face of scenarios like that - whether or not there's a better answer
than "return an instance of the parent class" for any given inherited
method depends entirely on the invariants of the subclass and how they
differ from those of the parent class.

It's certainly possible to write methods that return a new instance of the
current type (that's common in alternative constructors, for example), but
it usually involves placing additional expectations on the developers of
subclasses. That's most commonly seen within the confines of a single
project, or when defining a development framework, rather than in libraries
that choose to expose some public types and also supports their use as base
types.

Cheers,
Nick.

>
> If this message is unclear, it's because I don't really understand this
myself and I'm trying to articulate my best understanding of what's been
said on this thread and those it links to.
>
> ijs
>
> > On Fri, Feb 13, 2015 at 10:22 AM, Alexander Belopolsky <
alexander.belopol...@gmail.com(javascript:main.compose()> wrote:
> >
> > >
> > > On Fri, Feb 13, 2015 at 1:19 PM, Alexander Belopolsky <
alexander.belopol...@gmail.com(javascript:main.compose()> wrote:
> > > >>
> > > >> FWIW you're wrong when you claim that "a constructor is no
different from any other method". Someone else should probably explain this
(it's an old argument that's been thoroughly settled).
> > > >
> > > >
> > > > Well, the best answer I've got in the past [1] was "ask on
python-dev since Guido called the operator overriding expectation." :-)
> > >
> > >
> > > And let me repost this bit of history [1]:
> > >
> > > Here is the annotated pre-r82065 code:
> > >
> > > 39876 gvanrossum def __add__(self, other):
> > > 39876 gvanrossum if isinstance(other, timedelta):
> > > 39928 gvanrossum return self.__class__(self.__days + other.__days,
> > > 39876 gvanrossum self.__seconds + other.__seconds,
> > > 39876 gvanrossum self.__microseconds + other.__microseconds)
> > > 40207 tim_one return NotImplemented
> > > 39876 gvanrossum
> > >
> > >
> > >
> > > [1] http://bugs.python.org/issue2267#msg125979
> > >
> > >
> >
> >
> >
> >
> > --
> > --Guido van Rossum (python.org/~guido(http://python.org/~guido))
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Nick Coghlan]

On 14 Feb 2015 08:57, "Alexander Belopolsky" 
wrote:
>
>
> On Fri, Feb 13, 2015 at 4:44 PM, Neil Girdhar 
wrote:
>>
>> Interesting:
http://stackoverflow.com/questions/5490824/should-constructors-comply-with-the-liskov-substitution-principle
>
>
> Let me humbly conjecture that the people who wrote the top answers have
background in less capable languages than Python.
>
> Not every language allows you to call self.__class__().  In the languages
that don't you can get away with incompatible constructor signatures.
>
> However, let me try to focus the discussion on a specific issue before we
go deep into OOP theory.
>
> With python's standard datetime.date we have:
>
> >>> from datetime import *
> >>> class Date(date):
> ... pass
> ...
> >>> Date.today()
> Date(2015, 2, 13)
> >>> Date.fromordinal(1)
> Date(1, 1, 1)
>
> Both .today() and .fromordinal(1) will break in a subclass that redefines
__new__ as follows:
>
> >>> class Date2(date):
> ... def __new__(cls, ymd):
> ... return date.__new__(cls, *ymd)
> ...
> >>> Date2.today()
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: __new__() takes 2 positional arguments but 4 were given
> >>> Date2.fromordinal(1)
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: __new__() takes 2 positional arguments but 4 were given
>
> Why is this acceptable, but we have to sacrifice the convenience of
having Date + timedelta
> return Date to make it work  with Date2:
>
> >>> Date2((1,1,1)) + timedelta(1)
> datetime.date(1, 1, 2)

Coupling alternative constructors to the default constructor signature is
pretty normal - it just means that if you override the signature of the
default constructor, you may need to update the alternative ones
accordingly.

Cheers,
Nick.



>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Victor Stinner]

Le samedi 14 février 2015, Stephen J. Turnbull  a
écrit :
>
> IMO: Document the limitation (if no extra syscall) or inefficiency
> (with the syscall), and let the user choose.


 Hum, by the way, I don't know if we should dd the method on Windows. As I
said, I don't want to cache The result of the os.lstat(). Basically, there
is no benfit for  other methods to call inode().

 A method may be a trap for Windows users.

I propose something else: a DirEntry.inode read-only property which would
be None on Windows. So you see dirrectly that the property is for POSIX,
and that calling os.stat() is required on Windows. os.stat() not
DirEntry.stat(), DirEntry.stat() doesn't fill st_ino, st_dev and st_nlink
are not filled on Windows.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Victor Stinner]

2015-02-14 11:57 GMT+01:00 Victor Stinner :
> I propose something else: a DirEntry.inode read-only property (...)

Full DirEntry API:

- name (str) attribute
- path (str) read-only property, created at the first call
- inode (int or None) attribute <=== my proposition
- is_dir(*, follow_symlinks=True)
- is_file(*, follow_symlinks=True)
- is_symlink(*, follow_symlinks=True)
- stat(*, follow_symlinks=True)

is_dir(), is_file(), is_symlink() and stat() are method because they
may all require a syscall (os.stat or os.lstat). They all cache their
result. In some cases, the result is already known when DirEntry is
created. In most cases, a single call to os.stat() is required to fill
the result of all methods.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Steven D'Aprano]

On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote:
> I personally don't think this is a big enough issue to warrant any changes,
> but I think Serhiy's solution would be the ideal best with one additional
> parameter: the caller's type.  Something like
> 
> def __make_me__(self, cls, *args, **kwargs)
> 
> and the idea is that any time you want to construct a type, instead of
> 
> self.__class__(assumed arguments…)
> 
> where you are not sure that the derived class' constructor knows the right
> argument types, you do
> 
> def SomeCls:
>  def some_method(self, ...):
>return self.__make_me__(SomeCls, assumed arguments…)
> 
> Now the derived class knows who is asking for a copy.

What if you wish to return an instance from a classmethod? You don't 
have a `self` available.

class SomeCls:
def __init__(self, x, y, z):
...
@classmethod
def from_spam(cls, spam):
x, y, z = process(spam)
return cls.__make_me__(self, cls, x, y, z)  # oops, no self


Even if you are calling from an instance method, and self is available, 
you cannot assume that the information needed for the subclass 
constructor is still available. Perhaps that information is used in the 
constructor and then discarded.

The problem we wish to solve is that when subclassing, methods of some
base class blindly return instances of itself, instead of self's type:


py> class MyInt(int):
... pass
...
py> n = MyInt(23)
py> assert isinstance(n, MyInt)
py> assert isinstance(n+1, MyInt)
Traceback (most recent call last):
  File "", line 1, in ?
AssertionError


The means that subclasses often have to override all the parent's 
methods, just to ensure the type is correct:

class MyInt(int):
def __add__(self, other):
o = super().__add__(other)
if o is not NotImplemented:
o = type(self)(o)
return o


Something like that, repeated for all the int methods, should work:

py> n = MyInt(23)
py> type(n+1)



This is tedious and error prone, but at least once it is done, 
subclasses of MyInt will Just Work:


py> class MyOtherInt(MyInt):
... pass
...
py> a = MyOtherInt(42)
py> type(a + 1000)



(At least, *in general* they will work. See below.)

So, why not have int's methods use type(self) instead of hard coding 
int? The answer is that *some* subclasses might override the 
constructor, which would cause the __add__ method to fail:

# this will fail if the constructor has a different signature
o = type(self)(o)


Okay, but changing the constructor signature is quite unusual. Mostly, 
people subclass to add new methods or attributes, or to override a 
specific method. The dict/defaultdict situation is relatively uncommon.

Instead of requiring *every* subclass to override all the methods, 
couldn't we require the base classes (like int) to assume that the 
signature is unchanged and call type(self), and leave it up to the 
subclass to override all the methods *only* if the signature has 
changed? (Which they probably would have to do anyway.)

As the MyInt example above shows, or datetime in the standard library, 
this actually works fine in practice:

py> from datetime import datetime
py> class MySpecialDateTime(datetime):
... pass
...
py> t = MySpecialDateTime.today()
py> type(t)



Why can't int, str, list, tuple etc. be more like datetime?



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Paul Moore]

I'm looking at putting together a patch for CPython to implement PEP
441. In doing so, there are a few issues that I'd like to raise with
the PEP. These are all to do with the supporting app "pyzaa" (IIRC,
Nick proposed renaming this to "pyzapp", which I like, but it's not a
big deal either way).

1. The PEP says that the application will warn if you try to add C
extensions to the archive. In practice, this will be quite hard to do
- extensions vary across platforms, and it's not only C extensions
that won't work from an archive (consider a DLL with a pure-python
ctypes wrapper). We could check the filenames for "known" extensions
(maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies
by platform) but I think that's going to be fragile. IMO, the costs
aren't worth the benefits, and I'd like to remove this proposal and
simply document that applications packed up with pyzaa need to be
tested to ensure they work from a zipfile.

2. The option to compile Python source seems to me to be of limited
use. It ties the archive to a particular Python version (more
precisely, only one version gets the benefits of precompiled
bytecode). It's also tricky to implement, particularly the option to
compile an existing archive, because there standard library py_compile
and compileall modules work on the filesystem, and for compiling files
in an archive it would be better to build the bytecode in memory (no
need to use tempfiles). I'd prefer to say that if users want to
include bytecode, they can precompile on disk (using compileall) for
as many versions as they want.

FWIW, Daniel's standalone pyzaa program doesn't currently include
either of the above features.

As well as removing things, I'd like to add a programmable API for the
utility. At the moment I only have a rough idea of how that would
work, but I'm thinking of a PyZApp class, which can be created from a
filename or an open stream (like a ZipFile object) and which has
methods to set the shebang line, setting the main entry point, and
adding files or directories. Does that need to go into the PEP or is
it OK to review the API as part of the patch? I don't mind either way,
but my normal coding process tends to be to fiddle with the API as I
code it up and use it in the main app, so I'd normally leave
documenting the final API till it's written...

If the above seems like a reasonable plan, I'll work up a patch with
the intention that it gets approved and implemented in time for 3.5.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Steve Dower]

"Instead of requiring *every* subclass to override all the methods,
couldn't we require the base classes (like int) to assume that the
signature is unchanged and call type(self), and leave it up to the
subclass to override all the methods *only* if the signature has
changed?"

I assumed everyone was just saying this point over and over, so I haven't been 
following the thread closely. This is precisely how inheritance works: 
subclasses are constrained by the base class. If you want to play, you *must* 
play by its rules. (Else, use composition.)

It's fine for base classes to assume a compatible constructor, and if builtins 
can do it without hurting performance for the non-subclassed case, I don't see 
why not.

Cheers,
Steve

Top-posted from my Windows Phone

From: Steven D'Aprano
Sent: ‎2/‎14/‎2015 4:24
To: python-dev@python.org
Subject: Re: [Python-Dev] subclassing builtin data structures

On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote:
> I personally don't think this is a big enough issue to warrant any changes,
> but I think Serhiy's solution would be the ideal best with one additional
> parameter: the caller's type.  Something like
>
> def __make_me__(self, cls, *args, **kwargs)
>
> and the idea is that any time you want to construct a type, instead of
>
> self.__class__(assumed arguments…)
>
> where you are not sure that the derived class' constructor knows the right
> argument types, you do
>
> def SomeCls:
>  def some_method(self, ...):
>return self.__make_me__(SomeCls, assumed arguments…)
>
> Now the derived class knows who is asking for a copy.

What if you wish to return an instance from a classmethod? You don't
have a `self` available.

class SomeCls:
def __init__(self, x, y, z):
...
@classmethod
def from_spam(cls, spam):
x, y, z = process(spam)
return cls.__make_me__(self, cls, x, y, z)  # oops, no self


Even if you are calling from an instance method, and self is available,
you cannot assume that the information needed for the subclass
constructor is still available. Perhaps that information is used in the
constructor and then discarded.

The problem we wish to solve is that when subclassing, methods of some
base class blindly return instances of itself, instead of self's type:


py> class MyInt(int):
... pass
...
py> n = MyInt(23)
py> assert isinstance(n, MyInt)
py> assert isinstance(n+1, MyInt)
Traceback (most recent call last):
  File "", line 1, in ?
AssertionError


The means that subclasses often have to override all the parent's
methods, just to ensure the type is correct:

class MyInt(int):
def __add__(self, other):
o = super().__add__(other)
if o is not NotImplemented:
o = type(self)(o)
return o


Something like that, repeated for all the int methods, should work:

py> n = MyInt(23)
py> type(n+1)



This is tedious and error prone, but at least once it is done,
subclasses of MyInt will Just Work:


py> class MyOtherInt(MyInt):
... pass
...
py> a = MyOtherInt(42)
py> type(a + 1000)



(At least, *in general* they will work. See below.)

So, why not have int's methods use type(self) instead of hard coding
int? The answer is that *some* subclasses might override the
constructor, which would cause the __add__ method to fail:

# this will fail if the constructor has a different signature
o = type(self)(o)


Okay, but changing the constructor signature is quite unusual. Mostly,
people subclass to add new methods or attributes, or to override a
specific method. The dict/defaultdict situation is relatively uncommon.

Instead of requiring *every* subclass to override all the methods,
couldn't we require the base classes (like int) to assume that the
signature is unchanged and call type(self), and leave it up to the
subclass to override all the methods *only* if the signature has
changed? (Which they probably would have to do anyway.)

As the MyInt example above shows, or datetime in the standard library,
this actually works fine in practice:

py> from datetime import datetime
py> class MySpecialDateTime(datetime):
... pass
...
py> t = MySpecialDateTime.today()
py> type(t)



Why can't int, str, list, tuple etc. be more like datetime?



--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Gregory P. Smith]

On Sat Feb 14 2015 at 3:17:51 AM Victor Stinner 
wrote:

> 2015-02-14 11:57 GMT+01:00 Victor Stinner :
> > I propose something else: a DirEntry.inode read-only property (...)
>
> Full DirEntry API:
>
> - name (str) attribute
> - path (str) read-only property, created at the first call
> - inode (int or None) attribute <=== my proposition
>

+1 we need to provide the inode (we shouldn't be throwing anything from the
underlying directory entry away when possible). But...

I think the "or None" semantics are a bad idea.  It'd be better for this to
raise AttributeError on Windows so that someone can't write the most
natural form of code assuming that inode is valid and have it appear to
work on Windows when in fact it'd do the wrong thing.


> - is_dir(*, follow_symlinks=True)
> - is_file(*, follow_symlinks=True)
> - is_symlink(*, follow_symlinks=True)
> - stat(*, follow_symlinks=True)
>
> is_dir(), is_file(), is_symlink() and stat() are method because they
> may all require a syscall (os.stat or os.lstat). They all cache their
> result. In some cases, the result is already known when DirEntry is
> created. In most cases, a single call to os.stat() is required to fill
> the result of all methods.
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano 
wrote:

> Why can't int, str, list, tuple etc. be more like datetime?


They are.  In all these types, class methods call subclass constructors but
instance methods don't.

>>> class Int(int):
... pass
...
>>> Int.from_bytes(bytes([1,2,3]), 'big')
66051
>>> type(_)


>>> Int(1) + 1
2
>>> type(_)


In the case of int, there is a good reason for this behavior - bool.  In
python, we want True + True == 2.  In numpy, where binary operations
preserve subclasses, you have

>>> import numpy
>>> numpy.bool_(1) + numpy.bool_(1)
True

I don't see a similar argument for the date class, however.  Given
date.{to|from}ordinal(), date subclasses are pretty much bound to have
timedelta addition satisfy (d + td).toordinal() == d.toordinal() +
td.days.  Any other definition would be fighting the baseclass design and
would be better implemented via containment.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Georg Brandl]

On 02/14/2015 07:26 PM, Alexander Belopolsky wrote:

> In the case of int, there is a good reason for this behavior - bool.  In 
> python,
> we want True + True == 2.  In numpy, where binary operations preserve
> subclasses, you have
> 
 import numpy
 numpy.bool_(1) + numpy.bool_(1)
> True

I don't think numpy.bool_ subclasses some class like numpy.int_.

Georg

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

I think the __make_me__ pattern discussed earlier is still the most generic
cooperative solution.  Here it is with a classmethod version too:

class C(D, E):
def some_method(self):
return __make_me__(self, C)

def __make_me__(self, arg_cls, *args, **kwargs):
if arg_cls is C:
pass
elif issubclass(D, arg_cls):
args, kwargs = modified_args_for_D(args, kwargs)
elif issubclass(E, arg_cls):
args, kwargs = modified_args_for_D(args, kwargs)
else:
raise ValueError

if self.__class__ == C:
return C(*args, **kwargs)
return self.__make_me__(C, *args, **kwargs)

@classmethod
def __make_me_cls__(cls, arg_cls, *args, **kwargs):
if arg_cls is C:
pass
elif issubclass(D, arg_cls):
args, kwargs = modified_args_for_D(args, kwargs)
elif issubclass(E, arg_cls):
args, kwargs = modified_args_for_D(args, kwargs)
else:
raise ValueError

if cls == C:
return C(*args, **kwargs)
return cls.__make_me_cls__(C, *args, **kwargs)


On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano 
wrote:

> On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote:
> > I personally don't think this is a big enough issue to warrant any
> changes,
> > but I think Serhiy's solution would be the ideal best with one additional
> > parameter: the caller's type.  Something like
> >
> > def __make_me__(self, cls, *args, **kwargs)
> >
> > and the idea is that any time you want to construct a type, instead of
> >
> > self.__class__(assumed arguments…)
> >
> > where you are not sure that the derived class' constructor knows the
> right
> > argument types, you do
> >
> > def SomeCls:
> >  def some_method(self, ...):
> >return self.__make_me__(SomeCls, assumed arguments…)
> >
> > Now the derived class knows who is asking for a copy.
>
> What if you wish to return an instance from a classmethod? You don't
> have a `self` available.
>
> class SomeCls:
> def __init__(self, x, y, z):
> ...
> @classmethod
> def from_spam(cls, spam):
> x, y, z = process(spam)
> return cls.__make_me__(self, cls, x, y, z)  # oops, no self
>
>
> Even if you are calling from an instance method, and self is available,
> you cannot assume that the information needed for the subclass
> constructor is still available. Perhaps that information is used in the
> constructor and then discarded.
>
> The problem we wish to solve is that when subclassing, methods of some
> base class blindly return instances of itself, instead of self's type:
>
>
> py> class MyInt(int):
> ... pass
> ...
> py> n = MyInt(23)
> py> assert isinstance(n, MyInt)
> py> assert isinstance(n+1, MyInt)
> Traceback (most recent call last):
>   File "", line 1, in ?
> AssertionError
>
>
> The means that subclasses often have to override all the parent's
> methods, just to ensure the type is correct:
>
> class MyInt(int):
> def __add__(self, other):
> o = super().__add__(other)
> if o is not NotImplemented:
> o = type(self)(o)
> return o
>
>
> Something like that, repeated for all the int methods, should work:
>
> py> n = MyInt(23)
> py> type(n+1)
> 
>
>
> This is tedious and error prone, but at least once it is done,
> subclasses of MyInt will Just Work:
>
>
> py> class MyOtherInt(MyInt):
> ... pass
> ...
> py> a = MyOtherInt(42)
> py> type(a + 1000)
> 
>
>
> (At least, *in general* they will work. See below.)
>
> So, why not have int's methods use type(self) instead of hard coding
> int? The answer is that *some* subclasses might override the
> constructor, which would cause the __add__ method to fail:
>
> # this will fail if the constructor has a different signature
> o = type(self)(o)
>
>
> Okay, but changing the constructor signature is quite unusual. Mostly,
> people subclass to add new methods or attributes, or to override a
> specific method. The dict/defaultdict situation is relatively uncommon.
>
> Instead of requiring *every* subclass to override all the methods,
> couldn't we require the base classes (like int) to assume that the
> signature is unchanged and call type(self), and leave it up to the
> subclass to override all the methods *only* if the signature has
> changed? (Which they probably would have to do anyway.)
>
> As the MyInt example above shows, or datetime in the standard library,
> this actually works fine in practice:
>
> py> from datetime import datetime
> py> class MySpecialDateTime(datetime):
> ... pass
> ...
> py> t = MySpecialDateTime.today()
> py> type(t)
> 
>
>
> Why can't int, str, list, tuple etc. be more like datetime?
>
>
>
> --
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-d

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Ben Hoyt]

> +1 we need to provide the inode (we shouldn't be throwing anything from the
> underlying directory entry away when possible). But...
>
> I think the "or None" semantics are a bad idea.  It'd be better for this to
> raise AttributeError on Windows so that someone can't write the most natural
> form of code assuming that inode is valid and have it appear to work on
> Windows when in fact it'd do the wrong thing.

+1 for inode support. I agree with the above -- it should either raise
AttributeError on Windows if it's not going to be set ... or it should
be more like Victor's original proposal where .inode() is a method
that calls stat on Windows. I don't have strong feelings.

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Neil Girdhar]

Oops, I meant to call super if necessary:

@classmethod
def __make_me_cls__(cls, arg_cls, *args, **kwargs):
if arg_cls is C:
pass
elif arg_cls is D:
args, kwargs = modified_args_for_D(args, kwargs)
elif arg_cls is E:
args, kwargs = modified_args_for_D(args, kwargs)
else:
return super().__make_me_cls__(arg_cls, args, kwargs)

if cls is C:
return C(*args, **kwargs)
return cls.__make_me_cls__(C, *args, **kwargs)


On Sat, Feb 14, 2015 at 3:15 PM, Neil Girdhar  wrote:

> I think the __make_me__ pattern discussed earlier is still the most
> generic cooperative solution.  Here it is with a classmethod version too:
>
> class C(D, E):
> def some_method(self):
> return __make_me__(self, C)
>
> def __make_me__(self, arg_cls, *args, **kwargs):
> if arg_cls is C:
> pass
> elif issubclass(D, arg_cls):
> args, kwargs = modified_args_for_D(args, kwargs)
> elif issubclass(E, arg_cls):
> args, kwargs = modified_args_for_D(args, kwargs)
> else:
> raise ValueError
>
> if self.__class__ == C:
> return C(*args, **kwargs)
> return self.__make_me__(C, *args, **kwargs)
>
> @classmethod
> def __make_me_cls__(cls, arg_cls, *args, **kwargs):
> if arg_cls is C:
> pass
> elif issubclass(D, arg_cls):
> args, kwargs = modified_args_for_D(args, kwargs)
> elif issubclass(E, arg_cls):
> args, kwargs = modified_args_for_D(args, kwargs)
> else:
> raise ValueError
>
> if cls == C:
> return C(*args, **kwargs)
> return cls.__make_me_cls__(C, *args, **kwargs)
>
>
> On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano 
> wrote:
>
>> On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote:
>> > I personally don't think this is a big enough issue to warrant any
>> changes,
>> > but I think Serhiy's solution would be the ideal best with one
>> additional
>> > parameter: the caller's type.  Something like
>> >
>> > def __make_me__(self, cls, *args, **kwargs)
>> >
>> > and the idea is that any time you want to construct a type, instead of
>> >
>> > self.__class__(assumed arguments…)
>> >
>> > where you are not sure that the derived class' constructor knows the
>> right
>> > argument types, you do
>> >
>> > def SomeCls:
>> >  def some_method(self, ...):
>> >return self.__make_me__(SomeCls, assumed arguments…)
>> >
>> > Now the derived class knows who is asking for a copy.
>>
>> What if you wish to return an instance from a classmethod? You don't
>> have a `self` available.
>>
>> class SomeCls:
>> def __init__(self, x, y, z):
>> ...
>> @classmethod
>> def from_spam(cls, spam):
>> x, y, z = process(spam)
>> return cls.__make_me__(self, cls, x, y, z)  # oops, no self
>>
>>
>> Even if you are calling from an instance method, and self is available,
>> you cannot assume that the information needed for the subclass
>> constructor is still available. Perhaps that information is used in the
>> constructor and then discarded.
>>
>> The problem we wish to solve is that when subclassing, methods of some
>> base class blindly return instances of itself, instead of self's type:
>>
>>
>> py> class MyInt(int):
>> ... pass
>> ...
>> py> n = MyInt(23)
>> py> assert isinstance(n, MyInt)
>> py> assert isinstance(n+1, MyInt)
>> Traceback (most recent call last):
>>   File "", line 1, in ?
>> AssertionError
>>
>>
>> The means that subclasses often have to override all the parent's
>> methods, just to ensure the type is correct:
>>
>> class MyInt(int):
>> def __add__(self, other):
>> o = super().__add__(other)
>> if o is not NotImplemented:
>> o = type(self)(o)
>> return o
>>
>>
>> Something like that, repeated for all the int methods, should work:
>>
>> py> n = MyInt(23)
>> py> type(n+1)
>> 
>>
>>
>> This is tedious and error prone, but at least once it is done,
>> subclasses of MyInt will Just Work:
>>
>>
>> py> class MyOtherInt(MyInt):
>> ... pass
>> ...
>> py> a = MyOtherInt(42)
>> py> type(a + 1000)
>> 
>>
>>
>> (At least, *in general* they will work. See below.)
>>
>> So, why not have int's methods use type(self) instead of hard coding
>> int? The answer is that *some* subclasses might override the
>> constructor, which would cause the __add__ method to fail:
>>
>> # this will fail if the constructor has a different signature
>> o = type(self)(o)
>>
>>
>> Okay, but changing the constructor signature is quite unusual. Mostly,
>> people subclass to add new methods or attributes, or to override a
>> specific method. The dict/defaultdict situation is relatively uncommon.
>>
>> Instead of requiring *every* subclass to override all the methods,
>> couldn't we require the base classes (like int) to assume that the
>> si

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Antoine Pitrou]

On Sat, 14 Feb 2015 15:32:07 -0500
Ben Hoyt  wrote:
> > +1 we need to provide the inode (we shouldn't be throwing anything from the
> > underlying directory entry away when possible). But...
> >
> > I think the "or None" semantics are a bad idea.  It'd be better for this to
> > raise AttributeError on Windows so that someone can't write the most natural
> > form of code assuming that inode is valid and have it appear to work on
> > Windows when in fact it'd do the wrong thing.
> 
> +1 for inode support. I agree with the above -- it should either raise
> AttributeError on Windows if it's not going to be set ... or it should
> be more like Victor's original proposal where .inode() is a method
> that calls stat on Windows. I don't have strong feelings.

The whole point of scandir is to expose low-level system calls in a
cross-platform way. If you start raising some exceptions on some
platforms then that quality disappears.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Alexander Belopolsky]

On Sat, Feb 14, 2015 at 2:36 PM, Georg Brandl  wrote:

> > In the case of int, there is a good reason for this behavior - bool.  In
> python,
> > we want True + True == 2.  In numpy, where binary operations preserve
> > subclasses, you have
> >
>  import numpy
>  numpy.bool_(1) + numpy.bool_(1)
> > True
>
> I don't think numpy.bool_ subclasses some class like numpy.int_.


And  numpy.bool_ subclasses don't preserve type in addition:

>>> import numpy
>>> class Bool(numpy.bool_):
... pass
...
>>> numpy.bool_.mro()
[, , ]
>>> Bool(1) + Bool(1)
True
>>> type(_)


So there goes my theory. :-)

I think all these examples just highlight the need for a clear guidance
when self.__class__() can be called in base classes to construct instances
of derived classes.

Apparently numpy has it both ways.  One way for scalars (see above) and the
other for arrays:

>>> class Array(numpy.ndarray):
... pass
...
>>> a = Array(1)
>>> a[0] = 1
>>> a+a
Array([ 2.])
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Gregory P. Smith]

That suggests the .inode() method approach makes more sense then.

On Sat, Feb 14, 2015, 12:44 PM Antoine Pitrou  wrote:

> On Sat, 14 Feb 2015 15:32:07 -0500
> Ben Hoyt  wrote:
> > > +1 we need to provide the inode (we shouldn't be throwing anything
> from the
> > > underlying directory entry away when possible). But...
> > >
> > > I think the "or None" semantics are a bad idea.  It'd be better for
> this to
> > > raise AttributeError on Windows so that someone can't write the most
> natural
> > > form of code assuming that inode is valid and have it appear to work on
> > > Windows when in fact it'd do the wrong thing.
> >
> > +1 for inode support. I agree with the above -- it should either raise
> > AttributeError on Windows if it's not going to be set ... or it should
> > be more like Victor's original proposal where .inode() is a method
> > that calls stat on Windows. I don't have strong feelings.
>
> The whole point of scandir is to expose low-level system calls in a
> cross-platform way. If you start raising some exceptions on some
> platforms then that quality disappears.
>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> greg%40krypto.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Ben Hoyt]

>> +1 for inode support. I agree with the above -- it should either raise
>> AttributeError on Windows if it's not going to be set ... or it should
>> be more like Victor's original proposal where .inode() is a method
>> that calls stat on Windows. I don't have strong feelings.
>
> The whole point of scandir is to expose low-level system calls in a
> cross-platform way. If you start raising some exceptions on some
> platforms then that quality disappears.

I agree with that!

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Marko Rauhamaa]

Antoine Pitrou :

> The whole point of scandir is to expose low-level system calls in a
> cross-platform way.

Cross-platform is great and preferable, but low-level system facilities
should be made available even when they are unique to a particular OS.


Marko
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Ben Hoyt]

>> The whole point of scandir is to expose low-level system calls in a
>> cross-platform way.
>
> Cross-platform is great and preferable, but low-level system facilities
> should be made available even when they are unique to a particular OS.

Yes, but this can be made cross-platform fairly easily, just like the
other method calls. Just like DirEntry.stat() has different
cross-platform operation (no stat call on Windows, a stat call on
POSIX), DirEntry.inode() would have a different operation (stat call
on Windows, no stat call on POSIX).

-Ben
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Add a new DirEntry.inode() method?

[Victor Stinner]

Le 14 févr. 2015 18:47, "Gregory P. Smith"  a écrit :
> I think the "or None" semantics are a bad idea.

Oh, in fact it shouldn't be None but 0 onWindows to be consistent with
DirEntry.stat().st_ino which is also equal to 0.

The value 0 is not a valid inode number.

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Can I replace distutils.msvccompiler?

[Steve Dower]

I'm about to start doing a rework on distutils.msvc[9]compiler.MSVCCompiler for 
Python 3.5 to be able to handle forward-compatibility better. Right now I've 
tweaked msvc9compiler enough for VS 2015, but it won't handle later VS versions 
automatically.


Is there any reason to keep the old msvccompiler and msvc9compiler modules 
around? If possible I'd like to drop them completely and just have one 
implementation (probably distutils.msvccompiler.MSVCCompiler again).


Obviously it will be behaviourally compatible with 3.4 and compatible with the 
CCompiler interface, but it may not be function-for-function compatible with 
distutils.msvc9compiler (e.g. get_build_version is not useful). Do we need a 
deprecation period? msvccompiler is mentioned in the docs, but not documented, 
and msvc9compiler is not even mentioned.


Thanks,

Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Steven D'Aprano]

On Sat, Feb 14, 2015 at 01:26:36PM -0500, Alexander Belopolsky wrote:
> On Sat, Feb 14, 2015 at 7:23 AM, Steven D'Aprano 
> wrote:
> 
> > Why can't int, str, list, tuple etc. be more like datetime?
> 
> 
> They are.  In all these types, class methods call subclass constructors but
> instance methods don't.

But in datetime, instance methods *do*.

Sorry that my example with .today() was misleading.

py> from datetime import datetime
py> class MyDatetime(datetime):
... pass
...
py> MyDatetime.today()
MyDatetime(2015, 2, 15, 12, 45, 38, 429269)
py> MyDatetime.today().replace(day=20)
MyDatetime(2015, 2, 20, 12, 45, 53, 405889)


> In the case of int, there is a good reason for this behavior - bool.  In
> python, we want True + True == 2.

Sure. But bool is only one subclass. I expect that it should be bool's 
responsibility to override __add__ etc. to return an instance of the 
parent class (int) rather have nearly all subclasses have to override 
__add__ etc. to return instances of themselves.



-- 
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Nick Coghlan]

On 15 February 2015 at 00:44, Paul Moore  wrote:
> I'm looking at putting together a patch for CPython to implement PEP
> 441. In doing so, there are a few issues that I'd like to raise with
> the PEP. These are all to do with the supporting app "pyzaa" (IIRC,
> Nick proposed renaming this to "pyzapp", which I like, but it's not a
> big deal either way).
>
> 1. The PEP says that the application will warn if you try to add C
> extensions to the archive. In practice, this will be quite hard to do
> - extensions vary across platforms, and it's not only C extensions
> that won't work from an archive (consider a DLL with a pure-python
> ctypes wrapper). We could check the filenames for "known" extensions
> (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies
> by platform) but I think that's going to be fragile. IMO, the costs
> aren't worth the benefits, and I'd like to remove this proposal and
> simply document that applications packed up with pyzaa need to be
> tested to ensure they work from a zipfile.
>
> 2. The option to compile Python source seems to me to be of limited
> use. It ties the archive to a particular Python version (more
> precisely, only one version gets the benefits of precompiled
> bytecode). It's also tricky to implement, particularly the option to
> compile an existing archive, because there standard library py_compile
> and compileall modules work on the filesystem, and for compiling files
> in an archive it would be better to build the bytecode in memory (no
> need to use tempfiles). I'd prefer to say that if users want to
> include bytecode, they can precompile on disk (using compileall) for
> as many versions as they want.
>
> FWIW, Daniel's standalone pyzaa program doesn't currently include
> either of the above features.

"Do the simplest thing that could possibly work" applies, so I'd be OK
with dropping both of those features.

> As well as removing things, I'd like to add a programmable API for the
> utility. At the moment I only have a rough idea of how that would
> work, but I'm thinking of a PyZApp class, which can be created from a
> filename or an open stream (like a ZipFile object) and which has
> methods to set the shebang line, setting the main entry point, and
> adding files or directories. Does that need to go into the PEP or is
> it OK to review the API as part of the patch? I don't mind either way,
> but my normal coding process tends to be to fiddle with the API as I
> code it up and use it in the main app, so I'd normally leave
> documenting the final API till it's written...

I'd suggest including the API in the PEP, which can co-evolve with the
reference implementation.

The following suggestions are ones I came up with last time I looked
closely at the draft PEP:

* use "zipapp" for the support module name, distinct from the command
line tool name (think venv vs pyvenv)

* expose a zipapp.pack programmatic interface that takes all the input
pieces and builds a suitable archive

I'd definitely prefer a simple procedural API like that over offering
a public stateful object-oriented API, as the latter significantly
increases testing complexity without offering a sufficiently
compelling gain in expressiveness to justify the additional upfront
test development cost and the ongoing maintenance and support burden.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com