Re: [Tutor] Inheritance, superclass, ‘super’ (was: __repr__ and __str__)

2015-07-02 Thread Steven D'Aprano 
I mostly agree with what Ben says, comments below.

On Wed, Jul 01, 2015 at 12:48:47PM +1000, Ben Finney wrote:
> Alan Gauld  writes:
> 
> > Whilst I agree with the general use of super I'm not sure what
> > the MRO resolution has to do with this case?
> 
> When accessing the superclass, the MRO is always relevant.

For those who aren't sure, MRO is "Method Resolution Order".

To be precise, all (new style) classes have a MRO:

py> object.__mro__
(,)

In the multiple-inheritance world, all inheritance is supposed to follow 
the MRO, at least by default. It should be quite unusual to manually 
bypass the MRO or invent your own, that should stand out as a code 
smell.

One complication in Python 2 is that "classic classes" (old style 
classes) don't have an explicit MRO. But they do have an implicit one. 
Unfortunately classic classes' MRO is buggy when used with multiple 
inheritance with a diamond-shaped class hierachy. I won't explain that 
unless somebody asks, the important thing to take from this is that you 
should try to avoid inheriting from "classic classes" in Python 2.

 
> > It's explicitly single inheritance and they are explicitly calling the
> > superclass.
> 
> They don't know that the superclass will be what is written there;
> that's an implication of what I said with “in Python, any class an
> participate in multiple inheritance”.
> 
> The author of a class (e.g. the OP's example ‘MyList’) cannot know at
> time of writing whether the superclass is ‘list’. This is because some
> other class may be using ‘MyList’ in a different inheritance
> arrangement.

In principle, Ben is correct here, but in practice, multiple-inheritance 
in Python is explicitly described as *cooperative* multiple-inheritance. 
You cannot expect to take any two (or more) arbitrary classes and 
inherit from them both. Sometimes you cannot inherit from them both at 
all:

>>> class X(list, int):
... pass
...
Traceback (most recent call last):
  File "", line 1, in 
TypeError: multiple bases have instance lay-out conflict


Sometimes you cannot inherit from them both because there is no 
consistent MRO, and sometimes you cannot inherit from them both because 
they simply don't work well together. In this last case, Python cannot 
warn you about it, but your class will just not work right. ("Your save 
method over-writes the data my save method just wrote!" sort of thing.)

The simplest way to break M-I is to directly call your parent class by 
name, instead of using super(). That effectively guarantees that your 
class cannot be used in M-I, except perhaps by accident.

In a nutshell, you can physically inherit from "the" parent class by 
explicitly giving the name of the class, but by doing so you are 
effectively saying "Uh uh, oh no you don't, no way!" to any one wishing 
to use your class in a multiple inheritance situation. As the writer of 
the class, that is your right to do so (nobody can force you to write 
cooperative M-I classes), but in my opinion it's a bit rude or ignorant, 
like writing functions that never return their result but always print 
it instead. ("Nobody's going to re-use *my* functions, uh uh, no way!")

(In the M-I world, there is no such thing as "the" parent class. There 
is, however, the parent *classes*.)
 
[...]
> Multiple inheitance is a fact in Python, and good practice is to not
> arbitrarily write classes that break it. Hence my advice to avoid
> hard-coding the superclass, and only use ‘super’ to discover the
> superclass.

I agree that it is good practice to use super, even if you personally 
expect to only use single-inheritance. You never know when somebody 
else, or even your future self, will decide to use M-I. Don't 
unnecessarily cut off that possibility, especially in Python 3 where 
using super() is actually easier than explicitly calling the parent 
class by hand:

list.append(self, arg)

super().append(arg)


On the other hand, M-I in Python is cooperative, and it's not very often 
that you can expect to pick two arbitrary classes and inherit from them 
both.

And on the third hand, inheritance in general is often over-rated, and 
M-I in particular is hard to get right. There's a strong argument to be 
made that one should only use full cooperative M-I when easier, less 
powerful versions like mixins and traits aren't enough. Or use 
composition and/or delegation instead.


> So please use `super`, even in single inheritance. Otherwise you are
> restricting the usefulness of your class: it can never be used with
> multiple inheritance.

I cannot disagree with Ben here. Even if you only intend to use single 
inheritance, don't unnecessarily rule out the alternatives. Using 
super is still usually the right thing to do.


-- 
Steve
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Inheritance, superclass, ‘super’

2015-07-02 Thread Steven D'Aprano 
On Wed, Jul 01, 2015 at 10:40:20PM +0100, Alan Gauld wrote:

> >Multiple inheitance is a fact in Python, and good practice is to not
> >arbitrarily write classes that break it.
> 
> That depends on what you mean by break it., MI should allow the 
> inheriting class to specify which, if any, of its direct superclasses 
> methods are invoked. That's critical to the use of MI in any language
> The issues around that are addressed differently by different languages, 
> for example Eiffel allows you to rename inherited methods
> or attributes to avoid them being used involuntarily.

That sounds like Eiffel multiple-inheritance is closer to traits than 
full M-I:

http://www.artima.com/weblogs/viewpost.jsp?thread=246488


> Most just allow 
> the programmer to explicitly access the superclasses where default 
> behaviour is not required. That's what I thought Python was doing with 
> super. Making default easy to get but still allowing direct overriding 
> when required.

In Python, super() is designed to ensure that when you call the 
parent(s)'s method, *all* the parents that define that method get run, 
not just the immediate parent. (Assuming that they all cooperate by 
using super.)

To quote Michele Simionato from the above link:

[quote]
The point to notice is that the complication of the MRO is by design: 
languages with a non-trivial MRO where [sic] designed this way to make 
possible method cooperation via super calls. That means that if both 
parents P1 and P2 define a method m, a child class C can override it and 
still have access to the m methods of the parents via super: C.m will 
call first P1.m and then P2.m, if P1.m features a super call itself.
[end quote]

If your inheritance needs don't match that behaviour, then Python's M-I 
model is not a good fit for you, and you might be better off with 
Michele's strait module, some other library, or just managing the 
inheritance calls yourself. But if you do that, you're implicitly ruling 
your class out from participating in "regular" M-I hierarchies.


> >>class A(B,C,D):
> >>def m(self):
> >>C.m(self)   # call C's m but not B and D if they have them.
> >>
> >>Or am I missing something in the way Python evaluates
> >>the C.m() call above?
> 
> I still want to know how you would code the above scenario
> using super? If B,C and D all have an m() method but I only
> want to call the version in C how do I control that in
> Python other than by a direct call to C.m()?

That can't be done using super(). Super isn't designed for that 
use-case, it's designed for the case where you want to call B.m, C.m and 
D.m in that order.

Hence my comments in the previous post about not *unnecessarily* ruling 
out the use of multiple inheritance. If it's necessary, you can do it.


> >Any class one writes can
> >become part of some other class's multiple-inheritance chain, and it is
> >bad practice to assume one knows at time of writing what the superclass
> >will be at run time.
> 
> This is the bit I don't understand. The superclass of a given class 
> should never change. The sequence of default processing may change but 
> any other behavior would be a serious breach of the class definition.

Even in the single inheritance world, there is no such concept as "the 
superclass". There is *one or more* superclasses (note plural). (Zero 
only in the case of the root of the object hierarchy.)

In the most general case, your child class doesn't know whether it is 
inheriting the method from the parent, grandparent, great-grandparent, 
... great-to-the-Nth-grandparent. Now, in S-I world, it doesn't matter 
whether you write:

super().method(arg)

or 

   MyParent.method(self, arg)

it will just work. But in a M-I world, only the first case will operate 
correctly when using multiple inheritance of the particular model for 
M-I supported by Python. If your needs are different, you're on your 
own, and good luck to you.


> A rather contrived example is where I want to inherit a Pen and a 
> Curtain to produce a GraphicalCurtain class. Both Pen and Curtain define 
> a draw() method but they do very different things. So I define two 
> methods: draw() which invokes Pen.draw() and close() which invokes 
> Curtain.draw(). If I use super in either of those methods I'll get the 
> wrong behavior because both Pen.draw and Curtain.draw will be invoked,

I think Ben's super()-friendly answer to that would be to use an adaptor 
class:


# Untested
class MyCurtain(object):
def __init__(self, *args):
self.__curtain = Curtain(*args)
def __getattr__(self, name):
if name == 'draw':
raise AttributeError
return getattr(self.__curtain, name)
def close(self):
return self.__curtain.draw()


class GraphicalCurtain(Pen, MyCurtain):
def draw(self):
print("Drawing drawing drawing...")
super().draw()



> Unless I'm missing something clever in super? I never quite
> understood the parameters in super() so

Re: [Tutor] memory error

2015-07-02 Thread Danny Yoo 
On Thu, Jul 2, 2015 at 9:57 AM, Joshua Valdez  wrote:
>
> Hi so I figured out my problem, with this code and its working great but its 
> still taking a very long time to process...I was wondering if there was a way 
> to do this with just regular expressions instead of parsing the text with 
> lxml...


Be careful: there are assumptions here that may not be true.

To be clear: regular expressions are not magic.  Just because
something uses regular expressions does not make it fast.  Nor are
regular expressions appropriate for parsing tree-structured content.

For a humorous discussion of this, see:
http://blog.codinghorror.com/parsing-html-the-cthulhu-way/



> the idea would be to identify a  tag and then move to the next line of 
> a file to see if there is a match between the title text and the pages in my 
> pages file.

This makes another assumption about the input that isn't necessarily
true.  Just because you see tags and content on separate lines now
doesn't mean that this won't change in the future.  XML tree structure
does not depend on newlines.  Don't try parsing XML files
line-by-line.


> So again, my pages are just an array like: [Anarchism, Abrahamic Mythology, 
> ...] I'm a little confused as to how to even start this my initial idea was 
> something like this but I'm not sure how to execute it:
> wiki --> XML file
> page_titles -> array of strings corresponding to titles
>
> tag = r '()'
> wiki = wiki.readlines()
>
> for line in wiki:
>   page = re.search(tag,line)
>   if page:
> ..(I'm not sure what to do here)
>
> is it possible to look ahead in a loop to discover other lines and then 
> backtrack?
> I think this may be the solution but again I'm not sure how I would execute 
> such a command structure...


You should probably abandon this line of thinking.From your
initial problem description, the approach you have now should be
computationally *linear* in the size of the input.  So maybe your
program is fine.  The fact that your program was exhausting your
computer's entire memory in your initial attempt suggests that your
input file is large.  But how large?  It is much more likely that your
program is slow simply because your input is honking huge.


To support this hypothesis, we need more knowledge about the input
size.  How large is your input file?  How large is your collection of
page_titles?
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] memory error

2015-07-02 Thread Danny Yoo 
>
> So I got my code working now and it looks like this
>
> TAG = '{http://www.mediawiki.org/xml/export-0.10/}page'
> doc = etree.iterparse(wiki)
>
> for _, node in doc:
> if node.tag == TAG:
> title = 
> node.find("{http://www.mediawiki.org/xml/export-0.10/}title";).text
> if title in page_titles:
> print (etree.tostring(node))
> node.clear()
> Its mostly giving me what I want.  However it is adding extra formatting (I 
> believe name_spaces and attributes).  I was wondering if there was a way to 
> strip these out when I'm printing the node tostring?


I suspect that you'll want to do an explicit walk over the node.
Rather than use etree.tostring(), which indiscriminately walks the
entire tree, you'll probably want to write a function to walk over
selected portions of the tree structure.   You can see:

https://docs.python.org/2/library/xml.etree.elementtree.html#tutorial

for an introduction to navigating portions of the tree, given a node.


As a more general response: you have significantly more information
about the problem than we do.  At the moment, we don't have enough
context to effectively help; we need more information.  Do you have a
sample *input* file that folks here can use to execute on your
program?  Providing sample input is important if you want
reproducibility.  Reproducibility is important because then we'll be
on the same footing in terms of knowing what the problem's inputs are.
See: http://sscce.org/

As for the form of the desired output: can you say more precisely what
parts of the document you want?  Rather than just say: "this doesn't
look the way I want it to", it may be more helpful to say: "here's
*exactly* what I'd like it to look like..." and show us the desired
text output.

That is: by expressing what you want as a collection of concrete
input/output examples, you gain the added benefit that once you have
revised your program, you can re-run it and see if what it's producing
is what you anticipate.  That is, you can use these concrete examples
as a "regression test suite".  This technique is something that
software engineers use regularly in their day-to-day work, to make
believe that their incomplete programs are working, to write out
explicitly what those programs should do, and then to work on their
programs until they do what they want them to.


Good luck to you!
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] memory error

2015-07-02 Thread Joshua Valdez 
Hi so I figured out my problem, with this code and its working great but
its still taking a very long time to process...I was wondering if there was
a way to do this with just regular expressions instead of parsing the text
with lxml...

the idea would be to identify a  tag and then move to the next line
of a file to see if there is a match between the title text and the pages
in my pages file.  I would then want to write the entire page tag
fdsalkfdjadslf text to ouput...

So again, my pages are just an array like: [Anarchism, Abrahamic Mythology,
...] I'm a little confused as to how to even start this my initial idea was
something like this but I'm not sure how to execute it:
wiki --> XML file
page_titles -> array of strings corresponding to titles

tag = r '()'
wiki = wiki.readlines()

for line in wiki:
  page = re.search(tag,line)
  if page:
..(I'm not sure what to do here)

is it possible to look ahead in a loop to discover other lines and then
backtrack?
I think this may be the solution but again I'm not sure how I would execute
such a command structure...








*Joshua Valdez*
*Computational Linguist : Cognitive Scientist
 *

(440)-231-0479
jd...@case.edu  | j...@uw.edu | jo...@armsandanchors.com


On Wed, Jul 1, 2015 at 10:13 AM, Joshua Valdez  wrote:

> Hi Danny,
>
> So I got my code workin now and it looks like this
>
> TAG = '{http://www.mediawiki.org/xml/export-0.10/}page'
> doc = etree.iterparse(wiki)
>
> for _, node in doc:
> if node.tag == TAG:
> title = node.find("{
> http://www.mediawiki.org/xml/export-0.10/}title";).text
> if title in page_titles:
> print (etree.tostring(node))
> node.clear()
> Its mostly giving me what I want.  However it is adding extra formatting
> (I believe name_spaces and attributes).  I was wondering if there was a way
> to strip these out when I'm printing the node tostring?
>
> Here is an example of the last few lines of my output:
>
> [[Category:Asteroids| ]]
> [[Category:Spaceflight]]
>   h4rxxfq37qg30eqegyf4vfvkqn3r142
> 
>   
>
>
>
>
>
>
> *Joshua Valdez*
> *Computational Linguist : Cognitive Scientist
>*
>
> (440)-231-0479
> jd...@case.edu  | j...@uw.edu | jo...@armsandanchors.com
> 
>
> On Wed, Jul 1, 2015 at 1:17 AM, Danny Yoo  wrote:
>
>> Hi Joshua,
>>
>>
>>
>> The issue you're encountering sounds like XML namespace issues.
>>
>>
>> >> So I tried that code snippet you pointed me too and I'm not getting
>> any output.
>>
>>
>> This is probably because the tag names of the XML are being prefixed
>> with namespaces.  This would make the original test for node.tag to be
>> too stingy: it wouldn't exactly match the string we want, because
>> there's a namespace prefix in front that's making the string mismatch.
>>
>>
>> Try relaxing the condition from:
>>
>> if node.tag == "page": ...
>>
>> to something like:
>>
>> if node.tag.endswith("page"): ...
>>
>>
>> This isn't quite technically correct, but we want to confirm whether
>> namespaces are the issue that's preventing you from seeing those
>> pages.
>>
>>
>> If namespaces are the issue, then read:
>>
>> http://effbot.org/zone/element-namespaces.htm
>>
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] Are the methods in a class copied or just linked to?

2015-07-02 Thread Jim Mooney Py3.4.3winXP 
When an instance uses a class method, does it actually use the method that
is in the class object's memory space, or is the method copied to the
instance?

-- 
Jim
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Are the methods in a class copied or just linked to?

2015-07-02 Thread Danny Yoo 
On Thu, Jul 2, 2015 at 12:30 PM, Jim Mooney Py3.4.3winXP
 wrote:
> When an instance uses a class method, does it actually use the method that
> is in the class object's memory space, or is the method copied to the
> instance?


Unsure.  How would it be observable?



[The following is not beginner material, and mostly
implementation-specific experimentation.  Skip if you are a beginner.]


We know that bound functions are more than code: they need to include
references to the otherwise free variables.  But I think the
underlying function code is shared.  It's not logically required, but
it makes engineering sense.  Otherwise, making bound functions would
be significantly more expensive than the obvious approach of sharing
the part that doesn't change.

As an example:

#
>>> def f(name):
...def g():
...   return name
...return g
...
>>> d = f("danny")
>>> j = f("jim")
>>> d()
'danny'
>>> j()
'jim'
>>> id(d)
4407845544
>>> id(j)
4407847344
>>> d.__code__
", line 2>
>>> j.__code__
", line 2>
#

Here, 'd' and 'j' may be different values, but they share the same
underlying __code__, the compiled code object that defines 'g'.



__code__ and __func__ are special variables used to access the
underlying function values.  They're described in:
https://docs.python.org/3/reference/datamodel.html

The following section in the reference are immediately relevant to
your question:


When an instance method object is created by retrieving a user-defined
function object from a class via one of its instances, its __self__
attribute is the instance, and the method object is said to be bound.
The new method’s __func__ attribute is the original function object.

When a user-defined method object is created by retrieving another
method object from a class or instance, the behaviour is the same as
for a function object, except that the __func__ attribute of the new
instance is not the original method object but its __func__ attribute.

When an instance method object is created by retrieving a class method
object from a class or instance, its __self__ attribute is the class
itself, and its __func__ attribute is the function object underlying
the class method.



From my reading of this, the language reference is guaranteeing that
__func__ is shared.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Are the methods in a class copied or just linked to?

2015-07-02 Thread Peter Otten 
Danny Yoo wrote:

> On Thu, Jul 2, 2015 at 12:30 PM, Jim Mooney Py3.4.3winXP
>  wrote:
>> When an instance uses a class method, does it actually use the method
>> that is in the class object's memory space, or is the method copied to
>> the instance?

> Unsure.  How would it be observable?

[snip implementation details]

My understanding of the question is less involved. I'd suggest:

Create a class Foo with a method bar(), say, instantiate it, replace foo in 
the class and then invoke foo in the previously created instance. 

If the old method is executed the instance must keep a copy, if the new 
method is executed there is probably no copy.

>>> class Foo:
... def bar(self): print("original bar")
... 
>>> foo = Foo()
>>> def modified_bar(self):
... print("modified bar")
... 
>>> Foo.bar = modified_bar
>>> foo.bar()
modified bar

Conclusion: no copy.

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Are the methods in a class copied or just linked to?

2015-07-02 Thread Alan Gauld 

On 02/07/15 20:30, Jim Mooney Py3.4.3winXP wrote:

When an instance uses a class method, does it actually use the method that
is in the class object's memory space, or is the method copied to the
instance?


As soon as you mention memory space in relation to how something
works in Python you know you are barking up the wrong tree.
Memory space is a concept that only applies at the
implementation level in Python, never in the concepts.

Python objects (including classes, which are just another object)
are not necessarily constrained to a single memory space, Classes
are conglomerations of other objects (data and functions) and
linked to other classes (via inheritance). This is not like C++
(or even Java).

The method definitions inside a class define method objects
that live in their own conceptual space as values in the dictionary that 
defines the class structure. Two methods of the same class
could live in wildly different memory locations or they could all be 
part of the same memory area. That's an implementation detail and likely 
to be very different in CPython, Jython and IronPython etc..


So the question should be whether Python objects access the references 
to those methods by linking through the class object or whether they

get copies of the references internally.

I think Danny's answer covers that aspect.

But remember that questions of where Python objects live
in memory are nearly always the wrong question to ask.

Finally, I'm guessing that when you said a "class method" you
actually meant a "method in a class", ie an instance method? You
did not mean an actual class method as defined using the
@classmethod decorator?

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Inheritance, superclass, ‘super’

2015-07-02 Thread Ben Finney 
Steven D'Aprano  writes:

> I mostly agree with what Ben says, comments below.
>
> On Wed, Jul 01, 2015 at 12:48:47PM +1000, Ben Finney wrote:
> > So please use `super`, even in single inheritance. Otherwise you are
> > restricting the usefulness of your class: it can never be used with
> > multiple inheritance.
>
> I cannot disagree with Ben here. Even if you only intend to use single
> inheritance, don't unnecessarily rule out the alternatives. Using
> super is still usually the right thing to do.

I'm glad you agree with the above, since I was quoting your words from
2011 :-)

-- 
 \“… no testimony can be admitted which is contrary to reason; |
  `\   reason is founded on the evidence of our senses.” —Percy Bysshe |
_o__)Shelley, _The Necessity of Atheism_, 1811 |
Ben Finney

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Inheritance, superclass, ‘super’

2015-07-02 Thread Ben Finney 
Alan Gauld  writes:

> On 01/07/15 03:48, Ben Finney wrote:
> > Alan Gauld  writes:
> >
> >> Whilst I agree with the general use of super I'm not sure what
> >> the MRO resolution has to do with this case?
> >
> > When accessing the superclass, the MRO is always relevant
>
> Can you explain that?
> If I access a class explicitly

As explained on other posts: You're not getting “the superclass” if you
do that. Knowing what the superclass is requires run-time determination
of the MRO.

> If MyList is included in a MI lattice that doesn't change the fact
> that MyList inherits directly from list.

Correct. The distinction being drawn here is between “the class from
which MyList directly inherits” versus “the superclass of this class”.
They are not the same, in multiple inheritance.

To refer to the direct parent class, yes it's simple enough to hard-code
that. But that's not the same thing as accessing the superclass, which
must be determined at run-time.

> That depends on what you mean by break it., MI should allow the
> inheriting class to specify which, if any, of its direct superclasses
> methods are invoked.

That “should” is contrary to Python's collaborative multiple inheritance
model. Instead, multiple inheritance in Python entails that any class
should allow the methods of the same name, in all superclasses of this
one, to execute in the sequence determined by ‘super’.

So, if you want behaviour as you describe there, you want something that
Python's mutliple inheritance explicitly won't give you.

-- 
 \“Intellectual property is to the 21st century what the slave |
  `\  trade was to the 16th.” —David Mertz |
_o__)  |
Ben Finney

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] method conflict?

2015-07-02 Thread Jim Mooney Py3.4.3winXP 
Okay, it appears the method in a class has its own ID, but all
instantiations of that method have identical IDs. But what happens if we
have a huge number of instantiations trying to access the identical method
at the same time?

class MyClass:
def setdata(self, data):
self.data = data
def getdata(self):
print(self.data)


>>> MyClass.setdata

>>> id(MyClass.setdata)
40676568
>>> f = MyClass()
>>> g = MyClass()
>>> id(f.setdata)
43576616
>>> id(g.setdata)
43576616
>>> d = MyClass()
>>> id(d.setdata)
43576616


-- 
Jim
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Are the methods in a class copied or just linked to?

2015-07-02 Thread Steven D'Aprano 
On Thu, Jul 02, 2015 at 12:30:23PM -0700, Jim Mooney Py3.4.3winXP wrote:

> When an instance uses a class method, does it actually use the method that
> is in the class object's memory space, or is the method copied to the
> instance?

The first. And it is not just an implementation detail, it is part of 
Python's programming model.

When you define a class in you define methods and other attributes in 
the class namespace:

class C:
attr = 23
def method(self, arg):
pass

'attr' and 'method' exist inside C.__dict__. That is, I believe, more 
than just implementation, I believe that is considered part of the 
language. For instance, this works in IronPython too:

>>> class C:
... def method(self): pass
...
>>> C.__dict__['method']



even though classes in IronPython are based on a completely different 
technology (the .Net CLR) from the standard CPython implementation.

When you create an instance, under normal circumstances it gets its own 
__dict__, but that does *not* get a copy of the method, nor even a 
reference to the original method. In IronPython again:

>>> c = X()
>>> c.__dict__ != C.__dict__  # Separate dictionaries?
True
>>> 'method' in c.__dict__  # in the instance?
False
>>> 'method' in C.__dict__  # in the class?
True


When you call a method, c.method(), Python does a name look-up. 
Simplified, it looks in c.__dict__, then C.__dict__, then the __dict__s 
of each of C's parents (if any).

The upshot of this is that you can actually override the method for a 
specific instance. In Ruby, this technique is called "singleton method", 
in Python it doesn't have a standard name, but it can be useful. It only 
works on classes with an instance __dict__, so it doesn't work on ints, 
floats, strings and other built-in types, and for technical reasons it 
doesn't work on magic __dunder__ methods, and it is a little fiddly to 
get working, but it does work:


>>> class C:
... def method(self):
... return "shared"
...
>>> c = C()
>>> d = C()
>>> from types import MethodType
>>> c.method = MethodType(lambda self: 'overridden', c)
>>> c.method()
'overridden'
>>> d.method()
'shared'



-- 
Steve
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] method conflict?

2015-07-02 Thread Cameron Simpson 

On 02Jul2015 17:39, Jim Mooney Py3.4.3winXP  wrote:

Okay, it appears the method in a class has its own ID, but all
instantiations of that method have identical IDs. But what happens if we
have a huge number of instantiations trying to access the identical method
at the same time?


Um, they all get given access to the method? I'm not sure what you think should 
be happening here:



class MyClass:
   def setdata(self, data):
   self.data = data
   def getdata(self):
   print(self.data)


MyClass.setdata



id(MyClass.setdata)

40676568

f = MyClass()
g = MyClass()


Two instances of the class.


id(f.setdata)

43576616

id(g.setdata)

43576616


Two references to the same method obtained from the class.


d = MyClass()
id(d.setdata)

43576616


And a third.

Steven D'Aprano posted a decent explaination in your other thread ("Are the 
methods in a class copied or just linked to?"); it would have been helpful if 
your post had stayed in that thread - it is essentially the same discussion.


So all your "d.setdata" expressions consult various places in order to find the 
method definition, and they all find it in the class. And so you are given the 
same method every time, and thus the same id.


Cheers,
Cameron Simpson 
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] method conflict?

2015-07-02 Thread Steven D'Aprano 
On Thu, Jul 02, 2015 at 05:39:12PM -0700, Jim Mooney Py3.4.3winXP wrote:

> Okay, it appears the method in a class has its own ID, but all
> instantiations of that method have identical IDs.

I'm not sure what you mean by this. *All* objects have their own ID, and 
instantiations of methods *don't* have identical IDs, although it may 
appear so at first glance (see further below).

py> class C:
... def method(self):
... return 23
...
py> c = C()
py> x = c.method  # grab a reference to the method
py> y = c.method  # and another one
py> print(id(x), id(y))
3082036364 3082100428

*But* that's actually an implementation detail. It just so happens that 
the way CPython currently works, each time you look up a method you get 
a new instance. Technical details are technical, and involve the 
descriptor protocol, which you don't need to understand. I'm happy to go 
into detail if you like, but for now let's just skip over it and just 
call it "magic".

There's nothing in the language specification (that I know of) which 
*promises* that x and y must be separate objects with different IDs.

The underlying function object, retrieved directly from the class 
__dict__, *is* guaranteed to always give the same object:

py> a = C.__dict__['method']
py> b = C.__dict__['method']
py> print(id(a), id(b))
3083756356 3083756356

That's the way dictionaries work.


> But what happens if we
> have a huge number of instantiations trying to access the identical method
> at the same time?

What of it? I don't understand what you think the potential problem is. 
Regardless of whether you access the method wrapper, or the underlying 
function, calling it from multiple places is allowed. Each function call 
is independent of the others.

Naturally, the *action* of the function may not be safe to call twice. 
(It might, say, try to delete the same file twice, and the second time 
you get an error because the file no longer exists.) But the function 
calling infrastructure is safe to do multiple times:

len(a)  # Safe to call it once.
len(b)  # and still safe to call it twice
LEN = len; LEN(c)  # even if you use a different name




> class MyClass:
> def setdata(self, data):
> self.data = data
> def getdata(self):
> print(self.data)
> 
> 
> >>> MyClass.setdata
> 

In Python 3, looking up MyClass.setdata returns the underlying function 
object. But in Python 2, it actually returns an "unbound method" object, 
which is a wrapper around the underlying function. I mention this only 
for completion, in practice it makes little or no difference.


> >>> id(MyClass.setdata)
> 40676568
> >>> f = MyClass()
> >>> g = MyClass()
> >>> id(f.setdata)
> 43576616
> >>> id(g.setdata)
> 43576616
> >>> d = MyClass()
> >>> id(d.setdata)
> 43576616

You are misinterpreting what you are seeing. Python is free to reuse 
IDs. CPython does re-use them, IronPython and Jython do not. 

When you run this line of code:

id(f.setdata)

at least five things happen:

- Python does an attribute search for "setdata" starting with f;

- it finds the *function* f inside the class __dict__;

- because functions are descriptors ("magic"), Python creates a *method* 
  wrapper around the function, and returns that;

- the method gets passed to the id() function, which determines the ID 
  (in this case 43576616) and returns that;

- the method object is then garbage collected, which frees up the ID to 
  be re-used the next time you create a method.


If you did the same thing in (say) Jython, you would get very different 
results, something like this:

 >>> id(f.setdata)
 43
 >>> id(g.setdata)
 44
 >>> id(d.setdata)
 45

(the actual IDs may vary).

But *most of the time* none of this has the least bit of importance to 
your code. Whether you get the same method object or different method 
objects is more or less irrelevant to how your code works. The exception 
is when you start looking at introspection functions like id(), or 
testing for object identity ("is this the same object?") using `is`.

 

-- 
Steve
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] method conflict?

2015-07-02 Thread Mark Lawrence 

On 03/07/2015 01:39, Jim Mooney Py3.4.3winXP wrote:

Okay, it appears the method in a class has its own ID, but all
instantiations of that method have identical IDs. But what happens if we
have a huge number of instantiations trying to access the identical method
at the same time?


I understand that Python "just works".  What do you think happens?



class MyClass:
 def setdata(self, data):
 self.data = data
 def getdata(self):
 print(self.data)



I'd say the only conflict here is writing unneeded boilerplate code in 
Python :)


--
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor