Re: super or not super?
On Tue, Jul 16, 2019 at 3:32 PM Ian Kelly wrote: > > On Sun, Jul 14, 2019 at 7:14 PM Chris Angelico wrote: > > > > On Mon, Jul 15, 2019 at 10:51 AM Paulo da Silva > > wrote: > > > > > > Às 15:30 de 12/07/19, Thomas Jollans escreveu: > > > > On 12/07/2019 16.12, Paulo da Silva wrote: > > > >> Hi all! > > > >> > > > >> Is there any difference between using the base class name or super to > > > >> call __init__ from base class? > > > > > > > > There is, when multiple inheritance is involved. super() can call > > > > different 'branches' of the inheritance tree if necessary. > > > > ... > > > > > > Thank you Jollans. I forgot multiple inheritance. I never needed it in > > > python, so far. > > > > > > > Something to consider is that super() becomes useful even if someone > > else uses MI involving your class. Using super() ensures that your > > class will play nicely in someone else's hierarchy, not just your own. > > Just using super() is not enough. You need to take steps if you want to > ensure that you class plays nicely with MI. For example, consider the > following: > > class C1: > def __init__(self, name): > self._name = name > > class C2(C1): > def __init__(self, name, value): > super().__init__(name) > self._value = value > > This usage of super is just fine for the single-inheritance shown here. But > there are two reasons why this cannot be neatly pulled into an MI > hierarchy. Can you spot both of them? Well, obviously it's violating LSP by changing the signature of __init__, which means that you have to be aware of its position in the hierarchy. If you want things to move around smoothly, you HAVE to maintain a constant signature (which might mean using *args and/or **kwargs cooperatively). I'm not sure what the second issue is, though. It's kinda hard to imagine a real-world situation where this would come up (some things have names but not values, others have values... what third class could you bring in that would make sense here?), so I'm unsure whether you're pointing to "self._value" as a problem. Personally, I'd use "self.value = value" and not try to pretend that it's private. (Not a fan of "self.__value" because subclasses can't use it usefully AT ALL, so unless you're absolutely perfectly maintaining a consistent public signature - which you're not, see the first point - there's no way you can do anything good with it.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: super or not super?
On 16/07/19 09:18, Chris Angelico wrote: > On Tue, Jul 16, 2019 at 3:32 PM Ian Kelly wrote: >> On Sun, Jul 14, 2019 at 7:14 PM Chris Angelico wrote: >>> On Mon, Jul 15, 2019 at 10:51 AM Paulo da Silva >>> wrote: Às 15:30 de 12/07/19, Thomas Jollans escreveu: > On 12/07/2019 16.12, Paulo da Silva wrote: >> Hi all! >> >> Is there any difference between using the base class name or super to >> call __init__ from base class? > There is, when multiple inheritance is involved. super() can call > different 'branches' of the inheritance tree if necessary. > ... Thank you Jollans. I forgot multiple inheritance. I never needed it in python, so far. >>> Something to consider is that super() becomes useful even if someone >>> else uses MI involving your class. Using super() ensures that your >>> class will play nicely in someone else's hierarchy, not just your own. >> Just using super() is not enough. You need to take steps if you want to >> ensure that you class plays nicely with MI. For example, consider the >> following: >> >> class C1: >> def __init__(self, name): >> self._name = name >> >> class C2(C1): >> def __init__(self, name, value): >> super().__init__(name) >> self._value = value >> >> This usage of super is just fine for the single-inheritance shown here. But >> there are two reasons why this cannot be neatly pulled into an MI >> hierarchy. Can you spot both of them? > Well, obviously it's violating LSP by changing the signature of > __init__, which means that you have to be aware of its position in the > hierarchy. If you want things to move around smoothly, you HAVE to > maintain a constant signature (which might mean using *args and/or > **kwargs cooperatively). I guess the second problem is that C1 doesn't call super. Meaning that if someone else uses this in a multiple heritance scheme, and the MRO reaches C1, the call doesn't get propagated to the rest. -- Antoon Pardon -- https://mail.python.org/mailman/listinfo/python-list
Re: super or not super?
On Tue, Jul 16, 2019 at 6:05 PM Antoon Pardon wrote:
>
> On 16/07/19 09:18, Chris Angelico wrote:
> > On Tue, Jul 16, 2019 at 3:32 PM Ian Kelly wrote:
> >> Just using super() is not enough. You need to take steps if you want to
> >> ensure that you class plays nicely with MI. For example, consider the
> >> following:
> >>
> >> class C1:
> >> def __init__(self, name):
> >> self._name = name
> >>
> >> class C2(C1):
> >> def __init__(self, name, value):
> >> super().__init__(name)
> >> self._value = value
> >>
> >> This usage of super is just fine for the single-inheritance shown here. But
> >> there are two reasons why this cannot be neatly pulled into an MI
> >> hierarchy. Can you spot both of them?
> > Well, obviously it's violating LSP by changing the signature of
> > __init__, which means that you have to be aware of its position in the
> > hierarchy. If you want things to move around smoothly, you HAVE to
> > maintain a constant signature (which might mean using *args and/or
> > **kwargs cooperatively).
>
> I guess the second problem is that C1 doesn't call super. Meaning that if
> someone else uses this in a multiple heritance scheme, and the MRO reaches
> C1, the call doesn't get propagated to the rest.
>
My understanding of this tiny hierarchy is that C1 is the root of the
cooperative subtree. Example:
class Pizza: # like C1
def __init__(self, *, name="Generic Pizza", **kw):
self.name = name
if kw: raise TypeError("Unexpected keyword args - did you miss
a mixin?")
class MeatloversPizza(Pizza): # like C2
def __init__(self, *, pepperoni=True, **kw):
super().__init__(**kw)
if pepperoni: self.ingredients = "hot"
class CheesyCrustPizza(Pizza): # alternate mixin
def __init__(self, *, cheesetype="mozzarella", **kw):
super().__init__(**kw)
self.crust = cheesetype + "-stuffed"
class DecadentPizza(MeatloversPizza, CheesyCrustPizza): pass
The only change I've made here is to ensure that everyone's
cooperatively using **kwargs. You can instantiate a
DecadentPizza(cheesetype="cheddar", name="Eat Me", pepperoni=False),
and the different classes will pick up the parts they want. This sort
of hierarchy works just fine, since MeatloversPizza is using super().
The signature of all __init__ functions is (self, *, **kw), with the
variant that the arguments specifically wanted by this class are
listed in a more discoverable way (rather than just popping them out
of kw before calling super's init). It's therefore fine for
MeatloversPizza to chain into CheesyCrustPizza.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
super() in Python 3
Hi, 1. When we use super() in Python 3, we don't pass it the first argument (self). Why? What happens if the first argument is not self? def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) I think it would make more sense to use something like self.super().__init__(*args, **kwargs) or something like this. 2. I want to override a function called build_suite in an inherited class. The function receives an argument "test_labels" which I want to change (I define it if it's not defined), but I don't do anything with the argument "extra_tests". Is it possible to include "extra_tests" in *args, **kwargs and how? I think maybe they will release another version in the future without "extra_tests", or with additional arguments, and I don't want to have to change my code then. def build_suite(self, test_labels=None, extra_tests=None, **kwargs): return super().build_suite(test_labels=test_labels, extra_tests=extra_tests, **kwargs) (Notice there are no *args) Thanks! אורי [email protected] -- https://mail.python.org/mailman/listinfo/python-list
Re: super or not super?
On 16/07/19 10:18, Chris Angelico wrote: > On Tue, Jul 16, 2019 at 6:05 PM Antoon Pardon wrote: >> On 16/07/19 09:18, Chris Angelico wrote: >>> On Tue, Jul 16, 2019 at 3:32 PM Ian Kelly wrote: Just using super() is not enough. You need to take steps if you want to ensure that you class plays nicely with MI. For example, consider the following: class C1: def __init__(self, name): self._name = name class C2(C1): def __init__(self, name, value): super().__init__(name) self._value = value This usage of super is just fine for the single-inheritance shown here. But there are two reasons why this cannot be neatly pulled into an MI hierarchy. Can you spot both of them? >>> Well, obviously it's violating LSP by changing the signature of >>> __init__, which means that you have to be aware of its position in the >>> hierarchy. If you want things to move around smoothly, you HAVE to >>> maintain a constant signature (which might mean using *args and/or >>> **kwargs cooperatively). >> I guess the second problem is that C1 doesn't call super. Meaning that if >> someone else uses this in a multiple heritance scheme, and the MRO reaches >> C1, the call doesn't get propagated to the rest. >> > My understanding of this tiny hierarchy is that C1 is the root of the > cooperative subtree. Example: Why should that stop someone else from using C1 in it's own multiple inheritance scheme? Since object is the root of the whole inheritance hierarchy that should be workable. I find it a bit odd to construct something so narrowly cooperative instead of constructing it in a way that allows easier cooperation with future classes. -- Antoon. -- https://mail.python.org/mailman/listinfo/python-list
Re: Embedding classes' names
On 16Jul2019 10:20, Chris Angelico wrote:
On Tue, Jul 16, 2019 at 10:17 AM DL Neil wrote:
When used, do you embed a class's name within its own code, as a
literal?
[...]
So, what about other situations where one might need to access the
class's own name or that of its/a super-class? eg
class C2(C1):
def __init__(self, fred, barney ):
super().__init__( fred )
self.barney = barney
def __repr__( self ):
return f"C2( { self.fred }, { self.barney }"
### note: 'common practice' of "C2" embedded as constant
How 'purist' do you go, cf YAGNI?
In the case of __repr__, I would most definitely use
self.__class__.__name__, because that way, a subclass can leave repr
untouched and still get decent behaviour.
Yeah, me too, though I spell it "type(self).__name__" for no totally
rational reason.
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
Re: Books for Python 3.7
Thats a good idea. Indeed- take an online course, it is cheap, gets your thru basics and you have an instructor to help if you r stuck. Id vote for online course vs buying a book . On Mon, Jul 15, 2019, 19:43 DL Neil wrote: > Wlfraed probably knows a thing-or-two about kicking-over ants' > nests/wasps' nests... > > Talking about books is one thing. Judging them by asyncio coverage is > quite another - and rather unfair. The use and methods of asyncio have > changed frequently and markedly since '3.0'. Books take time to produce, > sell, buy, and consume... > > > Recommend OP takes a look at the LeanPub series: Python Apprentice; > Journeyman; and Master. They also publish Mike Driscoll - few of which I > have read personally [hangs head in embarrassment/shame], but I do > follow his "Mouse Vs Python" web site... > > > I much prefer to learn from a dead-tree presentation - and likely gain > as much benefit from being able to 'look stuff up', thereafter. However, > YMMV! > > Accordingly, the OP might like to broaden his analysis beyond books > (paper or on-line) and take a look at MOOCs (on-line courses). Each > platform seems to offer something on Python (some good, some tedious, > some little more than puffery) [disclaimer: 'my' courses (non-Python) > are hosted on edX]. > > Just this morning I noted a veritable wave of free courses being > released on the Swayam platform (Indian universities) including: The > Joy of Computing using Python > ( > https://www.classcentral.com/course/swayam-the-joy-of-computing-using-python-14329). > > > > NB sadly I don't have time to attempt/review this myself, but would be > intrigued to hear from you, should you... > > > Last comment (to OP): you should be aware of the Python version > 'covered'. Am not convinced that v3.7 is that important - to a beginner. > Thus, maybe accept v3.5+, and make a practice of reviewing the Python > docs - especially the Release Notes if you think version differences are > important/worthy of particular concern. > > > On 16/07/19 9:24 AM, Andrew Z wrote: > > Gys - hats off. > > > > Basically what Dennis is saying- you dont need a book "about python ". > > Tutorials and general search online will get you further and faster than > > any book. > > > > Blah-blah about myself: > > my bookshelf has 2 technical books, just because i got them to prepare > for > > certifications. > > For my trading app, i had to figure out how to work with asyncio module, > > at the time -2017 , there were no semi- decent explanation for it, let > > alone books. By 2018 it became "the thing" with a ton of books. > > Blah-blah= off > > > > Good luck. > > > > P.s. and if you want to implement your idea really fast and easy - look > at > > the go (golang.org). In my humble opinion- it is super easy and > excellent > > all around. Doing their golangtour is all you need to write a working > app. > > P.p.s. i just started a holy war .. damn. > > > > On Mon, Jul 15, 2019, 17:03 Dennis Lee Bieber > wrote: > > > >> On Mon, 15 Jul 2019 22:17:34 +0200, Gys declaimed > >> the > >> following: > >> > >>> I also would like to have a good book, but have not yet decided which > >>> one. There is a 50$ book on learning Python; the language reference (?) > >>> There is a 50$ book for learning PyQt5 programming of a GUI. There is a > >>> 50$ book on using Python in Pandas for analysing tabular data. > >>> > >> > >> For the language and "batteries" -- every distribution should > >> provide > >> the language reference, and the standard library reference. If one has > a) > >> experience with other languages, the LRM should be sufficient for > learning > >> the syntax; b) skill at interpreting technical documents, one should > become > >> familiar with the contents of the SL reference (this does not mean > >> memorizing all of it -- critical would be the chapters on data types > [which > >> explains what one can do with lists, dictionaries, tuples...] and then > get > >> an idea of the contents of other chapters, so one can look up specifics > for > >> tasks. > >> > >> After that, one ends up with print books that tend to focus on > >> narrow > >> application domains: XML, WxPython, SQLAlchemy, MatPlotLib, Win32 (just > >> from scanning my bookshelf). > >> > >> If one lacks both A and B, one ends up with various editions of > >> "Learning Python", "Programming Python", and "Fluent Python" (among many > >> others). > >> > >> > >> -- > >> Wulfraed Dennis Lee Bieber AF6VN > >> [email protected] > >> http://wlfraed.microdiversity.freeddns.org/ > >> > >> -- > >> https://mail.python.org/mailman/listinfo/python-list > >> > > -- > Regards =dn > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: super() in Python 3
Hi there! A lot of the answers to your questions are at least implied in the Fine Manual (https://docs.python.org/3/library/functions.html#super), but it's not very clear and written more for precision than comprehension. Here's my attempt at explaining :-) On 16/07/2019 11:08, אורי wrote: Hi, 1. When we use super() in Python 3, we don't pass it the first argument (self). Why? Actually the first argument to super() isn't self, it's the class that we want the superclass of. The *second* argument is self. In the normal course of using super() inside a class method, these arguments will almost always be the class itself and the instance the method was called on. Since that's almost always the case, the compiler offers us a short-cut: omit the "cls" and "self" and the compiler will fill them in for us. That way we don't have to repeat ourselves and risk mis-typing something. What happens if the first argument is not self? The first argument of what? I'm not sure what you're getting at here. I think it would make more sense to use something like self.super().__init__(*args, **kwargs) or something like this. That would mean either forbidding classes to have a method named "super" or accepting that users' classes could completely screw up inheritance for their subclasses. 2. I want to override a function called build_suite in an inherited class. The function receives an argument "test_labels" which I want to change (I define it if it's not defined), but I don't do anything with the argument "extra_tests". Is it possible to include "extra_tests" in *args, **kwargs Yes. and how? Don't list it in your parameters :-) def build_suite(self, test_labels=None, *args, **kwargs): ... return super().build_suite(test_labels=test_labels, *args, **kwargs) I think maybe they will release another version in the future without "extra_tests", or with additional arguments, and I don't want to have to change my code then. With a shim layer like this, your chances of getting away with making no changes to your code when an API you use changes are rather small. -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
Re: super() in Python 3
Hi, Thanks for your explanation. But I tried your code and it doesn't work (with Django==1.11.22): File "<...>\site-packages\django\test\runner.py", line 600, in run_tests suite = self.build_suite(test_labels, extra_tests) File "<...>\speedy\core\base\test\models.py", line 35, in build_suite return super().build_suite(test_labels=test_labels, *args, **kwargs) TypeError: build_suite() got multiple values for argument 'test_labels' אורי [email protected] On Tue, Jul 16, 2019 at 3:13 PM Rhodri James wrote: > Hi there! A lot of the answers to your questions are at least implied > in the Fine Manual > (https://docs.python.org/3/library/functions.html#super), but it's not > very clear and written more for precision than comprehension. Here's my > attempt at explaining :-) > > On 16/07/2019 11:08, אורי wrote: > > Hi, > > > > 1. When we use super() in Python 3, we don't pass it the first argument > > (self). Why? > > Actually the first argument to super() isn't self, it's the class that > we want the superclass of. The *second* argument is self. In the > normal course of using super() inside a class method, these arguments > will almost always be the class itself and the instance the method was > called on. Since that's almost always the case, the compiler offers us > a short-cut: omit the "cls" and "self" and the compiler will fill them > in for us. That way we don't have to repeat ourselves and risk > mis-typing something. > > > What happens if the first argument is not self? > > The first argument of what? I'm not sure what you're getting at here. > > > I think it would make more sense to use something like > > self.super().__init__(*args, **kwargs) or something like this. > > That would mean either forbidding classes to have a method named "super" > or accepting that users' classes could completely screw up inheritance > for their subclasses. > > > 2. I want to override a function called build_suite in an inherited > class. > > The function receives an argument "test_labels" which I want to change (I > > define it if it's not defined), but I don't do anything with the argument > > "extra_tests". Is it possible to include "extra_tests" in *args, **kwargs > > Yes. > > > and how? > > Don't list it in your parameters :-) > > def build_suite(self, test_labels=None, *args, **kwargs): > ... > return super().build_suite(test_labels=test_labels, *args, **kwargs) > > > I think maybe they will release another version in the future > > without "extra_tests", or with additional arguments, and I don't want to > > have to change my code then. > > With a shim layer like this, your chances of getting away with making no > changes to your code when an API you use changes are rather small. > > -- > Rhodri James *-* Kynesim Ltd > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: super() in Python 3
[Rearranged and snipped so this makes any kind of sense] On 16/07/2019 16:43, אורי wrote: On Tue, Jul 16, 2019 at 3:13 PM Rhodri James wrote: On 16/07/2019 11:08, אורי wrote: 2. I want to override a function called build_suite in an inherited class. The function receives an argument "test_labels" which I want to change (I define it if it's not defined), but I don't do anything with the argument "extra_tests". Is it possible to include "extra_tests" in *args, **kwargs Yes. and how? Don't list it in your parameters :-) def build_suite(self, test_labels=None, *args, **kwargs): ... return super().build_suite(test_labels=test_labels, *args, **kwargs) > > Thanks for your explanation. But I tried your code and it doesn't work > (with Django==1.11.22): > >File "<...>\site-packages\django\test\runner.py", line 600, in run_tests > suite = self.build_suite(test_labels, extra_tests) >File "<...>\speedy\core\base\test\models.py", line 35, in build_suite > return super().build_suite(test_labels=test_labels, *args, **kwargs) > TypeError: build_suite() got multiple values for argument 'test_labels' Ah, sorry, I was under the impression that you were calling the function with "test_labels" and "extra_tests" as keyword arguments. You can solve the immediate problem by omitting the "=test_labels" bit: return super().build_suite(test_labels, *args, **kwargs) assuming that "test_labels" is the first positional parameter. If it isn't, you've got trouble anyway :-) and you'll need to do some more involved parameter handling. -- Rhodri James *-* Kynesim Ltd -- https://mail.python.org/mailman/listinfo/python-list
Counting Python threads vs C/C++ threads
I'm looking at a performance problem in a large CPython 2.x/3.x codebase with quite a few dependencies. I'm not sure what's causing the slowness yet. The CPU isn't getting hit hard, and I/O on the system appears to be low - but throughput is poor. I'm wondering if it could be CPU-bound Python threads causing the problem (because of the threading+GIL thing). The non-dependency Python portions don't Appear to have much in the way of threading going on based on a quick grep, but csysdig says a process running the code has around 32 threads running - the actual thread count varies, but that's the ballpark. I'm wondering if there's a good way to find two counts of those threads - how many are from CPython code that could run afoul of the GIL, and how many of them are from C/C++ extension modules that wouldn't be responsible for a GIL issue. Does anyone know how to find out these different thread counts, without mounting a huge effort to scrutinize all the code and its dependencies? I'd prefer a tool that can scrutinize the process(es) from the outside, but modifying the code is not out of the question. Please see https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside for more information. Or perhaps https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation Thanks! -- https://mail.python.org/mailman/listinfo/python-list
Re: PyPi twine check does not like my README.rst files anymore
> On 16 Jul 2019, at 05:43, dieter wrote: > > Barry Scott writes: >> I am update some PyPI projects and found that twine was refusing the upload. >> ... >> Failed >> The project's long_description has invalid markup which will not be rendered >> on PyPI. The following syntax errors were detected: >> line 1: Severe: Unexpected section title or transition. >> ... >> The other reason is the show stopper. >> >> It does not allow me to have sections in my README.rst. >> >> The README.rst starts with: >> >> Module namedstruct >> -- >> >> namedstruct encapsulates struct.unpack() and struct.pack() with results >> accessed by name. >> ... > > It may not expect that your "README.rst" starts with a section: > PyPI will provide a title; typical descriptions then have some > introductory text and only then the section structuring begins. I did that and only the first line of the file is rendered. > > If this is not the cause, look at the source of the checking > implementation. It should tell you what is does not like. I was hoping to avoid reading the source that may well be a rabbit hole that keeps going down. Barry > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Class initialization with multiple inheritance
> On 16 Jul 2019, at 01:13, Joseph L. Casale wrote: > > I am trying to find explicit documentation on the initialization logic for a > Base class when multiple exist. For the example in the documentation at > https://docs.python.org/3/tutorial/classes.html#multiple-inheritance, > if Base1 and Base2 both themselves inherited from the same base class, > only Base1 would call __init__ on the subclass, Base2 would not. While > I know this from experience, I need to locate actual documentation, does > Anyone know if it exists? The docs are not clear on this. If you are using super().__init__() in all classes then the key to only calling __init__ once is that super() uses the MRO to find the next class's __init__ to call. The MRO algorithm takes any hierarchy of inheritance and turns it into a predictable ordered list of the classes. The __mro__ attribute allows you to see this. For example (m.py): class CommonBase(object): def __init__( self ): print( 'CommonBase.__init__()' ) super().__init__() class Left(CommonBase): def __init__( self ): print( 'Left.__init__()' ) super().__init__() class Right(CommonBase): def __init__( self ): print( 'Right.__init__()' ) super().__init__() class LeftAndRight(Left, Right): def __init__( self ): print( 'LeftAndRight.__init__()' ) super().__init__() instance = LeftAndRight() Outputs: LeftAndRight.__init__() Left.__init__() Right.__init__() CommonBase.__init__() And here is the MRO for LeftAndRight. >>> import m LeftAndRight.__init__() Left.__init__() Right.__init__() CommonBase.__init__() >>> m.LeftAndRight.__mro__ (, , , , ) >>> Notice that I explicitly derived from object that is the default. But here you see that class object is always there even if I omit it. >>> class Foo: ... pass ... >>> Foo.__mro__ (, ) Barry > > Thanks, > jlc > > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: super or not super?
On Tue, Jul 16, 2019 at 1:21 AM Chris Angelico wrote: > > On Tue, Jul 16, 2019 at 3:32 PM Ian Kelly wrote: > > > > Just using super() is not enough. You need to take steps if you want to > > ensure that you class plays nicely with MI. For example, consider the > > following: > > > > class C1: > > def __init__(self, name): > > self._name = name > > > > class C2(C1): > > def __init__(self, name, value): > > super().__init__(name) > > self._value = value > > > > This usage of super is just fine for the single-inheritance shown here. But > > there are two reasons why this cannot be neatly pulled into an MI > > hierarchy. Can you spot both of them? > > Well, obviously it's violating LSP by changing the signature of > __init__, which means that you have to be aware of its position in the > hierarchy. If you want things to move around smoothly, you HAVE to > maintain a constant signature (which might mean using *args and/or > **kwargs cooperatively). That's pretty close to what I had in mind. Many people treat __init__ as a constructor (I know, it's not) and so long as you're following that doctrine it's not really an LSP violation. But anything else that gets worked into an MI hierarchy has to somehow be compatible with both of these method signatures while also adding whatever new parameters it needs, which is a problem. The usual advice for working with this is to use **kwargs cooperatively as you suggested. *args doesn't work as well because every class needs to know the absolute index of every positional parameter it's interested in, and you can't just trim off the end as you go since you don't know if the later arguments have been consumed yet. With **kwargs you can just pop arguments off the dict as you go. On Tue, Jul 16, 2019 at 2:06 AM Antoon Pardon wrote: > > I guess the second problem is that C1 doesn't call super. Meaning that if > someone else uses this in a multiple heritance scheme, and the MRO reaches > C1, the call doesn't get propagated to the rest. That's it. With single inheritance it's both easy and common to assume that your base class is object, and since object.__init__ does nothing there's no point in calling it. But with MI, that assumption isn't valid. -- https://mail.python.org/mailman/listinfo/python-list
Re: Counting Python threads vs C/C++ threads
I'm going to assume you are on linux. On 16 Jul 2019, at 18:35, Dan Stromberg wrote: > > I'm looking at a performance problem in a large CPython 2.x/3.x codebase > with quite a few dependencies. > > I'm not sure what's causing the slowness yet. The CPU isn't getting hit > hard, and I/O on the system appears to be low - but throughput is poor. > I'm wondering if it could be CPU-bound Python threads causing the problem > (because of the threading+GIL thing). Does top show the process using 100% CPU? > > The non-dependency Python portions don't Appear to have much in the way of > threading going on based on a quick grep, but csysdig says a process > running the code has around 32 threads running - the actual thread count > varies, but that's the ballpark. > > I'm wondering if there's a good way to find two counts of those threads - > how many are from CPython code that could run afoul of the GIL, and how > many of them are from C/C++ extension modules that wouldn't be responsible > for a GIL issue. >From the docs on threading: threading.active_count() Return the number of Thread objects currently alive. The returned count is equal to the length of the list returned by enumerate() . Try running strace on the process to see what system calls its making. You could also connect gdb to the process and find out what code the threads are running. Barry > > Does anyone know how to find out these different thread counts, without > mounting a huge effort to scrutinize all the code and its dependencies? > I'd prefer a tool that can scrutinize the process(es) from the outside, but > modifying the code is not out of the question. > > Please see > https://stackoverflow.com/questions/56958009/how-can-i-tell-how-many-python-threads-a-process-has-from-the-outside > for more information. Or perhaps > https://python-forum.io/Thread-How-to-get-a-count-of-Python-threads-from-the-outside-or-via-code-instrumentation > > Thanks! > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: super or not super?
On Wed, Jul 17, 2019 at 3:58 AM Ian Kelly wrote: > > On Tue, Jul 16, 2019 at 1:21 AM Chris Angelico wrote: > > > > On Tue, Jul 16, 2019 at 3:32 PM Ian Kelly wrote: > > > > > > Just using super() is not enough. You need to take steps if you want to > > > ensure that you class plays nicely with MI. For example, consider the > > > following: > > > > > > class C1: > > > def __init__(self, name): > > > self._name = name > > > > > > class C2(C1): > > > def __init__(self, name, value): > > > super().__init__(name) > > > self._value = value > > > > > > This usage of super is just fine for the single-inheritance shown here. > But > > > there are two reasons why this cannot be neatly pulled into an MI > > > hierarchy. Can you spot both of them? > > > > Well, obviously it's violating LSP by changing the signature of > > __init__, which means that you have to be aware of its position in the > > hierarchy. If you want things to move around smoothly, you HAVE to > > maintain a constant signature (which might mean using *args and/or > > **kwargs cooperatively). > > That's pretty close to what I had in mind. Many people treat __init__ as a > constructor (I know, it's not) If you consider __new__ to be the constructor, and __init__ to be an implementation detail of how the constructor initializes the object, then it comes to the same thing anyway. > and so long as you're following that > doctrine it's not really an LSP violation. But anything else that gets > worked into an MI hierarchy has to somehow be compatible with both of these > method signatures while also adding whatever new parameters it needs, which > is a problem. The usual advice for working with this is to use **kwargs > cooperatively as you suggested. *args doesn't work as well because every > class needs to know the absolute index of every positional parameter it's > interested in, and you can't just trim off the end as you go since you > don't know if the later arguments have been consumed yet. With **kwargs you > can just pop arguments off the dict as you go. Yeah, makes sense. I'd advocate a slightly weakened form of LSP-style equivalence: the function signatures don't have to be *identical*, but they do have to be built with compatibility in mind. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Counting Python threads vs C/C++ threads
On Tue, Jul 16, 2019 at 11:13 AM Barry Scott wrote: > I'm going to assume you are on linux. > Yes, I am. Ubuntu 16.04.6 LTS sometimes, Mint 19.1 other times. On 16 Jul 2019, at 18:35, Dan Stromberg wrote: > > > > I'm looking at a performance problem in a large CPython 2.x/3.x codebase > > with quite a few dependencies. > > > > I'm not sure what's causing the slowness yet. The CPU isn't getting hit > > hard, and I/O on the system appears to be low - but throughput is poor. > > I'm wondering if it could be CPU-bound Python threads causing the problem > > (because of the threading+GIL thing). > > Does top show the process using 100% CPU? > Nope. CPU utilization and disk use are both low. We've been going into top, and then hitting '1' to see things broken down by CPU core (there are 32 of them, probably counting hyperthreads as different cores), but the CPU use is in the teens or so. I've also tried dstat and csysdig. The hardware isn't breaking a sweat, but throughput is poor. > The non-dependency Python portions don't Appear to have much in the way of > > threading going on based on a quick grep, but csysdig says a process > > running the code has around 32 threads running - the actual thread count > > varies, but that's the ballpark. > > > > I'm wondering if there's a good way to find two counts of those threads - > > how many are from CPython code that could run afoul of the GIL, and how > > many of them are from C/C++ extension modules that wouldn't be > responsible > > for a GIL issue. > > From the docs on threading: > > threading.active_count() > > > > Return the number of Thread > > objects currently alive. The returned count is equal to the length of the > list returned by enumerate() > . > Are you on a Mac? https://docs.python.org/2/library/threading.html appears to have some good info. I'll probably try logging threading.active_count() A question arises though: Does threading.active_count() only show Python threads created with the threading module? What about threads created with the thread module? Try running strace on the process to see what system calls its making. > I've tried it, but thank you. It's a good suggestion. I often find that when strace'ing a program, there's a bunch of mostly-irrelevant stuff at Initial Program Load (IPL), but then the main loop fails into a small cycle of system calls. Not with this program. Its main loop is busy and large. You could also connect gdb to the process and find out what code the > threads are running. > I used to use gdb, and wrappers for gdb, when I was doing C code, but I don't have much experience using it on a CPython interrpreter. Would I be doing a "thread apply all bt" or what? I'm guessing those backtraces could facilitate identifying the origin of a thread. Thanks a bunch. -- https://mail.python.org/mailman/listinfo/python-list
Re: Difficulties configuring LTO on macOS with clang
Hmm, removing the path didn’t help either. I ended up defining the `LLVM_AR=/path/to/homebrew/llvm-ar` environment variable after installing clang from brew. I was worried that there would be problems using ar from a different version of the toolchain than the compiler, but things seem to have worked properly. I guess llvm-ar just isn’t packaged with macOS? In any case, this problem is solved for me now. Thanks for your help, Scott On Jul 10, 2019, 16:22 -0400, Barry Scott , wrote: > > > > On 10 Jul 2019, at 15:25, Scott Colby wrote: > > > > Unfortunately that path is absent on my system. Could the issue be that the > > llvm version of ar is not present at all on macOS? What remains confusing > > in that case is that by using CC=cc (and I assume just ar) the LTO > > compilation succeeds, despite cc being a symlink to clang anyway. > > Oh that may be because I have MacPorts installed on my mac > I guess you have macPorts installed as well? > > I see you are using brew openssl. Maybe the macPorts tools > and the brew tools are conflicting? > > What if you remove /opt/local/bin from your PATH does that help configure > find the tools it needs? > > Barry > > > > > > > > Thanks, > > Scott > > > > On Tue, Jul 9, 2019, at 15:39, Barry Scott wrote: > > > > > > > > > > On 9 Jul 2019, at 19:28, Scott Colby wrote: > > > > > > > > Hello all, > > > > > > > > I am having difficulty building Python with `--with-lto` on macOS > > > > 10.14.5. With a clean checkout of the CPython source at the tagged > > > > release for 3.7.4, I ran: > > > > > > > > $ ./configure --prefix=/opt/python3.7.4 --enable-optimizations > > > > --enable-ipv6 --with-lto --with-openssl=$(brew --prefix openssl) > > > > CC=clang > > > > checking for git... found > > > > checking build system type... x86_64-apple-darwin18.6.0 > > > > checking host system type... x86_64-apple-darwin18.6.0 > > > > checking for python3.7... python3.7 > > > > checking for --enable-universalsdk... no > > > > checking for --with-universal-archs... no > > > > checking MACHDEP... checking for --without-gcc... no > > > > checking for --with-icc... no > > > > checking for gcc... clang > > > > checking whether the C compiler works... yes > > > > checking for C compiler default output file name... a.out > > > > checking for suffix of executables... > > > > checking whether we are cross compiling... no > > > > checking for suffix of object files... o > > > > checking whether we are using the GNU C compiler... yes > > > > checking whether clang accepts -g... yes > > > > checking for clang option to accept ISO C89... none needed > > > > checking how to run the C preprocessor... clang -E > > > > checking for grep that handles long lines and -e... /usr/bin/grep > > > > checking for a sed that does not truncate output... /usr/bin/sed > > > > checking for --with-cxx-main=... no > > > > checking for clang++... no > > > > configure: > > > > > > > > By default, distutils will build C++ extension modules with "clang++". > > > > If this is not intended, then set CXX on the configure command line. > > > > > > > > checking for the platform triplet based on compiler characteristics... > > > > darwin > > > > checking for -Wl,--no-as-needed... no > > > > checking for egrep... /usr/bin/grep -E > > > > checking for ANSI C header files... yes > > > > checking for sys/types.h... yes > > > > checking for sys/stat.h... yes > > > > checking for stdlib.h... yes > > > > checking for string.h... yes > > > > checking for memory.h... yes > > > > checking for strings.h... yes > > > > checking for inttypes.h... yes > > > > checking for stdint.h... yes > > > > checking for unistd.h... yes > > > > checking minix/config.h usability... no > > > > checking minix/config.h presence... no > > > > checking for minix/config.h... no > > > > checking whether it is safe to define __EXTENSIONS__... yes > > > > checking for the Android API level... not Android > > > > checking for --with-suffix... > > > > checking for case-insensitive build directory... yes > > > > checking LIBRARY... libpython$(VERSION)$(ABIFLAGS).a > > > > checking LINKCC... $(PURIFY) $(MAINCC) > > > > checking for GNU ld... no > > > > checking for --enable-shared... no > > > > checking for --enable-profiling... no > > > > checking LDLIBRARY... libpython$(VERSION)$(ABIFLAGS).a > > > > checking for ar... ar > > > > checking for readelf... no > > > > checking for a BSD-compatible install... /usr/local/bin/ginstall -c > > > > checking for a thread-safe mkdir -p... /usr/local/bin/gmkdir -p > > > > checking for --with-pydebug... no > > > > checking for --with-assertions... no > > > > checking for --enable-optimizations... yes > > > > checking for --with-lto... yes > > > > checking target system type... x86_64-apple-darwin18.6.0 > > > > checking for -llvm-ar... no > > > > checking for llvm-ar... '' > > > > configure: error: llvm-ar is required for a --with-lto build with clang > > > > but could not be found. > > > > > > I see /opt/local/bin/ll
Re: super or not super?
Às 02:11 de 15/07/19, Chris Angelico escreveu: > On Mon, Jul 15, 2019 at 10:51 AM Paulo da Silva > wrote: >> ... >> >> Thank you Jollans. I forgot multiple inheritance. I never needed it in >> python, so far. >> > > Something to consider is that super() becomes useful even if someone > else uses MI involving your class. Using super() ensures that your > class will play nicely in someone else's hierarchy, not just your own. > Yes, I see. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
