[Python-Dev] super() does not work during class initialization
Hi list, while a class is being initialized in a metaclass, it is not always possible to call classmethods of the class, as they might use super(), which in turn uses __class__, which is not initialized yet. I know that this is a known issue, but well, sometimes it even makes sense to fix already known issues... so I wrote a patch that moves the initialization of __class__ into type.__new__, so that one may use super() in a class once it starts existing. It's issue 23722 on the issue tracker. To illustrate what the problem is, the following code raises a RuntimeError: class Meta(type): def __init__(self, name, bases, dict): super().__init__(name, bases, dict) self.f() class A(metaclass=Meta): @classmethod def f(self): super() it works fine with my patch applied. Technically, my patch slightly changes the semantics of python if a metaclass returns something different in its __new__ than what type.__new__ returns. But I could not find any documentation of the current behavior, and also the tests don't test for it, and I consider the current behavior actually buggy. As an example let me give the following simple Singleton metaclass: class Singleton(type): def __new__(cls, name, bases, dict): return super().__new__(cls, name, bases, dict)() class A(metaclass=Singleton): def test(self): assert(isinstance(__class__, type)) A.test() The current python fails the assertion, while with my patch everything is fine, and I personally think __class__ should always actually refer to the class being defined, which means at the minimum that it is actually, well, a class. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] super() does not work during class initialization
> For folks that haven't looked at the tracker issue: I personally like > the change, but it does involve storing the cell object in a > dunder-variable in the class namespace while it's being defined (until > type.__new__ executes and both populates it and removes it from the > class namespace). I think I had the same weird feelings like you when writing the code. It feels a bit dodgy to abuse the class namespace to transport information from the compiler to type.__new__. Only when I saw that __qualname__ already does it, I decided it was probably not such a bad idea. The current implementation, setting the __class__ cell at the end of __build_class__ doesn't feel dodgy to me, it simply feels wrong. Nothing of real importance should happen there, as it is just an implementation detail. If we are fearing that we clutter namespace too much, we might call the entries @qualname and @cell, then we are sure they will never mask a user's class member. Most of my bad feelings actually only come from the fact that we don't know what we actually put our entries into, as __prepare__ might return something weird that doesn't do what we expect. Given Eric Snow's comment that class namespaces will be ordered by default soon anyways, we might deprecate __prepare__ altogether, eliminating most of the bad feelings. Speaking of which, instead of having OrderedDict re-implemented in C, maybe we could just change the compiler to leave the order in which things are defined in a class in the class namespace, say as a member __order__? Then we could use plain-old dicts for the class namespace, and we would not slow down class creation (not that it matters much), as determining the order would happen at compile time. > Since it introduces new behaviour that's visible to Python level code, > I've suggested that Martin roll the suggestion into his current PEP > 487 (which adds __init_subclass__ to similarly tidy up a few > challenges with the way classes are currently initialised). I thought about that, but while I stumbled over this issue while working on PEP 487, it is actually unrelated. I didn't want people to think that this change is needed to implement PEP 487. And while "and now for something completely different" might be a nice Monty Python reference, it is mostly frowned upon in PEPs as per PEP 1. Alternatively, I could write a new PEP. But I actually think that the change is not worth a PEP, because the changes will move the CPython implementation actually closer to what is documented, so I feel it is more a bugfix than an extension. I have no problem to pursue any of the above paths, how do other people feel about it? Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] super() does not work during class initialization
>> I don't think the compiler can determine the order in
>> all cases. Consider:
>>
>> class Spam:
>>
>> if moon_is_full:
>> alpha = 1
>> beta = 2
>> else:
>> beta = 2
>> alpha = 1
>
> This is also expected to work in class namespaces:
>
> locals()["alpha"] = 1
>
> The language reference suggests it isn't, there's an open tracker
> issue I filed some time ago to suggest clarifying it but haven't found
> the time to actually sit down and come up with readable wording:
> http://bugs.python.org/issue17960
Well, for sure the compiler cannot deduce things happening at runtime,
but it still has an order in which things appear at compile time. And
for nearly all use cases that's by far enough. We cannot stop people
from doing weird things, but unless there is a use case for those
weird things, we don't need to support them.
And I think that tampering with locals() is not really a good idea. In
your issue you mention that you want that so that you can create
enums programatically. This is already doable by writing:
from enum import Enum
from types import new_class
def cb(ns):
ns.update({"a{}".format(i): i for i in range(100)})
return ns
Calc = new_class("Calc", (Enum,), None, cb)
I think this is simple enough that we don't need another way
of doing it.
Btw, going on-topic again, what should I do to get my patch
to make super() work during class initialization into python?
Greetings
Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)
Hi everyone, for those new to the discussion, I am the author of PEP 487, which has been discussed here: https://mail.python.org/pipermail/python-ideas/2015-February/032249.html > The concern is twofold: it breaks proper information hiding/DRY, *and* > it fails silently. It should not be necessary for clients of package > A1 (that uses a decorator built using package B2) to mixin a metaclass > or decorator from package C3 (because B2 implemented its decorators > using C3), just for package A1's decorator to work properly in the > *client package's class*. (And then, of course, this all silently > breaks if you forget, and the breakage might happen at the A1, B2, or > C3 level.) I am just not capable to understand things at such an abstract level, would it be possible to give a simple example on what and how you did things in python 2? > IOW, there's no need to modify the core just to have *that* feature, > since if you control the base class you can already do what PEP 487 > does in essentially every version of Python, ever. If that's all PEP > 487 is going to do, it should just be a PyPI package on a > stdlib-inclusion track, not a change to core Python. It's not > actually adding back any of the dynamicness (dynamicity? > hookability?) that PEP 3115 took away. That was my point. You can find the PyPI package at: https://pypi.python.org/pypi/metaclass Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)
Hi everyone, > > If the PEP 487 metaclass library, > > however, were to just port some bits of my code to Python 3 this could > > be a done deal already and available in *all* versions of Python 3, > > not just the next one. > > Just for the heck of it, here's an actual implementation and demo of > PEP 487, that I've tested with 3.1, 3.2, and 3.4 (I didn't have a copy > of 3.3 handy): The implementation mentioned in PEP 487 itself even works on python 2.7. The whole point of PEP 487 was to reduce PEP 422 so much that it can be written in python and back-ported. This is also discussed in PEP 487. An updated discussion of PEP 487 can be found here: https://mail.python.org/pipermail/python-ideas/2015-March/032538.html Now you want to be able to write decorators whose details are filled in at class creation time. Currently this is typically done by a metaclass. With PEP 487 in place this can also be done using a mixin class. Your point is that you want to be able to use your decorators without having to ask users to also inherit a specific class. I personally don't think that's desirable. Many frameworks out there have such kind of decorators and mandatory base classes and that works fine. The only problem remains once you need to inherit more than one of those classes, as their metaclasses most likely clash. This is what PEP 487 fixes. It fixes this with just some handful lines of python code, but I consider that shortness an advantage. So my opinion is that it is not too hard a requirement to ask a user to inherit a specific mixin class for the sake of using a decorator. What do other people think? Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487 vs 422 (dynamic class decoration)
> When I first wrote PEP 422 I was of the view that "Python 2 allows > class definition postprocessing injection, we should allow it in > Python 3 as well". I've since changed my view to "Having to declare > post-processing of a class definition up front as a decorator, base > class or metaclass is a good thing for readability, as otherwise > there's nothing obvious when reading a class definition that tells you > whether or not postprocessing may happen, so you have to assume its > possible for *every* class definition". Nick, I couldn't agree more with you, yet I think PJ actually brought up a very interesting point. Post-processing is a very common thing these days, and has been re-written so many times that I think it is about time that something like it should be in the standard library. I'm less thinking about decorated methods, more about descriptors. They always have the problem that they don't know which attribute they belong to, so every author of a framework that defines descriptors writes a metaclass which goes through all the descriptors and tells them their attribute name. I propose to have some metaclass in the standard library that does that. I think it would fit nicely in my metaclass module proposed in PEP 487. It would basically do the following: class Metaclass(type): def __init__(self, name, bases, dict): super().__init__(name, bases, dict) for k, v in dict.items(): if hasattr(v, "__post_process__"): v.__post_process__(k, self) So each descriptor could define a __post_process__ hook that tells it the attribute name and also the class it belongs to. This would for sure also work for decorated methods. This should mature on PyPI, then introduced into the standard library, and if demand is really that high, maybe even be introduced into type.__init__. It should be noted that this can also be easily written as a PEP 487 class using __subclass_init__, I just used the classical metaclass notion as I guess people are more used to that. This proposal can actually be seen as an extension to the __class__ and super() mechanism of normal methods: methods currently have the priviledge to know which classes they are defined in, while descriptors don't. So we could unify all this by giving functions a __post_process__ method which sets the __class__ in the function body. This is about the same as what happened when functions got a __get__ method to turn them into object methods. While this all is in the making, PJ could monkey-patch __build_class__ to do the steps described above, until it gets accepted into cpython. So I pose the question to PJ: would such an approach solve the problems you have? Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] async/await PEP
Hi Yury, Hi List, I do certainly like the idea of PEP 492, just some small comments: why do we need two keywords? To me it is not necessarily intuitive when to use async and when to use await (why is it async for and not await for?), so wouldn't it be much simpler, and more symmetric, to just have one keyword? I personally prefer await for that, then it is "await def", and "await for" (and "await with", etc.). Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 557: Data Classes
Hi list, first, a big thanks to the authors of PEP 557! Great idea! For me, the dataclasses were a typical example for inheritance, to be more precise, for metaclasses. I was astonished to see them implemented using decorators, and I was not the only one, citing Guido: > I think it would be useful to write 1-2 sentences about the problem with > inheritance -- in that case you pretty much have to use a metaclass, and the > use of a metaclass makes life harder for people who want to use their own > metaclass (since metaclasses don't combine without some manual > intervention). Python is at a weird point here. At about every new release of Python, a new idea shows up that could be easily solved using metaclasses, yet every time we hesitate to use them, because of said necessary manual intervention for metaclass combination. So I think we have two options now: We could deprecate metaclasses, going down routes like PEP 487's __init_subclass__. Unfortunately, for data classes __init_subclass__ it is too late in the class creation process for it to influence the __slots__ mechanism. A __new_subclass__, that acts earlier, could do the job, but to me that simply sounds like reinventing the wheel of metaclasses. The other option would be to simply make metaclasses work properly. We would just have to define a way to automatically combine metaclasses. Guido once mention once (here: https://mail.python.org/pipermail/python-dev/2017-June/148501.html) that he left out automatic synthesis of combined metaclasses on purpose, but given that this seems to be a major problem, I think it is about time to overthink this decision. So I propose to add such an automatic synthesis. My idea is that a metaclass author can define the __or__ and __ror__ methods for automatic metaclass synthesis. Then if a class C inherits from two classes A and B with metaclasses MetaA and MetaB, the metaclass would be MetaA | MetaB. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
> Metaclasses currently tend to serve two distinct purposes: > > 1. Actually altering the runtime behaviour of a class and its children in > non-standard ways (e.g. enums, ABCs, ORMs) > 2. Boilerplate reduction in class definitions, reducing the amount of code > you need to write as the author of that class > > Nobody has a problem with using metaclasses for the first purpose - that's > what they're for. I am that nobody. The examples you give would be much nicer solved with decorators. Especially for ABCs it would be much better, because the fact that a class is an ABC is explicitly not inherited - its entire reason of existence is to be inherited with all the abstractness going away. And yet, currently the concrete class will still inherit from ABC. The following way of writing ABCs looks much nicer to me: @abstractclass class Spam: @abstractmethod def ham(self): ... The same holds for enums. Inheriting from enums is possible, but weird, given that you cannot add new enums to it. So, especially when comparing to the dataclasses, the following looks appealing to me: @enum class Breakfast: spam = 0 ham = 1 I'm not an expert on ORMs, but they seem to me a lot like data classes in the context of this discussion. I am aware that currently some things are easier done with metaclasses. But given that decorators can create an entirely new class if they like, they have all the power to do what they want, and even in a way much easier understood by people. As an example, here the autoslot decorator: def autoslot(cls): """turn all class variables into slots""" cls.__slots__ = list(cls.__dict__) return type(cls.__name__, cls.__bases__, class.__dict__) So I am personally more and more leaning towards a metaclass-free future. Cheers Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
> Things that will not work if Enum does not have a metaclass: > > list(EnumClass) -> list of enum members > dir(EnumClass) -> custom list of "interesting" items > len(EnumClass) -> number of members > member in EnumClass -> True or False > > - protection from adding, deleting, and changing members > - guards against reusing the same name twice > - possible to have properties and members with the same name (i.e. "value" > and "name") In current Python this is true. But if we would go down the route of PEP 560 (which I just found, I wasn't involved in its discussion), then we could just add all the needed functionality to classes. I would do it slightly different than proposed in PEP 560: classmethods are very similar to methods on a metaclass. They are just not called by the special method machinery. I propose that the following is possible: >>> class Spam: ... @classmethod ... def __getitem__(self, item): ... return "Ham" >>> Spam[3] Ham this should solve most of your usecases. When thinking about how an automatic metaclass combiner would look like, I realized that it should ideally just reproduce the class mro, just with metaclasses. So if a class has an mro of [A, B, C, object], its metaclass should have an mro of unique_everseen([type(A), type(B), type(C), type]). But in this case, why add this layer at all? Just give the class the ability to do everything a metaclass could do, using mechanisms like @classmethod, and we're done. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
>> I do worry that things like your autoslots decorator example might be >> problematic because they create a new class, throwing away a lot of work >> that was already done. But perhaps the right way to address this would be >> to move the decision about the instance layout to a later phase? (Not sure >> if that makes sense though.) >> >> --Guido > > > Just FYI, recreating the class with slots runs into problems with regards to > PEP 3135 (New Super). In attrs we resort to black magic to update the > __class__ cell in existing methods. Being able to add slotness later would > be good, but not that useful for us since we have to support down to 2.7. You're both bringing up an important point here: while in function decorators it is normal to return a completely new function (albeit one that wraps the original), this is close to impossible for classes. You cannot just wrap a class in another one. You may inherit from it, but that's already often not what one wants. While I am not worried about the poor computers having to do a lot of work creating a throwaway class, I do see the problem with the new super. What I would like to see is something like the @wraps decorator for classes, such that you could write something like: def class_decorator(cls): @wraps_class class MyNewCoolClass: """my cool functionality here""" return MyNewCoolClass wraps_class would then copy over everything such that the new class gets it. Unfortunately, this won't work, because of the new super. The new super is about the only thing that cannot be dynamically changed in Python. While it is no problem to make a function a method of a random class (just use __get__), it is not possible to move a function from one class to another, because you cannot change its binding to __class__, which is used by super(). And even if we could, the method whose __class__ we want to change might hide in a wrapper. The current behavior of __class__ is weird, it is set to the class that type.__new__ creates. So even if another metaclasses __new__ or a decorator returns another class, the method's __class__ would still point to the original class, which might even not exist anymore. One might argue that this is due to the fact that it is not well-defined what __class__ should be set to. But this is not true, it is crystal clear: __class__ should be set to the class from whose __dict__ the method was drawn. I thought about a new three-parameter __get__(self, instance, owner, supplier), which would then set __class__ to the supplier. This is a beautiful concept, that is unfortunately not so simple when it comes to methods hidden in wrappers. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 572: Why not := as standard assignment operator?
Hi list, when reading PEP 572 I actually liked it a lot - I think it's actually a cool idea. I think it's actually that cool an idea that it should be made the default way of doing an assignment, over time phasing out the good ole =. This would have several benefits: - people wouldn't have to worry about two different options - different things would have a different look: assignment is :=, keyword args is =, while comparison is ==. Especially beginners would benefit from this clarity. in this case, for sure, we should make it possible to chain :=s, for example by making it bind right-to-left, so that a := b := 3 would be a := (b := 3) I'm sorry if somebody brought that up already, but the discussion has grown so huge that I couldn't read through it entirely. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] async __setitem__
Hi everyone, I was really happy to see PEP 492 - that is really a big step forward. It makes many programming constructs possible in combination with asyncio, which have been cumbersome or impossible until now. One thing that still is impossible is to have __setitem__ be used asynchronously. As an example, I am working with a database that I access over the network, and it is easy to write data = await table[key] to get something out of the database. But getting something in, I have to write something like await table.set(key, value) It would be cool if I could just write await table[key] = value which would, I think, fit in nicely with all of PEP 492. That said, I was hesitating which keyword to use, await or async. PEP 492 says something like async is an adjective, await a command. Then for setitem, technically, I would need an adverb, so "asyncly table[key] = value". But I didn't want to introduce yet another keyword (two different ones for the same thing are already confusing enough), so I arbitrarily chose await. If we are really going for it (I don't see the use case yet, I just mention it for completeness), we could also be extend this to other points where variables are set, e.g. "for await i in something():" which is a bit similar to "async for i in something()" yet different. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Re: Understanding "is not safe" in typeobject.c
Hi Phil, Hi List, unfortunately you do not give enough code to reproduce what you are doing, but just guessing roughly: you say that you have a hierarchy like B -> A -> object, with B and A implemented in C, and then want to use B with a mixin. Programmers with a non-python background then often write class MyClass(B, Mixin): "whatever" this leads to an MRO of MyClass -> B -> Mixin -> A -> object. This is horror if B and A are written in C, because suddenly B needs to do something with Python code if it wants to have to do something with its superclass Mixin, like creating a new object. I am just guessing that this is what your code tries to do. And this is what the comment considers silly. With class MyClass(Mixin, B): "whatever" there is no problem at all. We get an MRO of MyClass -> Mixin -> B -> A -> object. There is no need for B to do anything special, being written in C it already knows by itself how to construct A as well, no need to fiddle with Python at all. In general, it is usually not necessary to deal with super() in C code at all. The statement that there is only single inheritance on the C level becomes obvious once you look at the MRO: that one is always linear, it is always effectively a single inheritance. This is also why you have to call super() only once even if you have multiple superclasses: super() just follows the effective single inheritance of the MRO. Hope that helps. Cheers Martin ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WHTRIPKQMVCQPOYBFWRW6HI6KOBUYJU3/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Understanding "is not safe" in typeobject.c
Hi Greg, Hi List, Greg wrote: > class MyClass(B, Mixin): > >"whatever" > > > > this leads to an MRO of MyClass -> B -> Mixin -> A -> object. > > If you do the tp_new stuff correctly at the C level, you can still > create such a class. The only limitation is that if Mixin has a __new__ > method written in Python, it won't get called. Yes, it is indeed possible to write a class like this. But that would be, as mentioned in the comment, silly. Now there is nothing bad with being silly once in a while, especially while programming that cannot be avoided at times, I am just warning to use such a construct in a wider context. Because the limitation is actually much more general: if the Mixin has any method written in Python, it won't get called, unless you do a lot of heavy lifting. Now why do you write something in C at all? Two reasons: once for speed, which we can effectively exclude here because the super() construct is slow, or because you want to interact with a library written in another language. Unless your library knows about cooperative inheritance, which is very unlikely, the code from class B is not aware that it is not supposed to call the method inherited from A, but the one from the Mixin instead. So by writing class MyClass(B, Mixin): "whatever" I have major problems to understand what you actually want to achieve, what you could not achieve by writing class MyClass(Mixin, B): "whatever" Since one cannot achieve anything with the former that could not be achieved by the latter, just with much less hassle, I would call the former code silly. That said, I might be wrong and there is indeed something one can achieve with the former but not with the latter, I would be very pleased to hear about this, so please come forward! (For sure, all is based on B and A being written in C, not Python) Cheers Martin ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/VQPMUJKMZ44MLUPULFERPLMM5IXEKFYE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] PEP 487: Simpler customization of class creation
Hi list, using metaclasses in Python is a very flexible method of customizing class creation, yet this customization comes at a cost: once you want to combine two classes with different metaclasses, you run into problems. This is why I proposed PEP 487 (see https://github.com/tecki/peps/blob/pep487/pep-0487.txt, which I also attached here for ease of discussion), proposing a simple hook into class creation, with which one can override in subclasses such that sub-subclasses get customized accordingly. Otherwise, the standard Python inheritance rules apply (super() and the MRO). I also proposed to store the order in which attributes in classes are defined. This is exactly the same as PEP 520, discussed here recently, just that unfortunately we chose different names, but I am open for suggestions for better names. After having gotten good feedback on python-ideas (see https://mail.python.org/pipermail/python-ideas/2016-February/038305.html) and from IPython traitlets as a potential user of the feature (see https://mail.scipy.org/pipermail/ipython-dev/2016-February/017066.html, and the code at https://github.com/tecki/traitlets/commits/pep487) I implemented a pure Python version of this PEP, to be introduced into the standard library. I also wrote a proof-of-concept for another potential user of this feature, django forms, at https://github.com/tecki/django/commits/no-metaclass. The code to be introduced into the standard library can be found at https://github.com/tecki/cpython/commits/pep487 (sorry for using github, I'll submit something using hg once I understand that toolchain). It introduces a new metaclass types.Type which contains the machinery, and a new base class types.Object which uses said metaclass. The naming was chosen to clarify the intention that eventually those classes may be implemented in C and replace type and object. As above, I am open to better naming. As a second step, I let abc.ABCMeta inherit from said types.Type, such that an ABC may also use the features of my metaclass, without the need to define a new mixing metaclass. I am looking forward to a lot of comments on this! Greetings Martin The proposed PEP for discussion: PEP: 487 Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann , Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6 Post-History: 27-Feb-2015, 5-Feb-2016 Replaces: 422 Abstract Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``__init_subclass__`` hook in the class body, a hook to initialize attributes, and a way to keep the order in which attributes are defined. Those hooks should at first be defined in a metaclass in the standard library, with the option that this metaclass eventually becomes the default ``type`` metaclass. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery. Background == Metaclasses are a powerful tool to customize class creation. They have, however, the problem that there is no automatic way to combine metaclasses. If one wants to use two metaclasses for a class, a new metaclass combining those two needs to be created, typically manually. This need often occurs as a surprise to a user: inheriting from two base classes coming from two different libraries suddenly raises the necessity to manually create a combined metaclass, where typically one is not interested in those details about the libraries at all. This becomes even worse if one library starts to make use of a metaclass which it has not done before. While the library itself continues to work perfectly, suddenly every code combining those classes with classes from another library fails. Proposal While there are many possible ways to use a metaclass, the vast majority of use cases falls into just three categories: some initialization code running after class creation, the initalization of descriptors and keeping the order in which class attributes were defined. Those three use cases can easily be performed by just one metaclass. If this metaclass is put into the standard library, and all libraries that wish to customize class creation use this very metaclass, no combination of metaclasses is necessary anymore. Said metaclass should live in the ``types`` module under the name ``Type``. This should hint the user that in the future, this metaclass may become the default metaclass ``type``. The three use cases are achieved as follows: 1. The metaclass contains an ``__init_subclass__`` hook that initializes all subclasses of a given cla
Re: [Python-Dev] PEP 487: Simpler customization of class creation
Hi Eric, hi List, > I'd be glad to give feedback on this, probably later today or > tomorrow. In particular, I'd like to help resolve the intersection > with PEP 520. :) Thanks in advance! Let me already elaborate on the differences, so that others can follow: You chose the name "__definition_order__", I chose "__attribute_order__", I am fine with either, what are other people's opinions? The bigger difference is actually the path to inclusion into Python: my idea is to first make it a standard library feature, with the later option to put it into the C core, while you want to put the feature directly into the C core. Again I'm fine with either, as long as the feature is eventually in. As a side note, you propose to use OrderedDict as the class definition namespace, and this is exactly how I implemented it. Nonetheless, I would like to keep this fact as an implementation detail, such that other implementations of Python (PyPy comes to mind) or even CPython at a later time may switch to a different way to implement this feature. I am thinking especially about the option to determine the -_order__ already at compile time. Sure, this would mean that someone could trick us by dynamically changing the order of attribute definition, but I would document that as an abuse of the functionality with undocumented outcome. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 487: Simpler customization of class creation
Hi list, just recently, I posted about the implementation of PEP 487. The discussion quickly diverted to PEP 520, which happened to be strongly related. Hoping to get some comments about the rest of PEP 487, I took out the part that is also in PEP 520. I attached the new version of the PEP. The implementation can be found on the Python issue tracker: http://bugs.python.org/issue27366 So PEP 487 is about simplifying the customization of class creation. Currently, this is done via metaclasses, which are very powerful, but often inflexible, as it is hard to combine two metaclasses. PEP 487 proposes a new metaclass which calls a method on all newly created subclasses. This way, in order to customize the creation of subclasses, one just needs to write a simple method. An absolutely classic example for metaclasses is the need to tell descriptors who they belong to. There are many large frameworks out there, e.g. enthought's traits, IPython's traitlets, Django's forms and many more. Their problem is: they're all fully incompatible. It's really hard to inherit from two classes which have different metaclasses. PEP 487 proposes to have one simple metaclass, which can do all those frameworks need, making them all compatible. As an example, imagine the framework has a generic descriptor called Integer, which describes, well, an integer. Typically you use it like that: class MyClass(FrameworkBaseClass): my_value = Integer() how does my_value know how it's called, how it should put its data into the object's __dict__? Well, this is what the framework's metaclass is for. With PEP 487, a framework doesn't need to declare an own metaclass anymore, but simply uses types.Object of PEP 487 as a base class: class FrameworkBaseClass(types.Object): def __init_subclass__(cls): super().__init_subclass__() for k, v in cls.__dict__.items(): if isinstance(v, FrameworkDescriptorBase): v.__set_owner__(cls, name) and all the framework's descriptors know their name. And if another framework should be used as well: no problem, they just work together easily. Actually, the above example is just that common, that PEP 487 includes it directly: a method __set_owner__ is called for every descriptor. That could make most descriptors in frameworks work out of the box. So now I am hoping for comments! Greetings Martin New version of the PEP follows: PEP: 487 Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann , Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6 Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016 Replaces: 422 Abstract Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``__init_subclass__`` hook in the class body, a hook to initialize attributes. Those hooks should at first be defined in a metaclass in the standard library, with the option that this metaclass eventually becomes the default ``type`` metaclass. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power Python's metaclass machinery. Background == Metaclasses are a powerful tool to customize class creation. They have, however, the problem that there is no automatic way to combine metaclasses. If one wants to use two metaclasses for a class, a new metaclass combining those two needs to be created, typically manually. This need often occurs as a surprise to a user: inheriting from two base classes coming from two different libraries suddenly raises the necessity to manually create a combined metaclass, where typically one is not interested in those details about the libraries at all. This becomes even worse if one library starts to make use of a metaclass which it has not done before. While the library itself continues to work perfectly, suddenly every code combining those classes with classes from another library fails. Proposal While there are many possible ways to use a metaclass, the vast majority of use cases falls into just three categories: some initialization code running after class creation, the initalization of descriptors and keeping the order in which class attributes were defined. Those three use cases can easily be performed by just one metaclass. If this metaclass is put into the standard library, and all libraries that wish to customize class creation use this very metaclass, no combination of metaclasses is necessary anymore. Said metaclass should live in the ``types`` module under the name ``Type``. This
Re: [Python-Dev] PEP 487: Simpler customization of class creation
Hi Nick, hi List, thanks for the good comments! I'll update the PEP and the implementation soon, as discussed at the end, I'll do it in C this time. For now just some quick responses: > This part isn't entirely clear to me, so you may want to give some > Python pseudo-code that: I will actually give the actual code, it's just 10 lines, that should be understandable. > - is explicit regarding exactly when this new code runs in the type > creation process > - whether the __set_owner__ hooks are called before or after > __init_subclass__ runs, or only when the subclass calls up to > super().__init_subclass__, and the implications of each choice (either > descriptors see a partially initialised class object, or init_subclass > sees partially initialised descriptor objects, or that choice is > delegated to individual subclasses) > - how the list of objects to be checked for "__set_owner__" methods is > retrieved (presumably via "ns.items()" on the class definition > namespace, but the PEP should be explicit) > > For the second point, my personal preference would be for descriptors > to have their owner set first and independently of __init_subclass__ > super calls (as it seems more likely that __init_subclass__ will > depend on having access to fully initialised descriptors than the > other way around). I intuitively programmed it the other way around, but I get your point and will change it. I agree that it should not be done in super().__init_subclass__ as people often forget to call it, or do weird things. > Honestly though, I'm not sure this additional user-visible complexity > is worth it - "The default type metaclass has this new behaviour" is a > lot easier to document and explain than "We added a new opt-in > alternate metaclass that you can use if you want, and in the next > version that will just become an alias for the builtin types again". > We'd also end up being stuck with types.Type and types.Object as > aliases for the type and object builtins forever (with the associated > "How does 'class name:' or 'class name(object)' differ from 'class > name(types.Object)'?" question and "It doesn't, unless you're using > Python 3.6" answer for folks learning the language for the first > time). My idea with a stepped approach was to have a chance to look how people use the feature, so that when we make it standard eventually we actually get it right. In the end, this is a maintainability question. I am fully OK with following experienced core developers here, so if the general idea here is that a two-step approach is not needed, then no problem, let's do it in the Python core directly! Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP487: Simpler customization of class creation
Hi list,
so this is the next round for PEP 487. During the last round, most of
the comments were in the direction that a two step approach for
integrating into Python, first in pure Python, later in C, was not a
great idea and everything should be in C directly. So I implemented it
in C, put it onto the issue tracker here:
http://bugs.python.org/issue27366, and also modified the PEP
accordingly.
For those who had not been in the discussion, PEP 487 proposes to add
two hooks, __init_subclass__ which is a classmethod called whenever a
class is subclassed, and __set_owner__, a hook in descriptors which
gets called once the class the descriptor is part of is created.
While implementing PEP 487 I realized that there is and oddity in the
type base class: type.__init__ forbids to use keyword arguments, even
for the usual three arguments it has (name, base and dict), while
type.__new__ allows for keyword arguments. As I plan to forward any
keyword arguments to the new __init_subclass__, I stumbled over that.
As I write in the PEP, I think it would be a good idea to forbid using
keyword arguments for type.__new__ as well. But if people think this
would be to big of a change, it would be possible to do it
differently.
Hoping for good comments
Greetings
Martin
The PEP follows:
PEP: 487
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann ,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.6
Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016, 2-Jul-2016
Replaces: 422
Abstract
Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.
This PEP proposes to instead support a wide range of customisation
scenarios through a new ``__init_subclass__`` hook in the class body,
and a hook to initialize attributes.
The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.
Background
==
Metaclasses are a powerful tool to customize class creation. They have,
however, the problem that there is no automatic way to combine metaclasses.
If one wants to use two metaclasses for a class, a new metaclass combining
those two needs to be created, typically manually.
This need often occurs as a surprise to a user: inheriting from two base
classes coming from two different libraries suddenly raises the necessity
to manually create a combined metaclass, where typically one is not
interested in those details about the libraries at all. This becomes
even worse if one library starts to make use of a metaclass which it
has not done before. While the library itself continues to work perfectly,
suddenly every code combining those classes with classes from another library
fails.
Proposal
While there are many possible ways to use a metaclass, the vast majority
of use cases falls into just three categories: some initialization code
running after class creation, the initalization of descriptors and
keeping the order in which class attributes were defined.
The first two categories can easily be achieved by having simple hooks
into the class creation:
1. An ``__init_subclass__`` hook that initializes
all subclasses of a given class.
2. upon class creation, a ``__set_owner__`` hook is called on all the
attribute (descriptors) defined in the class, and
The third category is the topic of another PEP 520.
As an example, the first use case looks as follows::
>>> class SpamBase:
...# this is implicitly a @classmethod
...def __init_subclass__(cls, **kwargs):
...cls.class_args = kwargs
...super().__init_subclass__(cls, **kwargs)
>>> class Spam(SpamBase, a=1, b="b"):
...pass
>>> Spam.class_args
{'a': 1, 'b': 'b'}
The base class ``object`` contains an empty ``__init_subclass__``
method which serves as an endpoint for cooperative multiple inheritance.
Note that this method has no keyword arguments, meaning that all
methods which are more specialized have to process all keyword
arguments.
This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion.
The second part of the proposal adds an ``__set_owner__``
initializer for class attributes, especially if they are descriptors.
Descriptors are defined in the body of a
class, but they do not know anything about that class, they do not
even know the name they are accessed with. They do g
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hi Nick, thanks for the nice review! > I think making __init_subclass__ implicitly a class method is still > the right thing to do if this proposal gets accepted, we'll just want > to see if we can do something to tidy up that aspect of the > documentation at the same time. I could write some documentation, I just don't know where to put it. I personally have no strong feelings whether __init_subclass__ should be implicitly a @classmethod or not - but as the general consensus here seemed to hint making it implicit is better, this is how I wrote it. >> While implementing PEP 487 I realized that there is and oddity in the >> type base class: type.__init__ forbids to use keyword arguments, even >> for the usual three arguments it has (name, base and dict), while >> type.__new__ allows for keyword arguments. As I plan to forward any >> keyword arguments to the new __init_subclass__, I stumbled over that. >> As I write in the PEP, I think it would be a good idea to forbid using >> keyword arguments for type.__new__ as well. But if people think this >> would be to big of a change, it would be possible to do it >> differently. > > [some discussion cut out] > > I think the PEP could be accepted without cleaning this up, though - > it would just mean __init_subclass__ would see the "name", "bases" and > "dict" keys when someone attempted to use keyword arguments with the > dynamic type creation APIs. Yes, this would be possible, albeit a bit ugly. I'm not so sure whether backwards compatibility is so important in this case. It is very easy to change the code to the fully cleaned up version Looking through old stuff I found http://bugs.python.org/issue23722, which describes the following problem: at the time __init_subclass__ is called, super() doesn't work yet for the new class. It does work for __init_subclass__, because it is called on the base class, but not for calls to other classmethods it does. This is a pity especially because also the two argument form of super() cannot be used as the new class has no name yet. The problem is solvable though. The initializations necessary for super() to work properly simply should be moved before the call to __init_subclass__. I implemented that by putting a new attribute into the class's namespace to keep the cell which will later be used by super(). This new attribute would be remove by type.__new__ again, but transiently it would be visible. This technique has already been used for __qualname__. The issue contains a patch that fixes that behavior, and back in the day you proposed I add the problem to the PEP. Should I? Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 487: Simpler customization of class creation
Hi Guido, sorry I missed your post... >> One of the big issues that makes library authors reluctant to use >> metaclasses >> (even when they would be appropriate) is the risk of metaclass conflicts. > > Really? I've written and reviewed a lot of metaclasses and this has never > worried me. The problem is limited to multiple inheritance, right? I worry a > lot about MI being imposed on classes that weren't written with MI in mind, > but I've never particularly worried about the special case of metaclasses. Yes, the problem only arises with MI. Unfortunately, that's not uncommon: if you want to implement an ABC with a class from a framework which uses metaclasses, you have a metaclass conflict. So then you start making MyFrameworkABCMeta-classes. The worst is if you already have a framework with users out there. No way you add a metaclass to your class, however convenient it would be. Because you never now if some user out there had gotten the idea to implement an ABC with it. Sure, you could let your metaclass inherit from ABCMeta, but is this really how it should be done? (This has already been mentioned by others over at python-ideas: https://mail.python.org/pipermail/python-ideas/2016-February/038506.html) Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
> This is an area of exceeding subtlety (and also not very well
> documented/specified, probably). I'd worry that changing anything here
> might break some code. When a metaclass overrides neither __init__ nor
> __new__, keyword args will not work because type.__init__ forbids
> them. However when a metaclass overrides them and calls them using
> super(), it's quite possible that someone ended up calling
> super().__init__() with three positional args but super().__new__()
> with keyword args, since the call sites are distinct (in the overrides
> for __init__ and __new__ respectively).
>
> What's your argument for changing this, apart from a desire for more
> regularity?
The implementation gets much simpler if __new__ doesn't take keyword
arguments. It's simply that if it does, I have to filter out __new__'s
three arguments.
That's easily done in Python, unfortunately not so much in C.
So we have two options: either type.__new__ is limited to accepting positional
arguments only, possibly breaking some code, but which could be changed
easily. This leads to a pretty simple implementation: pass over
keyword arguments
to __init_subclass__, that's it.
The other option is: filter out name, bases and dict from the keyword arguments
If people think that backwards compatibility is that important, I
could do that. But
that just leaves quite some code in places where there is already a lot of
complicated code.
Nick proposed a compromise, just don't filter for name, bases and dict, and
pass them over to __init_subclass__. Then the default implementation of
__init_subclass__ must support those three keyword arguments and do
nothing with them.
I'm fine with all three solutions, although I have a preference for the first.
I think passing keyword arguments to type.__new__ is already really rare
and if it does exist, it's super easy to fix.
> I'm confused. In the above example it would seem that the keyword args
> {'a': 1, 'b': 2} are passed right on to super9).__init_subclass__().
> Do you mean that it ignores all keyword args? Or that it has no
> positional args? (Both of which would be consistent with the example.)
The example is just wrong. I'll fix it.
> Can you state exactly at which point during class initialization
> __init_class__() is called? (Surely by now, having implemented it, you
> know exactly where. :-)
Further down in the PEP I give the exact
> [This is as far as I got reviewing when the weekend activities
> interrupted me. In the light of ongoing discussion I'm posting this
> now -- I'll continue later.]
I hope you had a good weekend not thinking too much about
metaclasses...
Greetings
Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __qualname__ exposed as a local variable: standard?
Hi list, > I noticed __qualname__ is exposed by locals() while defining a class. This > is handy but I'm not sure about its status: is it standard or just an > artifact of the current implementation? (btw, the pycodestyle linter -former > pep8- rejects its usage). I was unable to find any reference to this > behavior in PEP 3155 nor in the language reference. I would like to underline the importance of this question, and give some background, as it happens to resonate with my work on PEP 487. The __qualname__ of a class is originally determined by the compiler. One might now think that it would be easiest to simply set the __qualname__ of a class once the class is created, but this is not as easy as it sounds. The class is created by its metaclass, so possibly by user code, which might create whatever it wants, including something which is not even a class. So the decision had been taken to sneak the __qualname__ through user code, and pass it to the metaclasses __new__ method as part of the namespace, where it is deleted from the namespace again. This has weird side effects, as the namespace may be user code as well, leading to the funniest possible abuses, too obscene to publish on a public mailing list. A very different approach has been taken for super(). It has similar problems: the zero argument version of super looks in the surrounding scope for __class__ for the containing class. This does not exist yet at the time of creation of the methods, so a PyCell is put into the function's scope, which will later be filled. It is actually filled with whatever the metaclasses __new__ returns, which may, as already be said, anything (some sanity checks are done to avoid crashing the interpreter). I personally prefer the first way of doing things like for __qualname__, even at the cost of adding things to the classes namespace. It could be moved after the end of the class definition, such that it doesn't show up while the class body is executed. We might also rename it to __@qualname__, this way it cannot be accessed by users in the class body, unless they look into locals(). This has the large advange that super() would work immediately after the class has been defined, i.e. already in the __new__ of the metaclass after it has called type.__new__. All of this changes the behavior of the interpreter, but we are talking about undocumented behavior. The changes necessary to make super() work earlier are store in http://bugs.python.org/issue23722 Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP487: Simpler customization of class creation
Hi list, another round for PEP 487, is there any chance it still makes it into Python 3.6? The PEP should be effectively done, I updated the examples in it, given that I implemented the PEP I could actually test the examples, so now they work. The implementation is at http://bugs.python.org/issue27366, including documentation and tests. Unfortunately nobody has reviewed the patch yet. The new version of the PEP is attached. Greetings Martin PEP: 487 Title: Simpler customisation of class creation Version: $Revision$ Last-Modified: $Date$ Author: Martin Teichmann , Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Feb-2015 Python-Version: 3.6 Post-History: 27-Feb-2015, 5-Feb-2016, 24-Jun-2016, 2-Jul-2016, 13-Jul-2016 Replaces: 422 Abstract Currently, customising class creation requires the use of a custom metaclass. This custom metaclass then persists for the entire lifecycle of the class, creating the potential for spurious metaclass conflicts. This PEP proposes to instead support a wide range of customisation scenarios through a new ``__init_subclass__`` hook in the class body, and a hook to initialize attributes. The new mechanism should be easier to understand and use than implementing a custom metaclass, and thus should provide a gentler introduction to the full power of Python's metaclass machinery. Background == Metaclasses are a powerful tool to customize class creation. They have, however, the problem that there is no automatic way to combine metaclasses. If one wants to use two metaclasses for a class, a new metaclass combining those two needs to be created, typically manually. This need often occurs as a surprise to a user: inheriting from two base classes coming from two different libraries suddenly raises the necessity to manually create a combined metaclass, where typically one is not interested in those details about the libraries at all. This becomes even worse if one library starts to make use of a metaclass which it has not done before. While the library itself continues to work perfectly, suddenly every code combining those classes with classes from another library fails. Proposal While there are many possible ways to use a metaclass, the vast majority of use cases falls into just three categories: some initialization code running after class creation, the initalization of descriptors and keeping the order in which class attributes were defined. The first two categories can easily be achieved by having simple hooks into the class creation: 1. An ``__init_subclass__`` hook that initializes all subclasses of a given class. 2. upon class creation, a ``__set_owner__`` hook is called on all the attribute (descriptors) defined in the class, and The third category is the topic of another PEP 520. As an example, the first use case looks as follows:: >>> class QuestBase: ...# this is implicitly a @classmethod ...def __init_subclass__(cls, swallow, **kwargs): ...cls.swallow = swallow ...super().__init_subclass__(**kwargs) >>> class Quest(QuestBase, swallow="african"): ...pass >>> Quest.swallow 'african' The base class ``object`` contains an empty ``__init_subclass__`` method which serves as an endpoint for cooperative multiple inheritance. Note that this method has no keyword arguments, meaning that all methods which are more specialized have to process all keyword arguments. This general proposal is not a new idea (it was first suggested for inclusion in the language definition `more than 10 years ago`_, and a similar mechanism has long been supported by `Zope's ExtensionClass`_), but the situation has changed sufficiently in recent years that the idea is worth reconsidering for inclusion. The second part of the proposal adds an ``__set_owner__`` initializer for class attributes, especially if they are descriptors. Descriptors are defined in the body of a class, but they do not know anything about that class, they do not even know the name they are accessed with. They do get to know their owner once ``__get__`` is called, but still they do not know their name. This is unfortunate, for example they cannot put their associated value into their object's ``__dict__`` under their name, since they do not know that name. This problem has been solved many times, and is one of the most important reasons to have a metaclass in a library. While it would be easy to implement such a mechanism using the first part of the proposal, it makes sense to have one solution for this problem for everyone. To give an example of its usage, imagine a descriptor representing weak referenced values:: import weakref class WeakAttribute: def __get__(self, instance, owner): return instance.__dict__[self.name]() def __set__(self, instance, value): instance.__dict__[self.nam
Re: [Python-Dev] Status of Python 3.6 PEPs?
Hi, > "PEP 487 -- Simpler customisation of class creation" > https://www.python.org/dev/peps/pep-0487/ > => draft I would like to get that into Python 3.6. It's already implemented, including documentation and tests (http://bugs.python.org/issue27366). Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hi Guido, Hi list, Thanks for the nice review! I applied followed up your ideas and put it into a github pull request: https://github.com/python/peps/pull/53 Soon we'll be working there, until then, some responses to your comments: > I wonder if this should be renamed to __set_name__ or something else > that clarifies we're passing it the name of the attribute? The method > name __set_owner__ made me assume this is about the owning object > (which is often a useful term in other discussions about objects), > whereas it is really about telling the descriptor the name of the > attribute for which it applies. The name for this has been discussed a bit already, __set_owner__ was Nick's idea, and indeed, the owner is also set. Technically, __set_owner_and_name__ would be correct, but actually I like your idea of __set_name__. > That (inheriting type from type, and object from object) is very > confusing. Why not just define new classes e.g. NewType and NewObject > here, since it's just pseudo code anyway? Actually, it's real code. If you drop those lines at the beginning of the tests for the implementation (as I have done here: https://github.com/tecki/cpython/blob/pep487b/Lib/test/test_subclassinit.py), the test runs on older Pythons. But I see that my idea to formulate things here in Python was a bad idea, I will put the explanation first and turn the code into pseudo-code. >> def __init__(self, name, bases, ns, **kwargs): >> super().__init__(name, bases, ns) > > What does this definition of __init__ add? It removes the keyword arguments. I describe that in prose a bit down. >> class object: >> @classmethod >> def __init_subclass__(cls): >> pass >> >> class object(object, metaclass=type): >> pass > > Eek! Too many things named object. Well, I had to do that to make the tests run... I'll take that out. >> In the new code, it is not ``__init__`` that complains about keyword >> arguments, >> but ``__init_subclass__``, whose default implementation takes no arguments. >> In >> a classical inheritance scheme using the method resolution order, each >> ``__init_subclass__`` may take out it's keyword arguments until none are >> left, >> which is checked by the default implementation of ``__init_subclass__``. > > I called this out previously, and I am still a bit uncomfortable with > the backwards incompatibility here. But I believe what you describe > here is the compromise proposed by Nick, and if that's the case I have > peace with it. No, this is not Nick's compromise, this is my original. Nick just sent another mail to this list where he goes a bit more into the details, I'll respond to that about this topic. Greetings Martin P.S.: I just realized that my changes to the PEP were accepted by someone else than Guido. I am a bit surprised about that, but I guess this is how it works? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hi Nick, hi list, > It would be worth spelling out the end result of the new behaviour in > the PEP to make sure it's what we want. Trying to reason about how > that code works is difficult, but looking at some class definition > scenarios and seeing how they behave with the old semantics and the > new semantics should be relatively straightforward (and they can > become test cases for the revised implementation). I agree with that. All the examples that Nick gives should work the same way as they used to. I'll turn them into tests. I hope it is fine if the error messages change slightly, as long as the error type stays the same? Let's look at an example that won't work anymore if my proposal goes throught: class MyType(type): def __new__(cls, name, bases, namespace): return super().__new__(cls, name=name, bases=bases, dict=namespace) I guess this kind of code is pretty rare. Note that I need to call the third parameter dict, as this is how it is named. Even if there is code out there like that, it would be pretty easy to change. Just in case someone wondered: def __new__(cls, **kwargs): return super().__new__(cls, **kwargs) doesn't work now and won't work afterwards, as the interpreter calls the metaclass with positional arguments. That said, it would be possible, at the cost of quite some lines of code, to make it fully backwards compatible. If the consensus is that this is needed, I'll change the PEP and code accordingly. My proposal also has the advantage that name, bases and dict may be used as class keyword arguments. At least for name I see a usecase: class MyMangledClass(BaseClass, name="Nice class name"): pass Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hi Guido, Hi Nick, Hi list, so I just updated PEP 487, you can find it here: https://github.com/python/peps/pull/57 if it hasn't already been merged. There are no substantial updates there, I only updated the wording as suggested, and added some words about backwards compatibility as hinted by Nick. Greetings Martin 2016-07-14 17:47 GMT+02:00 Guido van Rossum : > I just reviewed the changes you made, I like __set_name__(). I'll just > wait for your next update, incorporating Nick's suggestions. Regarding > who merges PRs to the PEPs repo, since you are the author the people > who merge don't pass any judgment on the changes (unless it doesn't > build cleanly or maybe if they see a typo). If you intend a PR as a > base for discussion you can add a comment saying e.g. "Don't merge > yet". If you call out @gvanrossum, GitHub will make sure I get a > message about it. > > I think the substantial discussion about the PEP should remain here in > python-dev; comments about typos, grammar and other minor editorial > issues can go on GitHub. Hope this part of the process makes sense! > > On Thu, Jul 14, 2016 at 6:50 AM, Martin Teichmann > wrote: >> Hi Guido, Hi list, >> >> Thanks for the nice review! I applied followed up your ideas and put >> it into a github pull request: https://github.com/python/peps/pull/53 >> >> Soon we'll be working there, until then, some responses to your comments: >> >>> I wonder if this should be renamed to __set_name__ or something else >>> that clarifies we're passing it the name of the attribute? The method >>> name __set_owner__ made me assume this is about the owning object >>> (which is often a useful term in other discussions about objects), >>> whereas it is really about telling the descriptor the name of the >>> attribute for which it applies. >> >> The name for this has been discussed a bit already, __set_owner__ was >> Nick's idea, and indeed, the owner is also set. Technically, >> __set_owner_and_name__ would be correct, but actually I like your idea >> of __set_name__. >> >>> That (inheriting type from type, and object from object) is very >>> confusing. Why not just define new classes e.g. NewType and NewObject >>> here, since it's just pseudo code anyway? >> >> Actually, it's real code. If you drop those lines at the beginning of >> the tests for the implementation (as I have done here: >> https://github.com/tecki/cpython/blob/pep487b/Lib/test/test_subclassinit.py), >> the test runs on older Pythons. >> >> But I see that my idea to formulate things here in Python was a bad >> idea, I will put the explanation first and turn the code into >> pseudo-code. >> >>>> def __init__(self, name, bases, ns, **kwargs): >>>> super().__init__(name, bases, ns) >>> >>> What does this definition of __init__ add? >> >> It removes the keyword arguments. I describe that in prose a bit down. >> >>>> class object: >>>> @classmethod >>>> def __init_subclass__(cls): >>>> pass >>>> >>>> class object(object, metaclass=type): >>>> pass >>> >>> Eek! Too many things named object. >> >> Well, I had to do that to make the tests run... I'll take that out. >> >>>> In the new code, it is not ``__init__`` that complains about keyword >>>> arguments, >>>> but ``__init_subclass__``, whose default implementation takes no >>>> arguments. In >>>> a classical inheritance scheme using the method resolution order, each >>>> ``__init_subclass__`` may take out it's keyword arguments until none are >>>> left, >>>> which is checked by the default implementation of ``__init_subclass__``. >>> >>> I called this out previously, and I am still a bit uncomfortable with >>> the backwards incompatibility here. But I believe what you describe >>> here is the compromise proposed by Nick, and if that's the case I have >>> peace with it. >> >> No, this is not Nick's compromise, this is my original. Nick just sent >> another mail to this list where he goes a bit more into the details, >> I'll respond to that about this topic. >> >> Greetings >> >> Martin >> >> P.S.: I just realized that my changes to the PEP were accepted by >> someone else than Guido. I am a bit surprised about that, but I guess >> this is how it works? >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __qualname__ exposed as a local variable: standard?
Hi, so I did quite some research on this topic. And what I found out is that __qualname__ needs to exist in the namespace. Not necessarily because it should be used, but because it may be modified. The story goes as follows: the compiler sets the __qualname__ at the beginning of the class body. Within the class body, it may be modified as needed. Then type.__new__ takes it and uses it. Now one could think that instead of setting the __qualname__ at the beginning of the class body, we could do so at the end as to not clutter the namespace, and only if the __qualname__ has been set in the class body we would use the user-supplied version. But this is forgetting __prepare__: unfortunately, we have no good way to find out whether something has been set in a class body, because we have no guarantee that the object returned by __prepare__ doesn't do something weird, as autogenerating values for all requested keys. > To Martin: it would be easier for people (even myself, who implemented > this super() hack eons ago) to review your patch if you were able to > explain the current and proposed behavior more precisely. I tried to give some context on my issue (http://bugs.python.org/issue23722). Hope that helps. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
> This PEP is now accepted for inclusion in Python 3.6. Martin, > congratulations! Thank you very much! What a great news! Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hi list, Hi Nick, Sorry for my delayed response, it is summer here... > However, phrasing it that way suggest that it's possible we *did* miss > something in the PEP: we haven't specified whether or not __set_name__ > should be called when someone does someone does "cls.attr = descr". > Given the name, I think we *should* call it in that case, and then the > semantics during class creation are approximately what would happen if > we actually built up the class attributes as: > > for attr, value in cls_ns.items(): > setattr(cls, attr, value) That's a very good point and actually easy to solve: we would just need to override type.__setattr__ to do so. Actually, it is already overridden, so we just need to add code to type.__setattr__ to also call __set_name__. One could be of the other standpoint that in your above example it's the duty of the caller of setattr to also call __set_name__. It would be pretty easy to add a line in the loop that also calls __set_owner__. Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hello, there has been quite some discussion on why PEP 487's __init_subclass__ initializes subclasses, and not the class itself. I think the most important details have been already thoroughly discussed here. One thing I was missing in the discussion is practical examples. I have been using PEP 487-like metaclasses since several years now, and I have never come across an application where it would have even been nice to initialize the class itself. Also, while researching other people's code when I wrote PEP 487, I couldn't find any such code elsewhere, yet I found a lot of code where people took the wildest measure to prevent a metaclass in doing its job on the first class it is used for. One example is enum.EnumMeta, which contains code not to make enum.Enum an enum (I do not try to propose that the enum module should use PEP 487, it's way to complicated for that). So once we have a practical example, we could start discussing how to mitigate the problem. Btw, everyone is still invited to review the patch for PEP 487 at http://bugs.python.org/issue27366. Many thanks to Nick who already reviewed, and also to Guido who left helpful comments! Greetings Martin ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP487: Simpler customization of class creation
Hi Sylvain,
thanks for the example, it's a great example to illustrate PEP 487 and
its design decisions.
> What it does is setting some class attributes that are required for certain
> types of descriptors to be able to initialize themselves.
>
> class MetaHasTraits(MetaHasDescriptors):
>
> """A metaclass for HasTraits."""
> def setup_class(cls, classdict):
> cls._trait_default_generators = {}
> super(MetaHasTraits, cls).setup_class(classdict)
While in a metaclass solution this does need that the metaclass needs
to execute code on the first class it is used for, in a PEP 487
solution this is not the case. A PEP 487 class HasTraits (no Meta
here) would not have Traits-descriptors itself, the classes inheriting
from it would have traits to be initialized. The PEP 487 HasTraits
takes the role of the metaclass.
A completely different problem shows up here. In your example,
HasTraits needs to initialize things on the class before the
Descriptors are run. This is not possible with PEP 487, where the
descriptors are initialized before __init_subclass__ is even called.
There are two ways to mitigate that problem:
- the first initialized descriptor could do the necessary initialization, or
- the descriptors are initialized from within __init_subclass__
At first sight, the first option seems hackish and the second option
looks much saner. Nonetheless PEP 487 proposes the second solution.
The rationale behind that is that people tend to forget to call
super(), and suddenly descriptors don't work anymore.
I realized later there is another benefit to this: if the first
initialized descriptor is doing the class initialization, often
__init_subclass__ isn't needed at all anymore, which means that those
kind of descriptors can be used on any class, without the need to tell
users that they have to inherit from a certain base class for the
descriptors to work. Only if some finalizing code needs to run after
the last descriptor is initialized one needs to write an
__init_subclass__. This is unavoidable as the last descriptor doesn't
know that it is the last.
Greetings
Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
