[Python-Dev] Re: Enhancement request for PyUnicode proxies

[Inada Naoki]

On Mon, Dec 28, 2020 at 7:22 PM Phil Thompson
 wrote:
>
> > So I'm +1 to make Unicode simple by removing PyUnicode_READY(), and -1
> > to make Unicode complicated by adding customizable callback for lazy
> > population.
> >
> > Anyway, I am OK to un-deprecate PyUnicode_READY() and make it no-op
> > macro since Python 3.12.
> > But I don't know how many third-parties use it properly, because
> > legacy Unicode objects are very rare already.
>
> For me lazy population might not be enough (as I'm not sure precisely
> what you mean by it). I would like to be able to use my foreign unicode
> thing to be used as the storage.
>
> For example (where text() returns a unicode object with a foreign
> kind)...
>
> some_text = an_editor.text()
> more_text = another_editor.text()
>
> if some_text == more_text:
>  print("The text is the same")
>
> ...would not involve any conversions at all.

So you mean custom internal representation of exact Unicode object?

Then, I am more strong -1, sorry.
I can not believe the merits of it is bigger than the costs of its complexity.
If 3rd party wants to use completely different internal
representation, it must not be a unicode object at all.

Regards,
-- 
Inada Naoki  
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7SBDIVIPLG4JS7MTJOGDEQXME6TFK5FU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Guido van Rossum]

On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman  wrote:

> On 12/28/20 9:31 PM, Guido van Rossum wrote:
>
> > Let me see if I can unpack this.
> >
> > I observe that `type.__new__() ` is really the C function `type_new()`
> in typeobject.c, and hence I will refer to it by
> > that name.
> >
> > I understand that `type_new()` is the only way to create type objects,
> and it includes a call to `__init_subclass__()`.
> >
> > In the source code of `type_new()`, calling `__init_subclass__()` is the
> last thing it does before returning the newly
> > created class object, so at this point the class object is complete,
> *except* that any updates made by the caller of
> > `type_new()` after `type_new()` returns have not been made, of course.
> (For example, `new_class.some_attr = 9` from
> > Ethan's post, or `__abstractmethods__`, which is set by
> update_abstractmethods() in abc.py.)
>
> This is really the heart of the issue.  A major reason to write a custom
> metaclass is to be able to modify the returned
> class before giving it back to the user -- so even though `type_new` is
> complete the new class could easily not be
> complete, and calling `__init_subclass__` and `__set_name__` from
> `type_new` is premature.
>
> > Now here's something that Ethan said that I don't follow:
> >
> >> For Enum, this means that `__init_subclass__` doesn't have access to
> the new Enum's members (they haven't been added yet)
> >
> > I would presume that in an example like the following, the members *are*
> set by the time `type_new()` is called. What am
> > I missing?
> > ```
> > class Color(enum.Enum):
> >  RED = 1
> >  GREEN = 2
> >  BLUE = 4
> > ```
> > Maybe the problem is that the members are still set to their "naive"
> initial values (1, 2, 4) rather than to the
> > corresponding enum values (e.g. ``)?
>
> Before `type_new()` is called all the (future) members are removed from
> `namespace`, so they are not present when
> `__init_subclass__` is called.  Even if they were left in as `{'RED': 1,
> `GREEN`: 2, `BLUE`: 4}` it would not be
> possible for an `__init_subclass__` to customize the members further, or
> record them in custom data structures, or run
> validation code on them, or etc.
>
> > Without a more elaborate use case I can't give that more than a shrug.
> This is how the `__init_subclass__()` protocol is
> > designed.
>
> The `__init_subclass__` and `__set_name__` protocols are intended to be
> run before a new type is finished, but creating
> a new type has three major steps:
>
> - `__prepare__` to get the namespace
> - `__new__` to get the memory and data structures
> - `__init__` for any final polishing
>
> We can easily move the calls from `type_new()` to `type_init`.
>

No, we can't. There is a window where the subclass is initialized after
`typing_new()` returned before `__init__` starts, and you propose to move
the subclass after that window. There may be code that depends on the class
being initialized at that point, and you will break that code.

Honestly I disapprove of the shenanigans you're committing in enum.py, and
if those are in in the 3.10 (master) branch I recommend that you take them
out. It just looks too fragile and obscure.


> > Note that for ABC, if you add or change the abstraction status of some
> class attributes, you can just call
> > `update_abstractmethods()` and it will update `__abstractmethods__`
> based on the new contents of the class. This also
> > sounds like no biggie to me.
>
> The issue tracker sample code that fails (from #35815):
>
> ```
> import abc
>
> class Base(abc.ABC):
>  #
>  def __init_subclass__(cls, **kwargs):
>  instance = cls()
>  print(f"Created instance of {cls} easily: {instance}")
>  #
>  @abc.abstractmethod
>  def do_something(self):
>  pass
>
> class Derived(Base):
>  pass
> ```
>
> And the output:
>
> `Created instance of  easily: <__main__.Derived
> object at 0x10a6dd6a0>`
>
> If `Base` had been completed before the call to `__init_subclass__`, then
> `Derived` would have raised an error -- and it
> does raise an error with the patch I have submitted on Github.
>

Again, big shrug. A static checker will catch that.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HNLAXFRXAW3TOR6CZMXBGRLWQPTOKIN4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Enhancement request for PyUnicode proxies

[Antoine Pitrou]

On Mon, 28 Dec 2020 14:27:00 +0100
Ronald Oussoren via Python-Dev  wrote:
> > On 28 Dec 2020, at 14:00, Inada Naoki  wrote:
> > 
> > On Mon, Dec 28, 2020 at 8:52 PM Phil Thompson
> >  wrote:  
> >> 
> >> 
> >> I would have thought that an object was defined by its behaviour rather
> >> than by any particular implementation detail.
> >>   
> > 
> > As my understanding, the policy "an object was defined by its
> > behavior..." doesn't mean "put unlimited amount of implementation
> > behind one concrete type."
> > The policy means APIs shouldn't limit input to one concrete type
> > without a reason. In other words, duck typing and structural subtyping
> > are good.
> > 
> > For example, we can try making io.TextIOWrapper accepts not only
> > Unicode objects (including subclass) but any objects implementing some
> > protocol.
> > We already have __index__ for integers and buffer protocol for
> > byts-like objects. That is examples of the policy.  
> 
> I agree that that would be the cleanest approach, although I worry about
> how long it will take until 3th-party code is converted to the new protocol. 
> That’s
> why I wrote earlier that adding this feature to PyUnicode_Type is the most
> pragmantic solution ;-)

But the "pragmatic" solution will make a performance-critical type
(PyUnicode) more complicated and therefore potentially larger/slower.
I think Inada's concerns are valid here.

> There are two clear options for a new protocol:
> 
> 1. Add something similar to __index__ of __fspath__, but for “string-like” 
> objects
> 
> 2. Add an extension to the buffer protocol

The third option is to add a distinct "string view" protocol.  There
are peculiarities (such as the fact that different objects may have
different internal representations - some utf8, some utf16...) that
make the buffer protocol suboptimal for this.

Also, we probably don't want unicode-like objects to start being usable
in contexts where a buffer-like object is required (such as writing to
a binary file, or zlib-compressing a bunch of bytes).

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/62UYIIM5ZK6DZ2SNDROUQSH4HFOW5DS7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Ethan Furman]

On 12/29/20 8:59 AM, Guido van Rossum wrote:

On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:



The `__init_subclass__` and `__set_name__` protocols are intended to be run 
before a new type is finished, but creating
a new type has three major steps:

- `__prepare__` to get the namespace
- `__new__` to get the memory and data structures
- `__init__` for any final polishing

We can easily move the calls from `type_new()` to `type_init`.


No, we can't. There is a window where the subclass is initialized after `typing_new()` returned before `__init__` 
starts, and you propose to move the subclass after that window. There may be code that depends on the class being 
initialized at that point, and you will break that code.


True, there will be a few custom metaclasses that need to move some code from their `__new__` to `__init__` instead, and 
a few that need to add an `__init__` to consume any keyword arguments that don't need to get passed to 
`__init_subclass__`.  That seems like a small price to pay to be able to write custom metaclasses that are able to fully 
participate in the `__set_name__` and `__init_subclass__` protocols.


Just in the stdlib we have two custom metaclasses that would start working correctly with this change (for the 
`__init_subclass__` case).


Honestly I disapprove of the shenanigans you're committing in enum.py, and if those are in in the 3.10 (master) branch I 
recommend that you take them out. It just looks too fragile and obscure.


Trust me, I don't like them either.  But I like even less that a custom `__init_subclass__` for an Enum would fail if it 
tried to do anything with the members.  That's one of the reasons why I would like to see this fix put in (the 
shenanigans would be unnecessary then).


From the point of view of a metaclass author, the current behavior feels buggy.  In theory, `__init_subclass__` and 
`__set_name__` are supposed to be called after a class is created, and yet they are being called somewhere in the middle 
of my metaclass' `__new__`.


Looked at another way, `__init_subclass__` should be receiving a `cls` that is ready for use, i.e. done and fully 
constructed and complete. But if that class is being created by a custom metaclass, then the thing that is being given 
to `__init_subclass__` could easily be only partial.


--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Guido van Rossum]

I need to think about this more.

Technically the class *is* created at the point `__init_subclass__` is
called. What the metaclass does after that point is embellishment.

Most metaclasses will have sufficient control for their needs because they
can manipulate the contents of the namespace that's passed to type_new() --
I understand that doesn't work for Enum because the instances have a
reference to the class in them.

I suspect the classes that you claim are "buggy" are fine in practice, even
if you can construct a counterexample.

All in all I just worry about the backward compatibility here: one could
easily imagine a metaclass whose `__new__` relies on the effect of the
`__init_subclass__` called by type_new(), and moving the call would break
that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such
a case, the status quo should win.

--Guido

On Tue, Dec 29, 2020 at 10:44 AM Ethan Furman  wrote:

> On 12/29/20 8:59 AM, Guido van Rossum wrote:
> > On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:
>
> >> The `__init_subclass__` and `__set_name__` protocols are intended to be
> run before a new type is finished, but creating
> >> a new type has three major steps:
> >>
> >> - `__prepare__` to get the namespace
> >> - `__new__` to get the memory and data structures
> >> - `__init__` for any final polishing
> >>
> >> We can easily move the calls from `type_new()` to `type_init`.
> >
> > No, we can't. There is a window where the subclass is initialized after
> `typing_new()` returned before `__init__`
> > starts, and you propose to move the subclass after that window. There
> may be code that depends on the class being
> > initialized at that point, and you will break that code.
>
> True, there will be a few custom metaclasses that need to move some code
> from their `__new__` to `__init__` instead, and
> a few that need to add an `__init__` to consume any keyword arguments that
> don't need to get passed to
> `__init_subclass__`.  That seems like a small price to pay to be able to
> write custom metaclasses that are able to fully
> participate in the `__set_name__` and `__init_subclass__` protocols.
>
> Just in the stdlib we have two custom metaclasses that would start working
> correctly with this change (for the
> `__init_subclass__` case).
>
> > Honestly I disapprove of the shenanigans you're committing in enum.py,
> and if those are in in the 3.10 (master) branch I
> > recommend that you take them out. It just looks too fragile and obscure.
>
> Trust me, I don't like them either.  But I like even less that a custom
> `__init_subclass__` for an Enum would fail if it
> tried to do anything with the members.  That's one of the reasons why I
> would like to see this fix put in (the
> shenanigans would be unnecessary then).
>
>  From the point of view of a metaclass author, the current behavior feels
> buggy.  In theory, `__init_subclass__` and
> `__set_name__` are supposed to be called after a class is created, and yet
> they are being called somewhere in the middle
> of my metaclass' `__new__`.
>
> Looked at another way, `__init_subclass__` should be receiving a `cls`
> that is ready for use, i.e. done and fully
> constructed and complete. But if that class is being created by a custom
> metaclass, then the thing that is being given
> to `__init_subclass__` could easily be only partial.
>
> --
> ~Ethan~
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AD2ORFW6VCYQNIKCW2DZ3F5KVADDYYZC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: __init_subclass__ and metaclasses

[Glenn Linderman]

On 12/29/2020 10:00 PM, Guido van Rossum wrote:

I need to think about this more.


Both techniques could coexist.

Technically the class *is* created at the point `__init_subclass__` is 
called. What the metaclass does after that point is embellishment.


Most metaclasses will have sufficient control for their needs because 
they can manipulate the contents of the namespace that's passed to 
type_new() -- I understand that doesn't work for Enum because the 
instances have a reference to the class in them.


I suspect the classes that you claim are "buggy" are fine in practice, 
even if you can construct a counterexample.


All in all I just worry about the backward compatibility here: one 
could easily imagine a metaclass whose `__new__` relies on the effect 
of the `__init_subclass__` called by type_new(), and moving the call 
would break that. So it's probably 6 bugs fixed, half a dozen new bugs 
created. In such a case, the status quo should win.


--Guido

On Tue, Dec 29, 2020 at 10:44 AM Ethan Furman > wrote:


On 12/29/20 8:59 AM, Guido van Rossum wrote:
> On Mon, Dec 28, 2020 at 10:24 PM Ethan Furman wrote:

>> The `__init_subclass__` and `__set_name__` protocols are
intended to be run before a new type is finished, but creating
>> a new type has three major steps:
>>
>> - `__prepare__` to get the namespace
>> - `__new__` to get the memory and data structures
>> - `__init__` for any final polishing
>>
>> We can easily move the calls from `type_new()` to `type_init`.
>
> No, we can't. There is a window where the subclass is
initialized after `typing_new()` returned before `__init__`
> starts, and you propose to move the subclass after that window.
There may be code that depends on the class being
> initialized at that point, and you will break that code.

True, there will be a few custom metaclasses that need to move
some code from their `__new__` to `__init__` instead, and
a few that need to add an `__init__` to consume any keyword
arguments that don't need to get passed to
`__init_subclass__`.  That seems like a small price to pay to be
able to write custom metaclasses that are able to fully
participate in the `__set_name__` and `__init_subclass__` protocols.

Just in the stdlib we have two custom metaclasses that would start
working correctly with this change (for the
`__init_subclass__` case).

> Honestly I disapprove of the shenanigans you're committing in
enum.py, and if those are in in the 3.10 (master) branch I
> recommend that you take them out. It just looks too fragile and
obscure.

Trust me, I don't like them either.  But I like even less that a
custom `__init_subclass__` for an Enum would fail if it
tried to do anything with the members.  That's one of the reasons
why I would like to see this fix put in (the
shenanigans would be unnecessary then).

 From the point of view of a metaclass author, the current
behavior feels buggy.  In theory, `__init_subclass__` and
`__set_name__` are supposed to be called after a class is created,
and yet they are being called somewhere in the middle
of my metaclass' `__new__`.

Looked at another way, `__init_subclass__` should be receiving a
`cls` that is ready for use, i.e. done and fully
constructed and complete. But if that class is being created by a
custom metaclass, then the thing that is being given
to `__init_subclass__` could easily be only partial.

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/7WPNGL267FDGN6WWCHZQUAXWVBTUJOMN/


Code of Conduct: http://python.org/psf/codeofconduct/




--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AD2ORFW6VCYQNIKCW2DZ3F5KVADDYYZC/
Code of Conduct: http://python.org/psf/codeofconduct/


___
Python-Dev m

[Python-Dev] Re: __init_subclass__ and metaclasses

[Jonathan Goble]

On Wed, Dec 30, 2020, 1:05 AM Guido van Rossum  wrote:

> I need to think about this more.
>
> Technically the class *is* created at the point `__init_subclass__` is
> called. What the metaclass does after that point is embellishment.
>
> Most metaclasses will have sufficient control for their needs because they
> can manipulate the contents of the namespace that's passed to type_new() --
> I understand that doesn't work for Enum because the instances have a
> reference to the class in them.
>
> I suspect the classes that you claim are "buggy" are fine in practice,
> even if you can construct a counterexample.
>
> All in all I just worry about the backward compatibility here: one could
> easily imagine a metaclass whose `__new__` relies on the effect of the
> `__init_subclass__` called by type_new(), and moving the call would break
> that. So it's probably 6 bugs fixed, half a dozen new bugs created. In such
> a case, the status quo should win.
>

I admit I don't know anything about the metaclass internals, but if
backward compatibility is an issue, couldn't we create a new dunder instead
of moving the existing one? Say, call it __init_subclass_2__ (and bikeshed
the name later), and document that as being called at the point to which
it's currently proposed to move the calling of __init_subclass__.

With that, there is zero breakage of existing code, and code that can
benefit from the new behavior simply uses a different dunder.
__init_subclass__ could potentially be deprecated in the future if it's
determined that it has no benefit over the new dunder, but doesn't need to
deprecated immediately.

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VWNYDERWSPE7MIN2TVRSFM6AM7RLF6AO/
Code of Conduct: http://python.org/psf/codeofconduct/