[Python-Dev] Migrate typing in builtin

[Philippe Prados]

Hello,

I try to implement a *proof of concept* of Pep-0604
.

To do this, the _GenericAlias must be in builtin, because I would like to
update issubclass() and isinstance() to accept Union.
These methods are in builtin, and must be run without others modules.

But, the class _GenericAlias is in typing module, with the Python
implementation in Lib/typing.py, and the C implementation is in
Modules/_ctypes.

For you, what is the best approach to move the _GenericAlias and all
dependencies in the builtin "module" ?

Philippe
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FVDIL2DG3EP7CO452MUZYPATN4R2CEHN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migrate typing in builtin

[Philippe Prados]

Because this PEP propose to accept, for all classes

assert isinstance("", int | str)
assert issubclass(int, int | str)

and add an operator __or__() for type type.

def f(list: List[int | str], param: int | None) -> float | str:
pass

Regards

Philippe

Le lun. 7 oct. 2019 à 17:29, Random832  a écrit :

> On Mon, Oct 7, 2019, at 10:32, Philippe Prados wrote:
> > Hello,
> >
> > I try to implement a *proof of concept* of Pep-0604
> > <https://www.python.org/dev/peps/pep-0604/>.
> >
> > To do this, the _GenericAlias must be in builtin, because I would like
> > to update issubclass() and isinstance() to accept Union.
> > These methods are in builtin, and must be run without others modules.
>
> Why can't this be handled as an __instancecheck__/__subclasscheck__?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/UZRHLKTGR7USOD365N57Y2QVUWWMGPJ7/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LASLJP2MRQWSWUHD4PCJJ2D3SOWL6SAD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migrate typing in builtin

[Philippe Prados]

Ok,

But _GenericAlias and dependencies are written with Python (Lib/typing.py),
not with C.
So, I must rewrite the _GenericAlias in C or it's possible to merge the C
and Python in builtin and add a direct reference to _GenericAlias with C,
and add the reference in builtin module ?

Philippe


Le lun. 7 oct. 2019 à 22:58, Random832  a écrit :

> On Mon, Oct 7, 2019, at 12:02, Philippe Prados wrote:
> > Because this PEP propose to accept, for all classes
> > assert isinstance("", int | str)
> > assert issubclass(int, int | str)
> > and add an operator __or__() for type type.
> > def f(list: List[int | str], param: int | None) -> float | str:
> > pass
>
> Oh, sorry, I didn't realize that this also included the | operator, I
> thought this was just for isinstance("", Union[int, str]).
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GD7WXPD26VUPMZT6WAATCJJBB42DDYYQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migrate typing in builtin

[Philippe Prados]

Glups.

I am not an expert of Pyhton source code. May be after this patch ;-)

I think I should discuss with the authors of the module typing, to identify
the best strategy.
Who is he ?

Philippe


Le mar. 8 oct. 2019 à 10:19, Ivan Levkivskyi  a
écrit :

> You will need to rewrite most of things in C.
>
> --
> Ivan
>
>
> On Tue 8 Oct 2019, 08:53 Philippe Prados, 
> wrote:
>
>> Ok,
>>
>> But _GenericAlias and dependencies are written with Python
>> (Lib/typing.py), not with C.
>> So, I must rewrite the _GenericAlias in C or it's possible to merge the C
>> and Python in builtin and add a direct reference to _GenericAlias with C,
>> and add the reference in builtin module ?
>>
>> Philippe
>>
>>
>> Le lun. 7 oct. 2019 à 22:58, Random832  a écrit :
>>
>>> On Mon, Oct 7, 2019, at 12:02, Philippe Prados wrote:
>>> > Because this PEP propose to accept, for all classes
>>> > assert isinstance("", int | str)
>>> > assert issubclass(int, int | str)
>>> > and add an operator __or__() for type type.
>>> > def f(list: List[int | str], param: int | None) -> float | str:
>>> > pass
>>>
>>> Oh, sorry, I didn't realize that this also included the | operator, I
>>> thought this was just for isinstance("", Union[int, str]).
>>>
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/GD7WXPD26VUPMZT6WAATCJJBB42DDYYQ/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4K7WZ3RBGG7K6E6XK65MS44VQYZIKQS2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Migrate typing in builtin

[Philippe Prados]

In my current implementation (proof of concept), I must use a trick to
detect the _GenericAlias, without import. I check the tp_name.

static PyObject*
union_to_tuple(PyObject* cls) {
//printf("union_to_tuple");
if (!strcmp(Py_TYPE(cls)->tp_name,"_GenericAlias")) {
PyObject* origin = PyObject_GetAttrString(cls, "__origin__");
//printf("origin = %p\n",origin);
if (origin == NULL) {
printf("origin == NULL, return cls");
return cls;
}
//printf("X origin = %s\n",Py_TYPE(cls)->tp_name);
if (PyObject_HasAttrString(origin, "_name")) {
PyObject* name = PyObject_GetAttrString(origin, "_name");
if (name==NULL) {
printf("_name = NULL\n");
Py_DECREF(origin);
return cls;
}
//printf("name = %s\n",Py_TYPE(name)->tp_name);
const char* data = (char*)PyUnicode_1BYTE_DATA(name);
//printf("DATA=%s\n",data);
if (data != NULL && !strcmp(data,"Union")) {
PyObject* new_cls = PyObject_GetAttrString(cls, "__args__");
if (new_cls != NULL) {
cls = new_cls;
}
}
Py_DECREF(name);
}
Py_DECREF(origin);
}
return cls;
}

I can't use PyImport_ImportModule("typing") because of infinite recursivity.

One version of type_or (__or__) use the import.

static PyObject *
type_or(PyTypeObject* self, PyObject* param) {
PyObject* typing=PyImport_ImportModule("typing");
PyTypeObject* genericAlias =
(PyTypeObject*)PyObject_GetAttrString(typing,"_GenericAlias");

// Check param is a PyType or GenericAlias
if ((param == NULL) ||
(
 (param != Py_None) &&
 ! PyType_IsSubtype(genericAlias, Py_TYPE(param)) &&
 (PyObject_IsInstance(param, (PyObject *) &PyType_Type) != 1)
)
) {
PyErr_SetString(PyExc_TypeError, "'type' expected");
Py_DECREF(genericAlias);
Py_DECREF(typing);
return NULL;
}

// 1. Create a tuple with types
PyObject *tuple=PyTuple_Pack(2,self, param);
// 2. Create Union with tuple
PyObject* unionType = PyObject_GetAttrString(typing,"Union");
PyObject *newUnion=PyObject_GetItem(unionType, tuple);
// 3. Clean memory
Py_DECREF(genericAlias);
Py_DECREF(typing);
Py_DECREF(unionType);
Py_DECREF(tuple);
// 4. Return instance
return newUnion;

It's may be acceptable, but I'm not sure of that.

Philippe


Le mar. 8 oct. 2019 à 18:25, Guido van Rossum  a écrit :

> That would mean that all of typing will be imported as part of startup,
> and that module is too heavy for that. Also it might end up on recursion.
>
> On Tue, Oct 8, 2019 at 09:06 Brandt Bucher  wrote:
>
>> I’m a bit confused. For my own understanding: what’s stopping
>> PyObject_IsInstance/PyObject_IsSubclass from just trying
>> PyImport_GetModule("typing") here?
>>
>> If NULL, carry on. Otherwise, check the Union case.
>>
>> Brandt
>>
>> On Oct 8, 2019, at 05:44, Philippe Prados 
>> wrote:
>>
>> 
>>
>> Glups.
>>
>> I am not an expert of Pyhton source code. May be after this patch ;-)
>>
>> I think I should discuss with the authors of the module typing, to
>> identify the best strategy.
>> Who is he ?
>>
>> Philippe
>>
>>
>> Le mar. 8 oct. 2019 à 10:19, Ivan Levkivskyi  a
>> écrit :
>>
>>> You will need to rewrite most of things in C.
>>>
>>> --
>>> Ivan
>>>
>>>
>>> On Tue 8 Oct 2019, 08:53 Philippe Prados, 
>>> wrote:
>>>
>>>> Ok,
>>>>
>>>> But _GenericAlias and dependencies are written with Python
>>>> (Lib/typing.py), not with C.
>>>> So, I must rewrite the _GenericAlias in C or it's possible to merge the
>>>> C and Python in builtin and add a direct reference to _GenericAlias with C,
>>>> and add the reference in builtin module ?
>>>>
>>>> Philippe
>>>>
>>>>
>>>> Le lun. 7 oct. 2019 à 22:58, Random832  a
>>>> écrit :
>>>>
>>>>> On Mon, Oct 7, 2019, at 12:02, Philippe Prados wrote:
>>>>> > Because this PEP propose to accept, for all classes
>>>>> > assert isinstance("", int | str)
>>>>> > assert issubclass(int, int | str)
>>>>> > and add an operator

[Python-Dev] Implementation of PEP-0604

[Philippe Prados]

To implement a full version of PEP604
, I analyze the typing module,
started with _GenericAlias.
1) I must rewrite :

   - def _type_check(arg, msg, is_argument=True)
   - def _type_repr(obj)
   - def _collect_type_vars(types)
   - def _subs_tvars(tp, tvars, subs)
   - def _check_generic(cls, parameters)
   - def _remove_dups_flatten(parameters)
   - def _tp_cache(func)
   - class _Final
   - class _Immutable
   - class _SpecialForm(_Final, _Immutable, _root=True)
   - class ForwardRef(_Final, _root=True)
   - class TypeVar(_Final, _Immutable, _root=True)
   - def _is_dunder(attr)
   - class _GenericAlias(_Final, _root=True)
   - class Generic
   - class _TypingEmpty
   - class _TypingEllipsis
   - def _get_protocol_attrs(cls)
   - def _is_callable_members_only(cls)
   - def _allow_reckless_class_cheks()
   - class _ProtocolMeta(ABCMeta)
   - class Protocol(Generic, metaclass=_ProtocolMeta)

2) The function _tp_cache use functools.lru_cache()
def _tp_cache(func):
cached = functools.lru_cache()(func)
it's not reasonable to move the lru_cache() in the core

3) The method TypeVar.__init__() use:
def_mod = sys._getframe(1).f_globals['__name__']  # for pickling

4) The method def _allow_reckless_class_cheks() use:
return sys._getframe(3).f_globals['__name__'] in ['abc', 'functools']

5) The method Protocol.__init_subclass___proto_hook() use:
if (isinstance(annotations, collections.abc.Mapping)
it's not reasonable to move the Mapping type in the core

It's not enough to move the typing classes, I must move
functools.lru_cache() and dependencies, collections.abs.Mapping and
dependencies, and track the frame level.

*It's too big for me.*

May be, the approach with only PEP 563 is enough.
from __future__ import annotations
a:int|str=3

This new syntax is only usable in annotations. Without runtime evaluation
and without modifying issubclass() and isinstance() may be acceptable. Only
the mypy (and others tools like this) must be updated.


Philippe
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NX6IIO4XZ56XNBWDEQBDF4PZ23NW32HZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Implementation of PEP-0604

[Philippe Prados]

Cool !

I'm very happy you take time to do this job.

Your idea to use a shadowobject objet to promote just-in-time is very good
for me, and can be used for others extensions.

Thanks

Philippe Prados


Le sam. 9 nov. 2019 à 00:12, Richard Eames  a écrit :

> Hi Philippe and list,
>
> Long time lurker here, I've had a few spare cycles at $work this week that
> I spent exploring an alternate implementation to PEP604 starting with the
> implementation from Philippe. I don't say "reference implementation",
> because it doesn't implement PEP604 faithfully, but it does try to
> implement it's goal.  My thinking was that the "runtime" component to it
> only needs to support the isinstance/issubclass methods and so can be
> pretty minimal without need to import all the typing.py machinery. At which
> point, when actual type checking was done (anything in typing.py or mypy),
> the `union` type can be special cased, or promoted to a `typing.Union`.
>
> I started by creating a built-in "union" type, that was a thin wrapper
> around a tuple, such that type.__or__ will return `union(lhs, rhs)` which
> stored the two items in a 2-length tuple. I also changed the methods in
> `typing.Union` to return a `union`. However, I hit my first issue: `int |
> int | str` would then return `union(int, union(int, str))`, whereas it
> should deduplicate the types. So, I implemented logic to detect and merge
> union | union, but changed the underlying data to a plain flat tuple. And
> the next issue I hit was that some of the other `typing.py` types weren't
> being detected as valid types (probably due to my limited understanding of
> the PyObject datastructure/flags), specifically, `int | "forwardref"`, I
> needed a way to signal to typing.py that I had a forwardref without
> importing it. This is where I changed my approach to the problem.
>
> I wanted to find a way to have these metatypes (Union/TypeVar/ForwardRef)
> without re-implementing them in c, or importing them. My Idea was to change
> the builtin type to what I called a "shadow type". In the C layer it's just
> holds a tuple of objects/parameters, and has a type field to say what it
> is, then when in the type checking/typing.py layer, if it detects an object
> is a "shadowobject", then it promotes it to a full type. What I liked about
> this approach, is that for isinstance/issubclass, when it see a
> "shadowobject" it can just grab the underlying types from the
> shadowobject->params tuple and no import magic needs to happen. All of the
> type magic can be left to mypy/typing.py without much knowledge of the
> shadowobject. I've mostly got through this implementation with all but one
> test passing.
>
> At this point, I've got as far as I can go without help, so I'm posting
> here, looking for feedback or discussion. I expect that maybe the next step
> should be me commenting on the PEP604 itself for implementation concerns
> regarding `_GenericAlias`?
>
> Implementation is located at
> https://github.com/Naddiseo/cpython/tree/PEP604
>
> Richard
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/P56ZE6USNIGCQU5PIOLKC65VHD5DM37U/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FLRUBE4VNWCFUN6HOBXB7TZRRXXDWCZN/
Code of Conduct: http://python.org/psf/codeofconduct/