[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: I know this is quite an old bug that was closed almost 10 years ago. But I am wishing this had been accepted; it would have been quite useful for my case. I'm working on a new iteration of the protobuf extension for Python. At runtime we create types dynamically, one for each message defined in a .proto file, eg. from "message Foo" we dynamically construct a "class Foo". I need to support class variables like Foo.BAR_FIELD_NUMBER, but I don't want to put all these class variables into tp_dict because there are a lot of them and they are rarely used. So I want to implement __getattr__ for the class, which requires having a metaclass. This is where the proposed PyType_FromSpecEx() would have come in very handy. The existing protobuf extension gets around this by directly calling PyType_Type.tp_new() to create a type with a given metaclass: https://github.com/protocolbuffers/protobuf/blob/53365065d9b8549a5c7b7ef1e7e0fd22926dbd07/python/google/protobuf/pyext/message.cc#L278-L279 It's unclear to me if PyType_Type.tp_new() is intended to be a supported/public API. But in any case, it's not available in the limited API, and I am trying to restrict myself to the limited API. (I also can't use PyType_GetSlot(PyType_Type, Py_tp_new) because PyType_Type is not a heap type.) Put more succinctly, I do not see any way to use a metaclass from the limited C API. Possible solutions I see: 1. Add PyType_FromSpecEx() (or similar with a better name) to allow a metaclass to be specified. But I want to support back to at least Python 3.6, so even if this were merged today it wouldn't be viable for a while. 2. Use eval from C to create the class with a metaclass, eg. class Foo(metaclass=MessageMeta) 3. Manually set FooType->ob_type = &MetaType, as recommended here: https://stackoverflow.com/a/52957978/77070 . Since PyObject.ob_type is part of the limited API, I think this might be possible! -- nosy: +jhaberman ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > You can also call (PyObject_Call*) the metaclass with (name, bases, > namespace); But won't that just call my metaclass's tp_new? I'm trying to do this from my metaclass's tp_new, so I can customize the class creation process. Then Python code can use my metaclass to construct classes normally. > I wouldn't recommend [setting ob_type] after PyType_Ready is called. Why not? What bad things will happen? It seems to be working so far. Setting ob_type directly actually solves another problem that I had been having with the limited API. I want to implement tp_getattro on the metaclass, but I want to first delegate to PyType.tp_getattro to return any entry that may be present in the type's tp_dict. With the full API I could call self->ob_type->tp_base->tp_getattro() do to the equivalent of super(), but with the limited API I can't access type->tp_getattro (and PyType_GetSlot() can't be used on non-heap types). I find that this does what I want: PyTypeObject *saved_type = self->ob_type; self->ob_type = &PyType_Type; PyObject *ret = PyObject_GetAttr(self, name); self->ob_type = saved_type; Previously I had tried: PyObject *super = PyObject_CallFunction((PyObject *)&PySuper_Type, "OO", self->ob_type, self); PyObject *ret = PyObject_GetAttr(super, name); Py_DECREF(super); But for some reason this didn't work. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment:
I found a way to use metaclasses with the limited API.
I found that I can access PyType_Type.tp_new by creating a heap type derived
from PyType_Type:
static PyType_Slot dummy_slots[] = {
{0, NULL}
};
static PyType_Spec dummy_spec = {
"module.DummyClass", 0, 0, Py_TPFLAGS_DEFAULT, dummy_slots,
};
PyObject *bases = Py_BuildValue("(O)", &PyType_Type);
PyObject *type = PyType_FromSpecWithBases(&dummy_spec, bases);
Py_DECREF(bases);
type_new = PyType_GetSlot((PyTypeObject*)type, Py_tp_new);
Py_DECREF(type);
#ifndef Py_LIMITED_API
assert(type_new == PyType_Type.tp_new);
#endif
// Creates a type using a metaclass.
PyObject *uses_metaclass = type_new(metaclass, args, NULL);
PyType_GetSlot() can't be used on PyType_Type directly, since it is not a heap
type. But a heap type derived from PyType_Type will inherit tp_new, and we can
call PyType_GetSlot() on that.
Once we have PyType_Type.tp_new, we can use it to create a new type using a
metaclass. This avoids any of the class-switching tricks I was trying before.
We can also get other slots of PyType_Type like tp_getattro to do the
equivalent of super().
The PyType_FromSpecEx() function proposed in this bug would still be a nicer
solution to my problem. Calling type_new() doesn't let you specify object size
or slots. To work around this, I derive from a type I created with
PyType_FromSpec(), relying on the fact that the size and slots will be
inherited. This works, but it introduces an extra class into the hierarchy
that ideally could be avoided.
But I do have a workaround that appears to work, and avoids the problems
associated with setting ob_type directly (like PyPy incompatibility).
--
nosy: +haberman2
___
Python tracker
<https://bugs.python.org/issue15870>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > Passing the metaclass as a slot seems like the right idea for this API, > though I recall there being some concern about the API (IIRC, mixing function > pointers and data pointers doesn't work on some platforms?) PyType_Slot is defined as a void* (not a function pointer): https://github.com/python/cpython/blob/8492b729ae97737d22544f2102559b2b8dd03a03/Include/object.h#L223-L226 So putting a PyTypeObject* into a slot would appear to be more kosher than function pointers. Overall, a slot seems like a great first approach. It doesn't require any new functions, which seems like a plus. If the any linking issues a la tp_base are seen, a new function could be added later. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > It's better to pass the metaclass as a function argument, as with bases. I'd > prefer adding a new function that using a slot. Bases are available both as a slot (Py_tp_bases) and as an argument (PyType_FromSpecWithBases). I don't see why this has to be an either/or proposition. Both can be useful. Either would satisfy my use case. I'm constructing N such classes, so the spec won't be statically initialized anyway and the initialization issues on Windows don't apply. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > I consider Py_tp_bases to be a mistake: it's an extra way of doing things > that doesn't add any extra functionality I think it does add one extra bit of functionality: Py_tp_bases allows the bases to be retrieved with PyType_GetSlot(). This isn't quite as applicable to the metaclass, since that can easily be retrieved with Py_TYPE(type). > but is sometimes not correct (and it might not be obvious when it's not > correct). Yes I guess that most all slots are ok to share across sub-interpreters. I can see the argument for aiming to keep slots sub-interpreter-agnostic. As a tangential point, I think that the DLL case on Windows may be a case where Windows is not compliant with the C standard: https://mail.python.org/archives/list/python-...@python.org/thread/2WUFTVQA7SLEDEDYSRJ75XFIR3EUTKKO/ Practically speaking this doesn't change anything (extensions that want to be compatible with Windows DLLs will still want to avoid this kind of initialization) but I think the docs may be incorrect on this point when they describe Windows as "strictly standard conforming in this particular behavior." -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > "static" anything in C is completely irrelevant to how symbols are looked up > and resolved between modules That is not true. On ELF/Mach-O the "static" storage-class specifier in C will prevent a symbol from being added to the dynamic symbol table, which will make it unavailable for use across modules. > I wasn't aware the C standard covered dynamic symbol resolution? Well the Python docs invoke the C standard to justify the behavior of DLL symbol resolution on Windows, using incorrect arguments about what the standard says: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_base Fixing those docs would be a good first step. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > On ELF/Mach-O... nvm, I just realized that you were speaking about Windows specifically here. I believe you that on Windows "static" makes no difference in this case. The second point stands: if you consider LoadLibrary()/dlopen() to be outside the bounds of what the C standard speaks to, then the docs shouldn't invoke the C standard to explain the behavior. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: This behavior is covered by the standard. The following C translation unit is valid according to C99: struct PyTypeObject; extern struct PyTypeObject Foo_Type; struct PyTypeObject *ptr = &Foo_Type; Specifically, &Foo_Type is an "address constant" per the standard because it is a pointer to an object of static storage duration (6.6p9). The Python docs contradict this with the following incorrect statement: > However, the unary ‘&’ operator applied to a non-static variable like > PyBaseObject_Type() is not required to produce an address constant. This statement is incorrect: 1. PyBaseObject_Type is an object of static storage duration. (Note, this is true even though it does not use the "static" keyword -- the "static" storage-class specifier and "static storage duration" are separate concepts). 2. It follows that &PyBaseObject_Type is required to produce an address constant. because it is a pointer to an object of static storage duration. MSVC rejects this standard-conforming TU when __declspec(dllimport) is added: https://godbolt.org/z/GYrfTqaGn I am pretty sure this is out of compliance with C99. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45306] Docs are incorrect re: constant initialization in the C99 standard
New submission from Josh Haberman : I believe the following excerpt from the docs is incorrect (https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_base): > Slot initialization is subject to the rules of initializing > globals. C99 requires the initializers to be “address > constants”. Function designators like PyType_GenericNew(), > with implicit conversion to a pointer, are valid C99 address > constants. > > However, the unary ‘&’ operator applied to a non-static > variable like PyBaseObject_Type() is not required to produce > an address constant. Compilers may support this (gcc does), > MSVC does not. Both compilers are strictly standard > conforming in this particular behavior. > > Consequently, tp_base should be set in the extension module’s init function. I explained why in https://mail.python.org/archives/list/python-...@python.org/thread/2WUFTVQA7SLEDEDYSRJ75XFIR3EUTKKO/ and on https://bugs.python.org/msg402738. The short version: &foo is an "address constant" according to the standard whenever "foo" has static storage duration. Variables declared "extern" have static storage duration. Therefore strictly conforming implementations should accept &PyBaseObject_Type as a valid constant initializer. I believe the text above could be replaced by something like: > MSVC does not support constant initialization of of an address > that comes from another DLL, so extensions should be set in the > extension module's init function. -- assignee: docs@python components: Documentation messages: 402752 nosy: docs@python, jhaberman priority: normal severity: normal status: open title: Docs are incorrect re: constant initialization in the C99 standard ___ Python tracker <https://bugs.python.org/issue45306> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > Windows/MSVC defines DLLs as separate programs, with their own lifetime and > entry point (e.g. you can reload a DLL multiple times and it will be > reinitialised each time). All of this is true of so's in ELF also. It doesn't mean that the implementation needs to reject standards-conforming programs. I still think the Python documentation is incorrect on this point. I filed https://bugs.python.org/issue45306 to track this separately. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15870] PyType_FromSpec should take metaclass as an argument
Josh Haberman added the comment: > Everything is copied by `_FromSpec` after all. One thing I noticed isn't copied is the string pointed to by tp_name: https://github.com/python/cpython/blob/0c50b8c0b8274d54d6b71ed7bd21057d3642f138/Objects/typeobject.c#L3427 This isn't an issue if tp_name is initialized from a string literal. But if tp_name is created dynamically, it could lead to a dangling pointer. If the general message is that "everything is copied by _FromSpec", it might make sense to copy the tp_name string too. > However, I suppose that would replace a safe-by-design API with a "best > practice" to never define the spec/slots statically (a best practice that is > probably not generally followed or even advertised currently, I guess). Yes that seems reasonable. I generally prefer static declarations, since they will end up in .data instead of .text and will avoid a copy to the stack at runtime. But these are very minor differences, especially for code that only runs once at startup, and a safe-by-default recommendation of always initializing PyType_* on the stack makes sense. -- ___ Python tracker <https://bugs.python.org/issue15870> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
