[Python-Dev] Make extension module initialisation more like Python module initialisation
Hi,
I suspect that this will be put into a proper PEP at some point, but I'd
like to bring this up for discussion first. This came out of issues 13429
and 16392.
http://bugs.python.org/issue13429
http://bugs.python.org/issue16392
Stefan
The problem
===
Python modules and extension modules are not being set up in the same way.
For Python modules, the module is created and set up first, then the module
code is being executed. For extensions, i.e. shared libraries, the module
init function is executed straight away and does both the creation and
initialisation. This means that it knows neither the __file__ it is being
loaded from nor its package (i.e. its FQMN). This hinders relative imports
and resource loading. In Py3, it's also not being added to sys.modules,
which means that a (potentially transitive) re-import of the module will
really try to reimport it and thus run into an infinite loop when it
executes the module init function again. And without the FQMN, it's not
trivial to correctly add the module to sys.modules either.
We specifically run into this for Cython generated modules, for which it's
not uncommon that the module init code has the same level of complexity as
that of any 'regular' Python module. Also, the lack of a FQMN and correct
file path hinders the compilation of __init__.py modules, i.e. packages,
especially when relative imports are being used at module init time.
The proposal
I propose to split the extension module initialisation into two steps in
Python 3.4, in a backwards compatible way.
Step 1: The current module init function can be reduced to just creating
the module instance and returning it (and potentially doing some simple C
level setup). Optionally, after creating the module (and this is the new
part), the module init code can register a C callback function that will be
called after setting up the module.
Step 2: The shared library importer receives the module instance from the
module init function, adds __file__, __path__, __package__ and friends to
the module dict, and then checks for the callback. If non-NULL, it calls it
to continue the module initialisation by user code.
The callback
The callback is defined as follows::
int (*PyModule_init_callback)(PyObject* the_module,
PyModuleInitContext* context)
"PyModuleInitContext" is a struct that is meant mostly for making the
callback more future proof by allowing additional parameters to be passed
in. For now, I can see a use case for the following fields::
struct PyModuleInitContext {
char* module_name;
char* qualified_module_name;
}
Both names are encoded in UTF-8. As for the file path, I consider it best
to retrieve it from the module's __file__ attribute as a Python string
object to reduce filename encoding problems.
Note that this struct argument is not strictly required, but given that
this proposal would have been much simpler if the module init function had
accepted such an argument in the first place, I consider it a good idea not
to let this chance pass by again.
The registration of the callback uses a new C-API function:
int PyModule_SetInitFunction(PyObject* module,
PyModule_init_callback callback)
The function name uses "Set" instead of "Register" to make it clear that
there is only one such function per module.
An alternative would be a new module creation function "PyModule_Create3()"
that takes the callback as third argument, in addition to what
"PyModule_Create2()" accepts. This would require users to explicitly pass
in the (second) version argument, which might be considered only a minor issue.
Implementation
==
The implementation requires local changes to the extension module importer
and a new C-API function. In order to store the callback, it should use a
new field in the module object struct.
Open questions
==
It is not clear how extensions should be handled that register more than
one module in their module init function, e.g. compiled packages. One
possibility would be to leave the setup to the user, who would have to know
all FQMNs anyway in this case, although not the import file path.
Alternatively, the import machinery could use a stack to remember for which
modules a callback was registered during the last init function call, set
up all of them and then call their callbacks. It's not clear if this meets
the intention of the user.
Alternatives
1) It would be possible to make extension modules optionally export another
symbol, e.g. "PyInit2_modulename", that the shared library loader would
call in addition to the required function "PyInit_modulename". This would
remove the need for a new API that registers the above callback. The
drawback is that it also makes it easier to write broken code because a
Python version or implementation that does not support this second symbol
would simply not call it, with
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
On 08.11.2012 13:47, Stefan Behnel wrote: > Hi, > > I suspect that this will be put into a proper PEP at some point, but I'd > like to bring this up for discussion first. This came out of issues 13429 > and 16392. > > http://bugs.python.org/issue13429 > > http://bugs.python.org/issue16392 > > Stefan > > > The problem > === > > Python modules and extension modules are not being set up in the same way. > For Python modules, the module is created and set up first, then the module > code is being executed. For extensions, i.e. shared libraries, the module > init function is executed straight away and does both the creation and > initialisation. This means that it knows neither the __file__ it is being > loaded from nor its package (i.e. its FQMN). This hinders relative imports > and resource loading. In Py3, it's also not being added to sys.modules, > which means that a (potentially transitive) re-import of the module will > really try to reimport it and thus run into an infinite loop when it > executes the module init function again. And without the FQMN, it's not > trivial to correctly add the module to sys.modules either. > > We specifically run into this for Cython generated modules, for which it's > not uncommon that the module init code has the same level of complexity as > that of any 'regular' Python module. Also, the lack of a FQMN and correct > file path hinders the compilation of __init__.py modules, i.e. packages, > especially when relative imports are being used at module init time. > > The proposal > > > ... [callbacks] ... > > Alternatives > > ... > 3) The callback could be registered statically in the PyModuleDef struct by > adding a new field. This is not trivial to do in a backwards compatible way > because the struct would grow longer without explicit initialisation by > existing user code. Extending PyModuleDef_HEAD_INIT might be possible but > would still break at least binary compatibility. I think the above is the cleaner approach than the callback mechanism. There's no problem in adding new slots to the end of the PyModuleDef struct - we've been doing that for years in many other structs :-) All you have to do is bump the Python API version number. (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 08 2012) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
M.-A. Lemburg, 08.11.2012 14:01: > On 08.11.2012 13:47, Stefan Behnel wrote: >> I suspect that this will be put into a proper PEP at some point, but I'd >> like to bring this up for discussion first. This came out of issues 13429 >> and 16392. >> >> http://bugs.python.org/issue13429 >> >> http://bugs.python.org/issue16392 >> >> Stefan >> >> >> The problem >> === >> >> Python modules and extension modules are not being set up in the same way. >> For Python modules, the module is created and set up first, then the module >> code is being executed. For extensions, i.e. shared libraries, the module >> init function is executed straight away and does both the creation and >> initialisation. This means that it knows neither the __file__ it is being >> loaded from nor its package (i.e. its FQMN). This hinders relative imports >> and resource loading. In Py3, it's also not being added to sys.modules, >> which means that a (potentially transitive) re-import of the module will >> really try to reimport it and thus run into an infinite loop when it >> executes the module init function again. And without the FQMN, it's not >> trivial to correctly add the module to sys.modules either. >> >> We specifically run into this for Cython generated modules, for which it's >> not uncommon that the module init code has the same level of complexity as >> that of any 'regular' Python module. Also, the lack of a FQMN and correct >> file path hinders the compilation of __init__.py modules, i.e. packages, >> especially when relative imports are being used at module init time. >> >> The proposal >> >> >> ... [callbacks] ... >> >> Alternatives >> >> ... >> 3) The callback could be registered statically in the PyModuleDef struct by >> adding a new field. This is not trivial to do in a backwards compatible way >> because the struct would grow longer without explicit initialisation by >> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but >> would still break at least binary compatibility. > > I think the above is the cleaner approach than the callback mechanism. Oh, definitely. > There's no problem in adding new slots to the end of the PyModuleDef struct > - we've been doing that for years in many other structs :-) > > All you have to do is bump the Python API version number. > > (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details) The difference is that this specific struct is provided by user code and (typically) initialised statically. There is no guarantee that user code that does not expect the additional field will initialise it to 0. Failing that, I don't see how we could trust its value in any way. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
Stefan Behnel, 08.11.2012 14:20: > M.-A. Lemburg, 08.11.2012 14:01: >> On 08.11.2012 13:47, Stefan Behnel wrote: >>> I suspect that this will be put into a proper PEP at some point, but I'd >>> like to bring this up for discussion first. This came out of issues 13429 >>> and 16392. >>> >>> http://bugs.python.org/issue13429 >>> >>> http://bugs.python.org/issue16392 >>> >>> Stefan >>> >>> >>> The problem >>> === >>> >>> Python modules and extension modules are not being set up in the same way. >>> For Python modules, the module is created and set up first, then the module >>> code is being executed. For extensions, i.e. shared libraries, the module >>> init function is executed straight away and does both the creation and >>> initialisation. This means that it knows neither the __file__ it is being >>> loaded from nor its package (i.e. its FQMN). This hinders relative imports >>> and resource loading. In Py3, it's also not being added to sys.modules, >>> which means that a (potentially transitive) re-import of the module will >>> really try to reimport it and thus run into an infinite loop when it >>> executes the module init function again. And without the FQMN, it's not >>> trivial to correctly add the module to sys.modules either. >>> >>> We specifically run into this for Cython generated modules, for which it's >>> not uncommon that the module init code has the same level of complexity as >>> that of any 'regular' Python module. Also, the lack of a FQMN and correct >>> file path hinders the compilation of __init__.py modules, i.e. packages, >>> especially when relative imports are being used at module init time. >>> >>> The proposal >>> >>> >>> ... [callbacks] ... >>> >>> Alternatives >>> >>> ... >>> 3) The callback could be registered statically in the PyModuleDef struct by >>> adding a new field. This is not trivial to do in a backwards compatible way >>> because the struct would grow longer without explicit initialisation by >>> existing user code. Extending PyModuleDef_HEAD_INIT might be possible but >>> would still break at least binary compatibility. >> >> I think the above is the cleaner approach than the callback mechanism. > > Oh, definitely. > > >> There's no problem in adding new slots to the end of the PyModuleDef struct >> - we've been doing that for years in many other structs :-) >> >> All you have to do is bump the Python API version number. >> >> (Martin's PEP http://www.python.org/dev/peps/pep-3121/ has the details) > > The difference is that this specific struct is provided by user code and > (typically) initialised statically. There is no guarantee that user code > that does not expect the additional field will initialise it to 0. Failing > that, I don't see how we could trust its value in any way. Hmm - you're actually right. In C, uninitialised fields in a static struct are set to 0 automatically. Same case as the type structs. That makes your objection perfectly valid. I'll rewrite and shorten the proposal. Thanks! Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Update - Re: Make extension module initialisation more like Python module initialisation
Hi,
here's an updated proposal, adopting Marc-Andre's improvement that uses a
new field in the PyModuleDef struct to register the callback. Note that
this change no longer keeps up binary compatibility, which may or may not
be acceptable for Python 3.4.
Stefan
The problem
===
Python modules and extension modules are not being set up in the same way.
For Python modules, the module is created and set up first, then the module
code is being executed. For extensions, i.e. shared libraries, the module
init function is executed straight away and does both the creation and
initialisation. This means that it knows neither the __file__ it is being
loaded from nor its package (i.e. its FQMN). This hinders relative imports
and resource loading. In Py3, it's also not being added to sys.modules,
which means that a (potentially transitive) re-import of the module will
really try to reimport it and thus run into an infinite loop when it
executes the module init function again. And without the FQMN, it's not
trivial to correctly add the module to sys.modules either.
We specifically run into this for Cython generated modules, for which it's
not uncommon that the module init code has the same level of complexity as
that of any 'regular' Python module. Also, the lack of a FQMN and correct
file path hinders the compilation of __init__.py modules, i.e. packages,
especially when relative imports are being used at module init time.
The proposal
I propose to split the extension module initialisation into two steps in
Python 3.4, in a backwards compatible way.
Step 1: The current module init function can be reduced to just creating
the module instance and returning it (and potentially doing some simple C
level setup). Additionally, and this is the new part, the module init code
can register a C callback function in its PyModuleDef struct that will be
called after setting up the module.
Step 2: The shared library importer receives the module instance from the
module init function, adds __file__, __path__, __package__ and friends to
the module dict, and then checks for the callback. If non-NULL, it calls it
to continue the module initialisation by user code.
The callback
The callback is defined as follows::
int (*PyModule_init_callback)(PyObject* the_module,
PyModuleInitContext* context)
"PyModuleInitContext" is a struct that is meant mostly for making the
callback more future proof by allowing additional parameters to be passed
in. For now, I can see a use case for the following fields::
struct PyModuleInitContext {
char* module_name;
char* qualified_module_name;
}
Both names are encoded in UTF-8. As for the file path, I consider it best
to retrieve it from the module's __file__ attribute as a Python string
object to reduce filename encoding problems.
Note that this struct argument is not strictly required (it could be a
simple "inquiry" function), but given that this proposal would have been
much simpler if the module init function had accepted such an argument in
the first place, I consider it a good idea not to let this chance pass by
again. The counter arguments would be "keep it simple" and "we already pass
in the whole module (and its dict) anyway". Up for debate!
The registration of the callback uses a new field "m_init" in the
PyModuleDef struct::
typedef struct PyModuleDef{
PyModuleDef_Base m_base;
const char* m_name;
const char* m_doc;
Py_ssize_t m_size;
PyMethodDef *m_methods;
inquiry m_reload;
traverseproc m_traverse;
inquiry m_clear;
freefunc m_free; /* --- original fields up to here */
PyModule_init_callback m_init; /* post-setup init callback */
} PyModuleDef;
Implementation
==
The implementation requires local changes to the extension module importer
and a new field in the PyModuleDef struct.
Open questions
==
It is not clear how extensions should be handled that register more than
one module in their module init function, e.g. compiled packages. One
possibility would be to leave the setup to the user, who would have to know
all FQMNs anyway in this case, although not the import file path.
Alternatively, the import machinery could use a stack to remember for which
modules a callback was registered during the last init function call, set
up all of them and then call their callbacks. It's not clear if this meets
the intention of the user. It's not guaranteed that all of these modules
will be related to the module that registered them, in the sense that they
should receive the same setup. The best way to fix this correctly might be
to make users pass the setup explicitly into the module creation functions
in Python 4 (see alternatives below), so that the setup and sys.modules
registration can happen directly at this point.
Alternatives
1) It would be possible to make extension modules o
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote:
> Hi,
>
> I suspect that this will be put into a proper PEP at some point, but I'd
> like to bring this up for discussion first. This came out of issues 13429
> and 16392.
>
> http://bugs.python.org/issue13429
>
> http://bugs.python.org/issue16392
>
> Stefan
>
>
> The problem
> ===
>
> Python modules and extension modules are not being set up in the same way.
> For Python modules, the module is created and set up first, then the module
> code is being executed. For extensions, i.e. shared libraries, the module
> init function is executed straight away and does both the creation and
> initialisation. This means that it knows neither the __file__ it is being
> loaded from nor its package (i.e. its FQMN). This hinders relative imports
> and resource loading. In Py3, it's also not being added to sys.modules,
> which means that a (potentially transitive) re-import of the module will
> really try to reimport it and thus run into an infinite loop when it
> executes the module init function again. And without the FQMN, it's not
> trivial to correctly add the module to sys.modules either.
>
> We specifically run into this for Cython generated modules, for which it's
> not uncommon that the module init code has the same level of complexity as
> that of any 'regular' Python module. Also, the lack of a FQMN and correct
> file path hinders the compilation of __init__.py modules, i.e. packages,
> especially when relative imports are being used at module init time.
>
Or to put it another way, importlib doesn't give you a nice class to
inherit from which will handle all of the little details of creating a
blank module (or fetching from sys.modules if you are reloading), setting
__file__, __cached__, __package__, __name__, __loader__, and (optionally)
__path__ for you, and then cleaning up if something goes wrong. It's a pain
to do all of this yourself and to get all the details right (i.e. there's a
reason that @importlib.util.module_for_loader exists).
>
> The proposal
>
>
> I propose to split the extension module initialisation into two steps in
> Python 3.4, in a backwards compatible way.
>
> Step 1: The current module init function can be reduced to just creating
> the module instance and returning it (and potentially doing some simple C
> level setup). Optionally, after creating the module (and this is the new
> part), the module init code can register a C callback function that will be
> called after setting up the module.
>
Why even bother with the module creation? Why can't Python do that as well
and then call the callback?
>
> Step 2: The shared library importer receives the module instance from the
> module init function, adds __file__, __path__, __package__ and friends to
> the module dict, and then checks for the callback. If non-NULL, it calls it
> to continue the module initialisation by user code.
> The callback
>
>
> The callback is defined as follows::
>
> int (*PyModule_init_callback)(PyObject* the_module,
> PyModuleInitContext* context)
>
> "PyModuleInitContext" is a struct that is meant mostly for making the
> callback more future proof by allowing additional parameters to be passed
> in. For now, I can see a use case for the following fields::
>
> struct PyModuleInitContext {
> char* module_name;
> char* qualified_module_name;
> }
>
> Both names are encoded in UTF-8. As for the file path, I consider it best
> to retrieve it from the module's __file__ attribute as a Python string
> object to reduce filename encoding problems.
>
> Note that this struct argument is not strictly required, but given that
> this proposal would have been much simpler if the module init function had
> accepted such an argument in the first place, I consider it a good idea not
> to let this chance pass by again.
>
> The registration of the callback uses a new C-API function:
>
> int PyModule_SetInitFunction(PyObject* module,
> PyModule_init_callback callback)
>
> The function name uses "Set" instead of "Register" to make it clear that
> there is only one such function per module.
>
> An alternative would be a new module creation function "PyModule_Create3()"
> that takes the callback as third argument, in addition to what
> "PyModule_Create2()" accepts. This would require users to explicitly pass
> in the (second) version argument, which might be considered only a minor
> issue.
>
> Implementation
> ==
>
> The implementation requires local changes to the extension module importer
> and a new C-API function. In order to store the callback, it should use a
> new field in the module object struct.
>
> Open questions
> ==
>
> It is not clear how extensions should be handled that register more than
> one module in their module init function, e.g. compiled packages. One
> possibility would be to leave the setup to th
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
Hi Brett, thanks for the feedback. Brett Cannon, 08.11.2012 15:41: > On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote: >> I propose to split the extension module initialisation into two steps in >> Python 3.4, in a backwards compatible way. >> >> Step 1: The current module init function can be reduced to just creating >> the module instance and returning it (and potentially doing some simple C >> level setup). Optionally, after creating the module (and this is the new >> part), the module init code can register a C callback function that will be >> called after setting up the module. > > Why even bother with the module creation? Why can't Python do that as well > and then call the callback? > > >> Step 2: The shared library importer receives the module instance from the >> module init function, adds __file__, __path__, __package__ and friends to >> the module dict, and then checks for the callback. If non-NULL, it calls it >> to continue the module initialisation by user code. > [...] > An alternative to the alternative is that if the PyInit2 function exists > it's called instead of the the PyInit function, and then the PyInit > function is nothing more than a single line function call (or whatever the > absolute bare minimum is) into some helper that calls the PyInit2 call > properly for backwards ABI compatibility (i.e. passes in whatever details > are lost by the indirection in function call). That provides an eventual > upgrade path of dropping PyInit and moving over to PyInit2. In that case, you'd have to export the PyModuleDef descriptor as well, because that's what tells CPython how the module behaves and what to do with it to set it up properly (e.g. allocate module state space on the heap). In fact, if the module init function became a field in the descriptor, it would be enough (taking backwards compatibility aside) if *only* the descriptor was exported and used by the module loader. With the caveat that this might kill some less common but not necessarily illegitimate use cases that do more than just creating and initialising a single module... Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
On Thu, Nov 8, 2012 at 10:00 AM, Stefan Behnel wrote: > Hi Brett, > > thanks for the feedback. > > Brett Cannon, 08.11.2012 15:41: > > On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote: > >> I propose to split the extension module initialisation into two steps in > >> Python 3.4, in a backwards compatible way. > >> > >> Step 1: The current module init function can be reduced to just creating > >> the module instance and returning it (and potentially doing some simple > C > >> level setup). Optionally, after creating the module (and this is the new > >> part), the module init code can register a C callback function that > will be > >> called after setting up the module. > > > > Why even bother with the module creation? Why can't Python do that as > well > > and then call the callback? > > > > > >> Step 2: The shared library importer receives the module instance from > the > >> module init function, adds __file__, __path__, __package__ and friends > to > >> the module dict, and then checks for the callback. If non-NULL, it > calls it > >> to continue the module initialisation by user code. > > [...] > > An alternative to the alternative is that if the PyInit2 function exists > > it's called instead of the the PyInit function, and then the PyInit > > function is nothing more than a single line function call (or whatever > the > > absolute bare minimum is) into some helper that calls the PyInit2 call > > properly for backwards ABI compatibility (i.e. passes in whatever details > > are lost by the indirection in function call). That provides an eventual > > upgrade path of dropping PyInit and moving over to PyInit2. > > In that case, you'd have to export the PyModuleDef descriptor as well, > because that's what tells CPython how the module behaves and what to do > with it to set it up properly (e.g. allocate module state space on the > heap). > True. > > In fact, if the module init function became a field in the descriptor, it > would be enough (taking backwards compatibility aside) if *only* the > descriptor was exported and used by the module loader. > > Also true. > With the caveat that this might kill some less common but not necessarily > illegitimate use cases that do more than just creating and initialising a > single module... > You mean creating another module in the init function? That's fine, but that should be a call to __import__ anyway and that should handle things properly. Else you are circumventing the import system and you can do everything from scratch. I don't see why this would stop you from doing anything you want, it just simplifies the common case. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Make extension module initialisation more like Python module initialisation
Brett Cannon, 08.11.2012 16:06: > On Thu, Nov 8, 2012 at 10:00 AM, Stefan Behnel wrote: > >> Hi Brett, >> >> thanks for the feedback. >> >> Brett Cannon, 08.11.2012 15:41: >>> On Thu, Nov 8, 2012 at 7:47 AM, Stefan Behnel wrote: I propose to split the extension module initialisation into two steps in Python 3.4, in a backwards compatible way. Step 1: The current module init function can be reduced to just creating the module instance and returning it (and potentially doing some simple >> C level setup). Optionally, after creating the module (and this is the new part), the module init code can register a C callback function that >> will be called after setting up the module. >>> >>> Why even bother with the module creation? Why can't Python do that as >> well >>> and then call the callback? >>> >>> Step 2: The shared library importer receives the module instance from >> the module init function, adds __file__, __path__, __package__ and friends >> to the module dict, and then checks for the callback. If non-NULL, it >> calls it to continue the module initialisation by user code. >>> [...] >>> An alternative to the alternative is that if the PyInit2 function exists >>> it's called instead of the the PyInit function, and then the PyInit >>> function is nothing more than a single line function call (or whatever >> the >>> absolute bare minimum is) into some helper that calls the PyInit2 call >>> properly for backwards ABI compatibility (i.e. passes in whatever details >>> are lost by the indirection in function call). That provides an eventual >>> upgrade path of dropping PyInit and moving over to PyInit2. >> >> In that case, you'd have to export the PyModuleDef descriptor as well, >> because that's what tells CPython how the module behaves and what to do >> with it to set it up properly (e.g. allocate module state space on the >> heap). > > True. > >> In fact, if the module init function became a field in the descriptor, it >> would be enough (taking backwards compatibility aside) if *only* the >> descriptor was exported and used by the module loader. > > Also true. > >> With the caveat that this might kill some less common but not necessarily >> illegitimate use cases that do more than just creating and initialising a >> single module... > > You mean creating another module in the init function? That's fine, but > that should be a call to __import__ anyway and that should handle things > properly. Ok. > Else you are circumventing the import system and you can do > everything from scratch. I guess I'd be ok with putting that burden on users in this case. > I don't see why this would stop you from doing > anything you want, it just simplifies the common case. The only problematic case I see here would be a module that calculates the size of its state space at init time, e.g. based on some platform specifics or environment parameters, anything from the platform specific size of some data type to the runtime configured number of OpenMP threads. That would make the PyModuleDef a compile time static thing - not sure if that's currently required. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update - Re: Make extension module initialisation more like Python module initialisation
On Fri, Nov 9, 2012 at 12:32 AM, Stefan Behnel wrote: > here's an updated proposal, adopting Marc-Andre's improvement that uses a > new field in the PyModuleDef struct to register the callback. Note that > this change no longer keeps up binary compatibility, which may or may not > be acceptable for Python 3.4. > It's not acceptable, as PyModuleDef is part of PEP 384's stable ABI. All such public structures are locked at their original size. 3) The original proposal used a new C-API function to register the callback > explicitly, as opposed to extending the PyModuleDef struct. This has the > advantage of keeping up binary compatibility with existing Py3.3 > extensions. It has the disadvantage of adding another indirection to the > setup procedure where a static function pointer would suffice. > Module initialisation is (and must be) part of the stable ABI. Indirection (especially through Python) is a *good* thing, as, ideally, any new interfaces should be defined in a way that doesn't increase the maintenance burden for the stable ABI. I don't agree that the use of a new init API can fail silently, so long as it completely *replaces* the old API, rather than being an addition. That way, since you won't be defining the *old* init function at all, old versions will correctly refuse to load your module. So I propose that we simply *fix* extension module loading to work the same way as everything else: the loader creates the module object, and passes it in to a new init function to be fully populated. __file__ and __name__ would be passed in as preinitialised module attributes. The existing PyModule_Create functions would be complemented by a PyModule_SetDef function which allowed a PyModuleDef to be configured on a pre-existing module. Extension modules that wanted to create multiple Python modules would still be free to do so - it would just be up to the extension initialisation code to call PyModule_Create to construct them and set __file__ based on the __file__ of the passed in module. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Detecting tp_compare / tp_richcompare from Python
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
While porting the BTrees package (split out from ZODB) to Python3., my
first step is to get the pure-Python reference implementation working:
in order to do that, I need a way to check that objects used as keys are
*not* using the comparison semantics inherited from the base 'object'
class, because those semantics rely on properties which are not stable
across the lifetime of the (persisted / restored) key.
The existing C code does something like::
static int
check_argument_cmp(PyObject *arg)
{
if (arg->ob_type->tp_richcompare == NULL
&& arg->ob_type->tp_compare ==
((PyTypeObject *)object_)->ob_type->tp_compare)
{
PyErr_SetString(PyExc_TypeError, "Object has default comparison");
return 0;
}
return 1;
}
Unless I'm mistaken, there is no way to do the equivalent from pure
Python.. I tried a couple of approximations which relied on non-API
attributes (I"m looking at out, methodwrapper.__objclass__), but without
much success (and I need the code to work on PyPy / Jython, too).
Am I missing something?
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCb21gACgkQ+gerLs4ltQ4jBACfV0lQaQ2eW2vhAtWunLUsPQWM
esEAoMYeeQvlJVnckaBg4HM19LoxPIWB
=+d+0
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Detecting tp_compare / tp_richcompare from Python
Well, the default behavior has changed to raise an exception when
using <, <=, >, >=; i.e., inequalities do not have a default
implementation at all. Perhaps that is enough for your purpose? == and
!= still compare by pointer, but you're primarily interested in
ordering her, right?
On Thu, Nov 8, 2012 at 8:18 AM, Tres Seaver wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> While porting the BTrees package (split out from ZODB) to Python3., my
> first step is to get the pure-Python reference implementation working:
> in order to do that, I need a way to check that objects used as keys are
> *not* using the comparison semantics inherited from the base 'object'
> class, because those semantics rely on properties which are not stable
> across the lifetime of the (persisted / restored) key.
>
> The existing C code does something like::
>
> static int
> check_argument_cmp(PyObject *arg)
> {
> if (arg->ob_type->tp_richcompare == NULL
>&& arg->ob_type->tp_compare ==
> ((PyTypeObject *)object_)->ob_type->tp_compare)
> {
>PyErr_SetString(PyExc_TypeError, "Object has default comparison");
>return 0;
> }
>return 1;
> }
>
> Unless I'm mistaken, there is no way to do the equivalent from pure
> Python.. I tried a couple of approximations which relied on non-API
> attributes (I"m looking at out, methodwrapper.__objclass__), but without
> much success (and I need the code to work on PyPy / Jython, too).
>
> Am I missing something?
>
>
>
> Tres.
> - --
> ===
> Tres Seaver +1 540-429-0999 tsea...@palladion.com
> Palladion Software "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlCb21gACgkQ+gerLs4ltQ4jBACfV0lQaQ2eW2vhAtWunLUsPQWM
> esEAoMYeeQvlJVnckaBg4HM19LoxPIWB
> =+d+0
> -END PGP SIGNATURE-
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding
On 08.11.12 01:08, R. David Murray wrote: Alexandre's point was that the string did not appear to be arbitrary, but rather appeared to specifically be a string containing surrogates. Is this not the case? My intention was testing with filename which cannot be decoded as UTF-8 in strict mode. I agree that testing with name which is encodable in locale encoding can be useful too, but now the test has no effect on UTF-8 locale. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Detecting tp_compare / tp_richcompare from Python
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11/08/2012 12:38 PM, Guido van Rossum wrote: > Well, the default behavior has changed to raise an exception when > using <, <=, >, >=; i.e., inequalities do not have a default > implementation at all. Perhaps that is enough for your purpose? == > and != still compare by pointer, but you're primarily interested in > ordering her, right? The b-search algorithm needs ordering on keys, but also meaningful (non-identity) equality. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iEYEARECAAYFAlCb8iMACgkQ+gerLs4ltQ6WOACgwERmICWr80qnEoOVVLQtFwTH ttAAnAt4An0dPjaRuZyJlDAUGzH0CS7B =u2mG -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Detecting tp_compare / tp_richcompare from Python
In Python, you can write
if C.__eq__ == object.__eq__:
print('class C does not override __eq__')
Does that help?
On Thu, Nov 8, 2012 at 9:55 AM, Tres Seaver wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> On 11/08/2012 12:38 PM, Guido van Rossum wrote:
>> Well, the default behavior has changed to raise an exception when
>> using <, <=, >, >=; i.e., inequalities do not have a default
>> implementation at all. Perhaps that is enough for your purpose? ==
>> and != still compare by pointer, but you're primarily interested in
>> ordering her, right?
>
> The b-search algorithm needs ordering on keys, but also meaningful
> (non-identity) equality.
>
>
> Tres.
> - --
> ===
> Tres Seaver +1 540-429-0999 tsea...@palladion.com
> Palladion Software "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
>
> iEYEARECAAYFAlCb8iMACgkQ+gerLs4ltQ6WOACgwERmICWr80qnEoOVVLQtFwTH
> ttAAnAt4An0dPjaRuZyJlDAUGzH0CS7B
> =u2mG
> -END PGP SIGNATURE-
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] chained assignment weirdity
On 08.11.12 03:11, Ned Batchelder wrote:
> Sorry, I should have been clearer: I was asking about weird not to say,
> "This is weird and should be changed!", but to get clarification from
> Serhiy about his statement, " It will be weird if a dict comprehension
> and a plain loop will be inconsistent." I honestly didn't know which
> behavior he considered inconsistent and therefore weird.
I was referring to two of the most popular idioms to dynamically create a dict.
d = {}
for x in a:
d[k(x)] = v(x)
d = {k(x): v(x) for x in a}
For now these methods are consistent.
I agree that the use of the side effects here is not a sensible idea, but when
such effects occur by accident, it will cause a surprise.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding
On Thu, Nov 8, 2012 at 9:45 AM, Serhiy Storchaka wrote: > My intention was testing with filename which cannot be decoded as UTF-8 in > strict mode. I agree that testing with name which is encodable in locale > encoding can be useful too, but now the test has no effect on UTF-8 locale. So should we change the test back? Or just change the test name? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: Issue #16218: skip test if filesystem doesn't support required encoding
On 08.11.12 20:10, Alexandre Vassalotti wrote: So should we change the test back? Or just change the test name? No. I also missed some details. The issue is still not closed. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Detecting tp_compare / tp_richcompare from Python
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/08/2012 12:59 PM, Guido van Rossum wrote:
> In Python, you can write
>
> if C.__eq__ == object.__eq__: print('class C does not override
> __eq__')
>
> Does that help?
That works in Python3, but not in Python2 -- methodwrappers don't compare
equal the same way slotwrappers do :(. I still need to straddle (back to
Python 2.6.
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCcDVoACgkQ+gerLs4ltQ49kwCgtBJX1oyi5cudZeE4LBEdcAmO
aTAAn0jL56d3eK/sDUD3G7zzZ/ZdQH4W
=NG86
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Detecting tp_compare / tp_richcompare from Python
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On 11/08/2012 02:51 PM, Tres Seaver wrote:
> On 11/08/2012 12:59 PM, Guido van Rossum wrote:
>> In Python, you can write
>
>> if C.__eq__ == object.__eq__: print('class C does not override
>> __eq__')
>
>> Does that help?
>
> That works in Python3, but not in Python2 -- methodwrappers don't
> compare equal the same way slotwrappers do :(. I still need to
> straddle (back to Python 2.6.
I think I'm going to need two separate checkers:
- - In Py3k, use your check above for both '__eq__' and '__lt__', e.g:
def check_comparable(key);
klass = type(key)
if klass.__eq__ == object.__eq__ or klass.__lt__ == object.__lt__:
raise TypeError('Incomparable')
- - In Python2, I think I can just check that the instance (not its class)
has either ('__lt__' and '__eq__') *or* '__cmp__', e.g.:
def check_comparable(key);
if not ((getattr(key, '__eq__', None) is not None and
getattr(key, '__lt__', None) is not None) or
getattr(key, '__cmp__', None) is not None):
raise TypeError('Incomparable')
Tres.
- --
===
Tres Seaver +1 540-429-0999 tsea...@palladion.com
Palladion Software "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlCcFrcACgkQ+gerLs4ltQ6JDgCeJpDmo70H9Xgz3preNHVvaTl5
dYkAoNS+fpSXwZiaD2J2ONyQZ2mPIcFC
=OiMY
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon
On 11/06/2012 02:56 PM, tim.golden wrote:
> http://hg.python.org/cpython/rev/dafca4714298
> changeset: 80273:dafca4714298
> user:Tim Golden
> date:Tue Nov 06 13:50:42 2012 +
> summary:
> issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon
>
> files:
> Doc/library/glob.rst | 11 ++--
> Lib/glob.py | 65 ++
> Lib/test/test_glob.py | 64 +-
> Misc/NEWS | 3 +
> 4 files changed, 118 insertions(+), 25 deletions(-)
>
>
> diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst
> --- a/Doc/library/glob.rst
> +++ b/Doc/library/glob.rst
> @@ -13,10 +13,10 @@
>
> The :mod:`glob` module finds all the pathnames matching a specified pattern
> according to the rules used by the Unix shell. No tilde expansion is done,
> but
> -``*``, ``?``, and character ranges expressed with ``[]`` will be correctly
> -matched. This is done by using the :func:`os.listdir` and
> -:func:`fnmatch.fnmatch` functions in concert, and not by actually invoking a
> -subshell. (For tilde and shell variable expansion, use
> +``*``, ``?``, character ranges expressed with ``[]`` and list of options
> +expressed with ``{}`` will be correctly matched. This is done by using the
> +:func:`os.listdir` and :func:`fnmatch.fnmatch` functions in concert, and not
> by
> +actually invoking a subshell. (For tilde and shell variable expansion, use
> :func:`os.path.expanduser` and :func:`os.path.expandvars`.)
Needs a versionchanged.
In any case, brace expansion is not part of globbing (see the bash or zsh
manuals) because it does not generate valid file names, and it is a non-POSIX
expansion of some shells. Are you sure it should be put into the glob module?
(Not speaking of the backward incompatibility it creates.)
Georg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon
On 11/08/2012 09:43 PM, Georg Brandl wrote: > > Needs a versionchanged. > > In any case, brace expansion is not part of globbing (see the bash or zsh > manuals) because it does not generate valid file names, and it is a non-POSIX > expansion of some shells. Are you sure it should be put into the glob module? > (Not speaking of the backward incompatibility it creates.) Sorry, just saw the reversion. Never mind, my concerns have been addressed. Georg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython: issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon
On 08/11/2012 20:43, Georg Brandl wrote:
On 11/06/2012 02:56 PM, tim.golden wrote:
http://hg.python.org/cpython/rev/dafca4714298
changeset: 80273:dafca4714298
user:Tim Golden
date:Tue Nov 06 13:50:42 2012 +
summary:
issue9584: Add {} list expansion to glob. Original patch by Mathieu Bridon
files:
Doc/library/glob.rst | 11 ++--
Lib/glob.py | 65 ++
Lib/test/test_glob.py | 64 +-
Misc/NEWS | 3 +
4 files changed, 118 insertions(+), 25 deletions(-)
diff --git a/Doc/library/glob.rst b/Doc/library/glob.rst
--- a/Doc/library/glob.rst
+++ b/Doc/library/glob.rst
@@ -13,10 +13,10 @@
The :mod:`glob` module finds all the pathnames matching a specified pattern
according to the rules used by the Unix shell. No tilde expansion is done,
but
-``*``, ``?``, and character ranges expressed with ``[]`` will be correctly
-matched. This is done by using the :func:`os.listdir` and
-:func:`fnmatch.fnmatch` functions in concert, and not by actually invoking a
-subshell. (For tilde and shell variable expansion, use
+``*``, ``?``, character ranges expressed with ``[]`` and list of options
+expressed with ``{}`` will be correctly matched. This is done by using the
+:func:`os.listdir` and :func:`fnmatch.fnmatch` functions in concert, and not by
+actually invoking a subshell. (For tilde and shell variable expansion, use
:func:`os.path.expanduser` and :func:`os.path.expandvars`.)
Needs a versionchanged.
In any case, brace expansion is not part of globbing (see the bash or zsh
manuals) because it does not generate valid file names, and it is a non-POSIX
expansion of some shells. Are you sure it should be put into the glob module?
(Not speaking of the backward incompatibility it creates.)
I backed it out very soon afterwards, Georg. It had had some (quite a
bit of) discussion on the issue, but I'd messed up the patch somehow and
the backwards compat issue was raised pretty much immediately by Serhiy.
So I pulled the commit.
TJG
Insofar as
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
