Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-08 Thread Stefan Behnel
Eric Snow, 08.09.2013 00:22: > On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan wrote: > >> The hook API I currently have in mind is a two step initialisation: >> >> PyImport_PrepareNAME (optional) >> PyImport_ExecNAME >> > > Should we also look at an API change for the initfunc() of PyImport

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-07 Thread Eric Snow
On Mon, Sep 2, 2013 at 7:02 AM, Nick Coghlan wrote: > The hook API I currently have in mind is a two step initialisation: > > PyImport_PrepareNAME (optional) > PyImport_ExecNAME > Should we also look at an API change for the initfunc() of PyImport_Inittab entries? Currently the function

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Stefan Behnel
Antoine Pitrou, 06.09.2013 07:43: > Proxying wrapper? We shouldn't need that kind of tricks. The advantage is that it's controlled by the loader, and transparent to the module itself. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://ma

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Mon, Sep 2, 2013 at 2:16 AM, Antoine Pitrou wrote: > I think the biggest challenge here is to propose an API that's simple > and easy to use (i.e. that doesn't make extension module writing more > complicated than it currently is). > +1 > > The basic concept of putting custom module objects

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Sat, Aug 31, 2013 at 1:16 PM, Stefan Behnel wrote: > Nick Coghlan, 31.08.2013 18:49: > > This is actually my primary motivation for trying to improve the > > "can this be reloaded or not?" aspects of the loader API in PEP 451. > > I assume you mean that the extension module would be able to cl

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Sat, Aug 24, 2013 at 10:12 PM, PJ Eby wrote: > I haven't had a chance to address this on the import-sig discussion > yet about ModuleSpec, but I would like to just mention that one > property of the existing module system that I'm not sure either this > proposal or the ModuleSpec proposal pres

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Antoine Pitrou
On Thu, 5 Sep 2013 23:26:31 -0600 Eric Snow wrote: > On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel wrote: > > > PEP 3121 would no longer be necessary. Extension types can do all we need. > > No more special casing of modules, that was the idea. > > > > One nice thing about PEP 3121 is the addi

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-05 Thread Eric Snow
On Sat, Aug 24, 2013 at 7:07 AM, Stefan Behnel wrote: > PEP 3121 would no longer be necessary. Extension types can do all we need. > No more special casing of modules, that was the idea. > One nice thing about PEP 3121 is the addition of md_state to module objects to store internal module state.

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-02 Thread Nick Coghlan
On 2 September 2013 18:16, Antoine Pitrou wrote: > Le Sun, 1 Sep 2013 02:19:48 +1000, > Nick Coghlan a écrit : >> On 1 Sep 2013 00:10, "Stefan Behnel" wrote: >> > >> > *bump* >> > >> > Does this sound like a viable solution? >> >> This isn't likely to progress until we have Eric's PEP 451 to a p

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-02 Thread Antoine Pitrou
Le Sun, 1 Sep 2013 02:19:48 +1000, Nick Coghlan a écrit : > On 1 Sep 2013 00:10, "Stefan Behnel" wrote: > > > > *bump* > > > > Does this sound like a viable solution? > > This isn't likely to progress until we have Eric's PEP 451 to a point > where it's ready for python-dev discussion and pronou

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Antoine Pitrou
On Mon, 2 Sep 2013 00:10:08 +1000 Nick Coghlan wrote: > > $ ./python > Python 3.4.0a1+ (default:575071257c92+, Aug 25 2013, 00:42:17) > [GCC 4.7.2 20121109 (Red Hat 4.7.2-8)] on linux > Type "help", "copyright", "credits" or "license" for more information. > >>> import atexit > >>> atexit.registe

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Nick Coghlan
On 1 September 2013 23:03, Antoine Pitrou wrote: > On Sun, 1 Sep 2013 11:28:36 +1000 > Nick Coghlan wrote: >> * PEP 3121 with a size of "0". As above, but avoids the module state APIs >> in order to support reloading. All module state (including type >> cross-references) is stored in hidden state

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Antoine Pitrou
On Sun, 1 Sep 2013 11:28:36 +1000 Nick Coghlan wrote: > * PEP 3121 with a size of "0". As above, but avoids the module state APIs > in order to support reloading. All module state (including type > cross-references) is stored in hidden state (e.g. an instance of a custom > type not exposed to Pyth

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Stefan Behnel
Nick Coghlan, 01.09.2013 14:23: > That means the powers any new extension initialisation API will offer > will be limited to: > > * letting the module know its own name (and other details) > * letting the module explicitly block reloading > * letting the module support loading multiple copies at o

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Nick Coghlan
On 1 September 2013 18:11, Stefan Behnel wrote: > Nick Coghlan, 01.09.2013 03:28: >> On 1 Sep 2013 05:18, "Stefan Behnel" wrote: >>> I can't really remember a case where I could afford the >>> runtime overhead of implementing a wrapper in Python and going through >>> something like ctypes or cffi.

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-09-01 Thread Stefan Behnel
Nick Coghlan, 01.09.2013 03:28: > On 1 Sep 2013 05:18, "Stefan Behnel" wrote: >> I can't really remember a case where I could afford the >> runtime overhead of implementing a wrapper in Python and going through >> something like ctypes or cffi. I mean, testing C libraries with Python >> tools would

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Nick Coghlan
On 1 Sep 2013 05:18, "Stefan Behnel" wrote: > > Nick Coghlan, 31.08.2013 18:49: > > On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > > One key point to note is that it *doesn't* call > > _PyImport_FixupExtensionObject, which is the API that handles all the > > PEP 3121 per-module state s

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Stefan Behnel
Antoine Pitrou, 31.08.2013 21:27: > On Sat, 31 Aug 2013 21:16:10 +0200 > Stefan Behnel wrote: >>> Our experience is very different - my perspective is that the normal case >>> either eschews C level global state in the extension module, because it >>> causes so many problems, or else just completel

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Antoine Pitrou
On Sat, 31 Aug 2013 21:16:10 +0200 Stefan Behnel wrote: > > Our experience is very different - my perspective is that the normal case > > either eschews C level global state in the extension module, because it > > causes so many problems, or else just completely ignores subinterpreter > > support

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Stefan Behnel
Nick Coghlan, 31.08.2013 18:49: > On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > One key point to note is that it *doesn't* call > _PyImport_FixupExtensionObject, which is the API that handles all the > PEP 3121 per-module state stuff. Instead, the idea will be for modules > that do

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Nick Coghlan
Oops, had a draft from a few days ago that I was interrupted before sending. Finished editing the parts I believe are still relevant. On 25 Aug 2013 21:56, "Stefan Behnel" wrote: > > Nick Coghlan, 24.08.2013 23:43: > > On 25 Aug 2013 01:44, "Stefan Behnel" wrote: > >> Nick Coghlan, 24.08.2013 16:

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Nick Coghlan
On 1 Sep 2013 00:10, "Stefan Behnel" wrote: > > *bump* > > Does this sound like a viable solution? This isn't likely to progress until we have Eric's PEP 451 to a point where it's ready for python-dev discussion and pronouncement. However, the revised loader API is being designed to allow for th

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-31 Thread Stefan Behnel
*bump* Does this sound like a viable solution? Stefan Stefan Behnel, 25.08.2013 14:36: > Hi, > > thanks for bringing this up. It clearly shows that there is more to this > problem than I initially thought. > > Let me just add one idea that your post gave me. > > PJ Eby, 25.08.2013 06:12: >>

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Terry Reedy
On 8/25/2013 7:54 AM, Stefan Behnel wrote: And what if you do "from extmodule import some_function" in a Python module? Then reloading couldn't replace that reference, just as for normal Python modules. Meaning that you'd still have to keep both modules properly alive in order to prevent crashes

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Stefan Behnel
Hi, thanks for bringing this up. It clearly shows that there is more to this problem than I initially thought. Let me just add one idea that your post gave me. PJ Eby, 25.08.2013 06:12: > My "Importing" package offers lazy imports by creating module objects > in sys.modules that are a subtype of

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-25 Thread Stefan Behnel
Nick Coghlan, 24.08.2013 23:43: > On 25 Aug 2013 01:44, "Stefan Behnel" wrote: >> Nick Coghlan, 24.08.2013 16:22: >>> The new _PyImport_CreateAndExecExtensionModule function does the heavy >>> lifting: >>> >>> https://bitbucket.org/ncoghlan/cpython_sandbox/src/081f8f7e3ee27dc309463b48e6c67cf4880fca

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 25 August 2013 14:12, PJ Eby wrote: > That is to say, in the pure PEP 302 world, there is no special status > for "reload" that is different from "load" -- the *only* thing that's > different is that there is already a module object to use, and there > is *no guarantee that it's a module object

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread PJ Eby
On Fri, Aug 23, 2013 at 4:50 AM, Stefan Behnel wrote: > Reloading and Sub-Interpreters > == > > To "reload" an extension module, the module create function is executed > again and returns a new module type. This type is then instantiated as by > the original module load

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 25 Aug 2013 01:44, "Stefan Behnel" wrote: > > Nick Coghlan, 24.08.2013 16:22: > > On 24 August 2013 23:19, Stefan Behnel wrote: > >> Nick Coghlan, 24.08.2013 13:36: > >>> On 24 August 2013 15:51, Nick Coghlan wrote: > My current plan is to create an experimental prototype of this > ap

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 25 Aug 2013 05:19, "Benjamin Peterson" wrote: > > 2013/8/24 Terry Reedy : > > On 8/24/2013 8:51 AM, Stefan Behnel wrote: > >> > >> Antoine Pitrou, 24.08.2013 13:53: > >>> > >>> This would also imply extension module have to be subclasses of the > >>> built-in module type. They can't be arbitrar

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Benjamin Peterson
2013/8/24 Terry Reedy : > On 8/24/2013 8:51 AM, Stefan Behnel wrote: >> >> Antoine Pitrou, 24.08.2013 13:53: >>> >>> This would also imply extension module have to be subclasses of the >>> built-in module type. They can't be arbitrary objects like Stefan >>> proposed. I'm not sure what the latter e

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Terry Reedy
On 8/24/2013 8:51 AM, Stefan Behnel wrote: Antoine Pitrou, 24.08.2013 13:53: This would also imply extension module have to be subclasses of the built-in module type. They can't be arbitrary objects like Stefan proposed. I'm not sure what the latter enables, but it would probably make things mor

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Nick Coghlan, 24.08.2013 16:22: > On 24 August 2013 23:19, Stefan Behnel wrote: >> Nick Coghlan, 24.08.2013 13:36: >>> On 24 August 2013 15:51, Nick Coghlan wrote: My current plan is to create an experimental prototype of this approach this weekend. That will include stdlib test cases, so

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 24 August 2013 23:19, Stefan Behnel wrote: > Nick Coghlan, 24.08.2013 13:36: >> On 24 August 2013 15:51, Nick Coghlan wrote: >>> My current plan is to create an experimental prototype of this >>> approach this weekend. That will include stdlib test cases, so it will >>> also show how it looks f

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Nick Coghlan, 24.08.2013 13:36: > On 24 August 2013 15:51, Nick Coghlan wrote: >> My current plan is to create an experimental prototype of this >> approach this weekend. That will include stdlib test cases, so it will >> also show how it looks from the extension developer's point of view. > > I p

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Antoine Pitrou, 24.08.2013 15:00: > On Sat, 24 Aug 2013 14:51:42 +0200 > Stefan Behnel wrote: >> Antoine Pitrou, 24.08.2013 13:53: >>> This would also imply extension module have to be subclasses of the >>> built-in module type. They can't be arbitrary objects like Stefan >>> proposed. I'm not sure

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Antoine Pitrou
On Sat, 24 Aug 2013 14:51:42 +0200 Stefan Behnel wrote: > Antoine Pitrou, 24.08.2013 13:53: > > This would also imply extension module have to be subclasses of the > > built-in module type. They can't be arbitrary objects like Stefan > > proposed. I'm not sure what the latter enables, but it would

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Stefan Behnel
Antoine Pitrou, 24.08.2013 13:53: > This would also imply extension module have to be subclasses of the > built-in module type. They can't be arbitrary objects like Stefan > proposed. I'm not sure what the latter enables, but it would probably > make things more difficult internally. My line of th

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Antoine Pitrou
On Sat, 24 Aug 2013 21:36:51 +1000 Nick Coghlan wrote: > On 24 August 2013 15:51, Nick Coghlan wrote: > > My current plan is to create an experimental prototype of this > > approach this weekend. That will include stdlib test cases, so it will > > also show how it looks from the extension develop

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-24 Thread Nick Coghlan
On 24 August 2013 15:51, Nick Coghlan wrote: > My current plan is to create an experimental prototype of this > approach this weekend. That will include stdlib test cases, so it will > also show how it looks from the extension developer's point of view. I prototyped as much as I could without PEP

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-23 Thread Nick Coghlan
On 23 August 2013 19:18, Antoine Pitrou wrote: > > Hi, > > Le Fri, 23 Aug 2013 10:50:18 +0200, > Stefan Behnel a écrit : >> >> Here's an initial attempt at a PEP for it. It is based on the >> (unfinished) ModuleSpec PEP, which is being discussed on the >> import-sig mailing list. > > Thanks for t

Re: [Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-23 Thread Antoine Pitrou
Hi, Le Fri, 23 Aug 2013 10:50:18 +0200, Stefan Behnel a écrit : > > Here's an initial attempt at a PEP for it. It is based on the > (unfinished) ModuleSpec PEP, which is being discussed on the > import-sig mailing list. Thanks for trying this. I think the PEP should contain working example cod

[Python-Dev] Pre-PEP: Redesigning extension modules

2013-08-23 Thread Stefan Behnel
Hi, this has been subject to a couple of threads on python-dev already, for example: http://thread.gmane.org/gmane.comp.python.devel/135764/focus=140986 http://thread.gmane.org/gmane.comp.python.devel/141037/focus=141046 It originally came out of issues 13429 and 16392. http://bugs.python.org/