[Python-Dev] Re: Providing fix for modulefinder.py regression on Windows

[Barry Scott]

> On 12 Apr 2020, at 21:49, Eric V. Smith  wrote:
> 
> On 4/12/2020 4:08 PM, Barry Scott wrote:
>> 
>>> On 11 Apr 2020, at 16:28, Barry Scott >> > wrote:
>>> 
>>> modulefinder.py does not open source files in "rb" which
>>> prevents compile() from applying the encoding rules.
>>> 
>> snip
>> 
>> I have created the bpo, the pr and have the checks passing except that I 
>> need a NEWS.d entry
>> to make bedevere/news happy. I see the files have a special filename is that 
>> from an automatic
>> process of a tool I need to run?
>> 
>> Barry
>> 
> The tool is blurb (or the online blurb-it): 
> https://devguide.python.org/committing/#what-s-new-and-news-entries 
> 
> 
I was reading this https://devguide.python.org/#contributing 
 that does not mention blurb.

I would not have though to look in the section on Accepting Pull Requests, as 
that not what I'm doing.

Barry

> Eric
> 

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H6B6QU3CIXQJCWB4ULFEOJ3XYAKZHK75/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 11Apr2020 0111, Victor Stinner wrote:

Steve: the use case is to debug very rare Python crashes (ex: once
every two months) of customers who fail to provide a reproducer. My
*expectation* is that a debug build should help to reproduce the bug
and/or provide more information when the bug happens. My motivation
for this feature is also to show that the bug is not on Python but in
third-party C extensions ;-)


I think your expectation is wrong. If a stack trace of the crash doesn't 
show that it belongs to the third party module (which most of the ones 
that are sent back on Windows indeed show), then you need more invasive 
tracing to show that the issue came from the module. Until we actually 
have opaque, non-static objects, that doesn't seem to be possible.


All you've done right now is enable new inconsistencies and potential 
issues when mixing debug and release builds. That just makes things 
harder to diagnose.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/66K6TAPXEDBGSZFJLRNZNKNPCHIPUAAY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 11Apr2020 1156, Rhodri James wrote:

On 10/04/2020 18:20, Victor Stinner wrote:

Note: Cython and cffi should be preferred to write new C extensions.
This PEP is about existing C extensions which cannot be rewritten with
Cython.


If this is true, the documentation on python.org needs a serious 
rewrite.  I am in the throes of writing a C extension, and using Cython 
or cffi never even crossed my mind.




Sorry you missed the first two sections: "Recommended third party tools" 
and "Creating extensions without third party tools".


https://docs.python.org/3/extending/index.html

If you have any suggestions on how to make this recommendation more 
obvious, please open an issue and describe what would have helped.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7QZOSBXNNVQV6LYX3DPKI2RLSJ2K7XRY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 11Apr2020 0025, Antoine Pitrou wrote:

On Fri, 10 Apr 2020 23:33:28 +0100
Steve Dower  wrote:

On 10Apr2020 2055, Antoine Pitrou wrote:

On Fri, 10 Apr 2020 19:20:00 +0200
Victor Stinner  wrote:


Note: Cython and cffi should be preferred to write new C extensions.
This PEP is about existing C extensions which cannot be rewritten with
Cython.


Using Cython does not make the C API irrelevant.  In some
applications, the C API has to be low-level enough for performance.
Whether the application is written in Cython or not.


It does to the code author.

The point here is that we want authors who insist on coding against the
C API to be aware that they have fewer compatibility guarantees [...]


Yeah, you missed the point of my comment here.  Cython *does* call into
the C API, and it's quite insistent on performance optimizations too.
Saying "just use Cython" doesn't make the C API unimportant - it just
hides it from your own sight.


It centralises the change. I have no problem giving Cython access to 
things that we discourage every developer from using, provided they 
remain responsive to change and use the special access responsibly (e.g. 
by not touching reserved fields at all).


We could do a better job of helping them here.


**Backward compatibility:** backward incompatible on purpose. Break the
limited C API and the stable ABI, with the assumption that `Most C
extensions don't rely directly on CPython internals`_ and so will remain
compatible.


The problem here is not only compatibility but potential performance
regressions in C extensions.


I don't think we've ever guaranteed performance between releases.
Correctness, sure, but not performance.


That's a rather weird argument.  Just because you don't guarantee
performance doesn't mean it's ok to introduce performance regressions.

It's especially a weird argument to make when discussing a PEP where
most of the arguments are distant promises of improved performance.


If you've guaranteed compatibility but not performance, it means you can 
make changes that prioritise compatibility over performance.


If you promise to keep everything the same, you can never change 
anything. Arguing that everything is an implied contract between major 
version releases is the weird argument.



Fork and "Copy-on-Read" problem
...

Solve the "Copy on read" problem with fork: store reference counter
outside ``PyObject``.


Nowadays it is strongly recommended to use multiprocessing with the
"forkserver" start method:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

With "forkserver", the forked process is extremely lightweight and
there are little savings to be made in the child.


Unfortunately, a recommendation that only applies to a minority of
Python users. Oh well.


Which "minority" are you talking about?  Neither of us has numbers, but
I'm quite sure that the population of Python users calling into
multiprocessing (or a third-party library relying on multiprocessing,
such as Dask) is much larger than the population of Python users
calling fork() directly and relying on copy-on-write for optimization
purposes.

But if you have a different experience to share, please do so.


Neither Windows not macOS support fork (macOS only recently).

Break that down however you like, but by number of *developers* (as 
opposed to number of machines), and factoring in those who care about 
cross-platform compatibility, fork is not a viable thing to rely on.



Separating refcounts theoretically improves cache locality, specifically
the case where cache invalidation impacts multiple CPUs (and even the
case where a single thread moves between CPUs).


I'm a bit curious why it would improve, rather than degrade, cache
locality. If you take the typical example of the eval loop, an object
is incref'ed and decref'ed just about the same time that it gets used.


Two CPUs can read the contents of a string from their own cache. As soon 
as one touches the refcount, the cache line containing both the refcount 
and the string data in the other CPU is invalidated, and now it has to 
wait for synchronisation before reading the data.


If the refcounts are in a separate cache line, this synchronization 
doesn't have to happen.



I'll also note that the PEP proposes to remove APIs which return
borrowed references... yet increasing the number of cases where
accessing an object implies updating its refcount.


Yeah, I'm more okay with keeping borrowed references in some cases, but 
it does make things more complicated. Apparently some developers get it 
wrong consistently enough that we have to fix it? (ALL developers get it 
wrong during development ;) )



and this code
that they're using doesn't have any system dependencies that differ in
debug builds (spoiler: they do).


Are you talking about Windows?  On non-Windows systems, I don't think
there are "system dependencies that differ in debug builds".


Of course I'm talki

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Antoine Pitrou]

On Mon, 13 Apr 2020 11:35:34 +0100
Steve Dower  wrote:
> 
> Neither Windows not macOS support fork (macOS only recently).

Victor's argument: "fork() is not terrific with inline reference
counts".

My argument: people shouldn't generally use fork() anyway, because it
has other issues.

My statement that people should prefer "forkserver" was in that
context (if you are trying to build parallel applications using fork()
calls, think twice). Obviously on Windows you'll use the "spawn"
method, because it's the only available one ;-)  And on macOS, you'll
probably do whatever the latest recommended thing to do is ("spawn", I
suppose).

> >> Separating refcounts theoretically improves cache locality, specifically
> >> the case where cache invalidation impacts multiple CPUs (and even the
> >> case where a single thread moves between CPUs).  
> > 
> > I'm a bit curious why it would improve, rather than degrade, cache
> > locality. If you take the typical example of the eval loop, an object
> > is incref'ed and decref'ed just about the same time that it gets used.  
> 
> Two CPUs can read the contents of a string from their own cache. As soon 
> as one touches the refcount, the cache line containing both the refcount 
> and the string data in the other CPU is invalidated, and now it has to 
> wait for synchronisation before reading the data.

Ah, you're right.  However, the GIL should make such events less
frequent than in a language like C++.  Compared to the overhead of
look up reference counts in a different memory area (probably using a
non-trivial algorithm to determine the exact address), I'm not sure
which factor would dominate.

> >> and this code
> >> that they're using doesn't have any system dependencies that differ in
> >> debug builds (spoiler: they do).  
> > 
> > Are you talking about Windows?  On non-Windows systems, I don't think
> > there are "system dependencies that differ in debug builds".  
> 
> Of course I'm talking about Windows. I'm about the only person here who 
> does, and I'm having to represent at least half of our overall userbase 
> (look up my PyCon 2019 talk for the charts).

Ok :-)  However, Victor's point holds for non-Windows platforms, which
is *also* half of our userbase.

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OYM7W6QFVGGVP4EX32BEM37WB5O7DFIX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 13Apr2020 1157, Antoine Pitrou wrote:

On Mon, 13 Apr 2020 11:35:34 +0100
Steve Dower  wrote:

and this code
that they're using doesn't have any system dependencies that differ in
debug builds (spoiler: they do).


Are you talking about Windows?  On non-Windows systems, I don't think
there are "system dependencies that differ in debug builds".


Of course I'm talking about Windows. I'm about the only person here who
does, and I'm having to represent at least half of our overall userbase
(look up my PyCon 2019 talk for the charts).


Ok :-)  However, Victor's point holds for non-Windows platforms, which
is *also* half of our userbase.


True, though probably not the half sending him binary extension modules 
that nobody can rebuild ;)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CFJXEYIVPCJWCL27WN4BKEYN2RBKY3O5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 13Apr2020 1122, Steve Dower wrote:

On 11Apr2020 0111, Victor Stinner wrote:

Steve: the use case is to debug very rare Python crashes (ex: once
every two months) of customers who fail to provide a reproducer. My
*expectation* is that a debug build should help to reproduce the bug
and/or provide more information when the bug happens. My motivation
for this feature is also to show that the bug is not on Python but in
third-party C extensions ;-)


I think your expectation is wrong. If a stack trace of the crash doesn't 
show that it belongs to the third party module (which most of the ones 
that are sent back on Windows indeed show), then you need more invasive 
tracing to show that the issue came from the module. Until we actually 
have opaque, non-static objects, that doesn't seem to be possible.


All you've done right now is enable new inconsistencies and potential 
issues when mixing debug and release builds. That just makes things 
harder to diagnose.


I think what you really wanted to do here was have a build option 
_other_ than the debug flag to turn on additional checks. Like you did 
with tracemalloc.


The debug flag turns on additional runtime checks in the underlying C 
compiler and runtime on Windows (and I presume elsewhere? Is this such a 
crazy idea?), such as buffer overrun detection and memory misuse. The 
only way to make a debug build properly compatible with a release build 
is to disable these checks, which leaves us completely unable to take 
advantage of them. It also significantly speeds up compile time, which 
is very useful as a developer.


But if your goal is to have a release build that includes additional 
ABI-transparent checks, then I don't see why you wouldn't just build 
with those options? It's not like CPython takes that long to build from 
a clean working directory.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AY3QEYN6JEQFEZVJ2MUT5A2SJ5I72RAS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Paul Moore]

On Mon, 13 Apr 2020 at 11:20, Steve Dower  wrote:
>
> On 11Apr2020 1156, Rhodri James wrote:
> > On 10/04/2020 18:20, Victor Stinner wrote:
> >> Note: Cython and cffi should be preferred to write new C extensions.
> >> This PEP is about existing C extensions which cannot be rewritten with
> >> Cython.
> >
> > If this is true, the documentation on python.org needs a serious
> > rewrite.  I am in the throes of writing a C extension, and using Cython
> > or cffi never even crossed my mind.
> >
>
> Sorry you missed the first two sections: "Recommended third party tools"
> and "Creating extensions without third party tools".
>
> https://docs.python.org/3/extending/index.html
>
> If you have any suggestions on how to make this recommendation more
> obvious, please open an issue and describe what would have helped.

Personally, I'd say that "recommended 3rd party tools" reads as saying
"if you want a 3rd party tool to build extensions, these are good (and
are a lot easier than using the raw C API)". That's a lot different
than saying "we recommend that people writing C extensions do not use
the raw C API, but use one of these tools instead".

Also, if we *are* going to push people away from the raw C API, then I
think we should be recommending a particular tool (likely Cython) as
what people writing their first extension (or wanting to switch from
the raw C API for the first time) should use. Faced with the API docs,
and a list of 3rd party options, I know that *I* am likely to say
"yeah, leave that research for another day, I'll use what's in the
docs in front of me for now". Also, if we are expecting to push people
towards 3rd party tools, that seems to me to be a relatively
significant shift in emphasis, and one we should be publicising more
directly (via What's New, and blog postings / release announcements,
etc.) In the absence of anything like that, I think it's quite
reasonable for people to gravitate towards the traditional C API.

Having said all this, I *do* think that promoting some 3rd party tool
(as I say, I suspect this would be Cython) as the recommended means of
writing C extensions, is a reasonable approach to take. I just object
to it happening "quietly" via changes like this which make it harder
to use the raw C API, justifying themselves by saying "you shouldn't
do that anyway".

On a related but different note, what is the recommended policy
(assuming it's not to use the C API) for embedding Python, and for
exposing the embedding app to Python as a C extension? My standard
example of this is the Vim interface to Python - see
https://github.com/vim/vim/blob/master/src/if_python3.c. I originally
wrote this back in the Python 1.5 days, so it's *very* old, and quite
likely not how I'd write it now, even using the C API. But what's the
recommendation for code like that in the face of these changes, and
the suggestion that using 3rd party tools is the normal way to write C
extensions?

Paul
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TWTWVC4IHBNV2P4HCFFOQZ2WZKEONQHB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Rhodri James]

On 13/04/2020 11:17, Steve Dower wrote:

On 11Apr2020 1156, Rhodri James wrote:

On 10/04/2020 18:20, Victor Stinner wrote:

Note: Cython and cffi should be preferred to write new C extensions.
This PEP is about existing C extensions which cannot be rewritten with
Cython.


If this is true, the documentation on python.org needs a serious 
rewrite.  I am in the throes of writing a C extension, and using 
Cython or cffi never even crossed my mind.




Sorry you missed the first two sections: "Recommended third party tools" 
and "Creating extensions without third party tools".


https://docs.python.org/3/extending/index.html


"Creating extensions without third party tools" is what I read, because 
the preceding sections suggested to me that was what I was supposed to do.


The opening paragraph of the document more or less reads as "This is how 
you write C extensions."  It's an intro.  That's fair enough.


The next section, "Recommended third party tools", basically says "Third 
party tools exist."  It notably does not say "Use them in preference to 
what follows," so I didn't even look at them.  There is in fact a mild 
statement at the end of the first paragraph of the next section that 
should have clued me in, but I missed it because I'd already skipped to 
the table of contents.


If you have any suggestions on how to make this recommendation more 
obvious, please open an issue and describe what would have helped.


I'll give it some thought, but fundamentally if you want people to use 
the third party tools, you need a much stronger statement to that 
effect.  I'm sure I'm not the only one whose reaction to "third party" 
is "not official then".


--
Rhodri James *-* Kynesim Ltd
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4NDJSPJC7FKY7QGHLE3PVK4FGPMBW3EV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Fabio Zadrozny]

> * Hide implementation details from the C API to be able to `optimize
>   CPython`_ and make PyPy more efficient.
> * The expectation is that `most C extensions don't rely directly on
>   CPython internals`_ and so will remain compatible.
> * Continue to support old unmodified C extensions by continuing to
>   provide the fully compatible "regular" CPython runtime.
> * Provide a `new optimized CPython runtime`_ using the same CPython code
>   base: faster but can only import C extensions which don't use
>   implementation details. Since both CPython runtimes share the same
>   code base, features implemented in CPython will be available in both
>   runtimes.
>
>
Adding my 2cents from someone who does use the CPython API (for a debugger).

I must say I'm -1 until alternative APIs needed are available in the
optimized CPython runtime (I'd also say that this is a really big
incompatible change and would need a Python 4.0 to do)... I guess that in
order for this to work, the first step wouldn't be breaking everyone but
talking to extension authors (maybe checking for the users of the APIs
which will be deprecated) and seeing alternatives before pushing something
which will break CPython extensions which rely on such APIs.

I also don't think that CPython should have 2 runtimes... if the idea is to
leverage extensions to other CPython implementations, I think going just
for a more limited API is the way to go (but instead of just breaking
extensions that use the CPython internal API, try to come up with
alternative APIs for the users of the current CPython API -- for my use
case, I know the debugger could definitely do with just a few simple
additions: it uses the internal API mostly because there aren't real
alternatives for a couple of use cases). i.e.: if numpy/pandas/ doesn't adopt the optimized runtime because they don't have the
needed support they need, it won't be useful to have it in the first place
(you'd just be in the same place where other Python implementations already
are).

Also, this should probably follow the usual deprecation cycle: do a major
CPython release which warns about using the APIs that'll be deprecated and
only in the next CPython release should those APIs be actually removed (and
when that's done it probably deserves to be called Python 4).

Cheers,

Fabio
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IT6TQLDRII66K4T42NU2ZFTYOE6GYBRI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need port some extension modules to the modules directory?

[Eric Snow]

On Sun, Apr 12, 2020 at 8:36 AM Hai Shi  wrote:
> I notice some modules not in modules directory(for example: _warnings、marshal 
> in python directory). Do we need port those modules to modules directory?

FWIW, this is something I've been meaning to look at for a while and
is relatively high on my "back burner" [1] TODO list. :)  Not only do
we have some "extension" modules defined in Python/, but we also have
some core runtime functionality defined in the Modules/ directory
(e.g. in gcmodule.c).

In practice neither has been a problem over the years.  However, it
does make discovery a bit more complicated, both for new contributors
and for tools.  So I consider it at the least worth looking into the
value (and downsides, if any) of moving the seemingly out-of-place
pieces to the appropriate locations.

Anyway, I haven't looked at this super closely (and likely won't for a
while), so at this point the above is mostly just my gut reaction.
Take it for what it's worth. :)

-eric

[1] as in "on the back burner of the stove"
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SG5JEED3SZQHCC47I7BB5JMOSBQ2MJFF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 13Apr2020 1325, Paul Moore wrote:

Personally, I'd say that "recommended 3rd party tools" reads as saying
"if you want a 3rd party tool to build extensions, these are good (and
are a lot easier than using the raw C API)". That's a lot different
than saying "we recommend that people writing C extensions do not use
the raw C API, but use one of these tools instead".


Yeah, that's fair. But at the same time, saying anything more strong is 
an endorsement that we might have to withdraw at some point in the 
future (if the project we recommend implodes, for example).



Also, if we *are* going to push people away from the raw C API, then I
think we should be recommending a particular tool (likely Cython) as
what people writing their first extension (or wanting to switch from
the raw C API for the first time) should use. Faced with the API docs,
and a list of 3rd party options, I know that *I* am likely to say
"yeah, leave that research for another day, I'll use what's in the
docs in front of me for now". Also, if we are expecting to push people
towards 3rd party tools, that seems to me to be a relatively
significant shift in emphasis, and one we should be publicising more
directly (via What's New, and blog postings / release announcements,
etc.) In the absence of anything like that, I think it's quite
reasonable for people to gravitate towards the traditional C API.


Right, except we haven't decided to do it yet. There's still a debate 
about whether the current third party tools are even sufficient (not to 
mention what "sufficient" means).



Having said all this, I *do* think that promoting some 3rd party tool
(as I say, I suspect this would be Cython) as the recommended means of
writing C extensions, is a reasonable approach to take. I just object
to it happening "quietly" via changes like this which make it harder
to use the raw C API, justifying themselves by saying "you shouldn't
do that anyway".


Agreed, I'd rather be up front about it.


On a related but different note, what is the recommended policy
(assuming it's not to use the C API) for embedding Python, and for
exposing the embedding app to Python as a C extension? My standard
example of this is the Vim interface to Python - see
https://github.com/vim/vim/blob/master/src/if_python3.c. I originally
wrote this back in the Python 1.5 days, so it's *very* old, and quite
likely not how I'd write it now, even using the C API. But what's the
recommendation for code like that in the face of these changes, and
the suggestion that using 3rd party tools is the normal way to write C
extensions?


I don't think any current 3rd party tools really help with embedding (I 
say that as a regular embedder, not as someone who skim-read their 
docs). In this case, you really do need low-level access to Python's 
thread and memory management, and the ability to interact directly with 
the rest of your application's data structures.


PyBind11 is the best I've used here - Cython insists on including all 
its boilerplate to make a complete module, which often is not what you 
want. But there's a lot of core things that need to be improved if 
embedding is going to get any better, as I've posted often enough. We 
can't rely on third-party tools here, yet.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F6F6HQPSOEEQKLW2M6OQSSVMWXZHQ6Y3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need port some extension modules to the modules directory?

[Hai Shi]

Skip Montanaro wrote:
> > I notice some modules not in modules
> > directory(for example: _warnings、marshal in python directory). Do we need 
> > port those
> > modules to modules directory?
> > 
> > I strongly suspect the answer is "no." Modules which aren't in the
> Modules directory are built directly into the Python executable. Using
> your example of the _warnings module, note that in Makefile.pre.in
> Python/_warnings.o is listed in the PYTHON_OBJS list (as are
> Python/sysmodule.o and Python/marshal.o). This is evidence they are
> built directly into the interpreter itself. Another is that at runtime
> those modules have no __file__ attribute:
> > import marshal
> > marshal.__file__
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > AttributeError: module 'marshal' has no attribute '__file__'
> > import sys
> > sys.__file__
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > AttributeError: module 'sys' has no attribute '__file__'
> > import _warnings
> > _warnings.__file__
> > Traceback (most recent call last):
> >   File "", line 1, in 
> > AttributeError: module '_warnings' has no attribute '__file__'
> > Skip

Hi, skip,
Thank you for your replay. IMHO, it's because those module is 
builtinmodules, so they have no __file__ attribute and listed in PYTHON_OBJS(I 
am not sure I understand clearly). But I have no clear criteria what module 
should be putted in python directory or modules directory. I found some core 
developer have mentioned this detail 13 years 
ago(https://bugs.python.org/issue1631171).

Best Wishs,
Hai Shi
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XQD3KSBERANOR5VUFLCZALVUD3DCYKCQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Do we need port some extension modules to the modules directory?

[Hai Shi]

Eric Snow wrote:
> In practice neither has been a problem over the years.  However, it
> does make discovery a bit more complicated, both for new contributors
> and for tools.  So I consider it at the least worth looking into the
> value (and downsides, if any) of moving the seemingly out-of-place
> pieces to the appropriate locations.

Hi, Eric,
Thank you for your speedy reply. You put my thoughts into words ;)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZZR2OA36FGNANJCOGDTPWDICHC5NQHLJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Chris Meyer]

> On Apr 13, 2020, at 5:25 AM, Paul Moore  wrote:
> 
> On a related but different note, what is the recommended policy
> (assuming it's not to use the C API) for embedding Python, and for
> exposing the embedding app to Python as a C extension? My standard
> example of this is the Vim interface to Python - see
> https://github.com/vim/vim/blob/master/src/if_python3.c 
> . I originally
> wrote this back in the Python 1.5 days, so it's *very* old, and quite
> likely not how I'd write it now, even using the C API. But what's the
> recommendation for code like that in the face of these changes, and
> the suggestion that using 3rd party tools is the normal way to write C
> extensions?

I’d like to +1 this request for a standard for embedding Python while
at the same time exposing the embedding app to Python as a C extension. We do
something remarkably similar to Vim here (along with other files in the same 
directory).

https://github.com/nion-software/nionui-tool/blob/master/launcher/PythonStubs.cpp
 


I’ve looked into cffi but it seems to only solve a fraction of the problem. Our
Qt-based application embeds Python and provides callbacks from Python to our
application. It runs on macOS, Linux, and Windows and runs unchanged on
Python 3.6, 3.7, and 3.8 since it dynamically links to Python.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5RMFHARU7AY5TY7TKU356UTNHCLYBDYB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Eric Fahlgren]

On Mon, Apr 13, 2020 at 9:00 AM Steve Dower  wrote:

> On 13Apr2020 1325, Paul Moore wrote:
> > Personally, I'd say that "recommended 3rd party tools" reads as saying
> > "if you want a 3rd party tool to build extensions, these are good (and
> > are a lot easier than using the raw C API)". That's a lot different
> > than saying "we recommend that people writing C extensions do not use
> > the raw C API, but use one of these tools instead".
>
> Yeah, that's fair. But at the same time, saying anything more strong is
> an endorsement that we might have to withdraw at some point in the
> future (if the project we recommend implodes, for example).
>

Ok, so put that in a Pros/Cons list that provides guidance as to what
interface and tools to choose when writing a new extension module.
Personally, I'd put Cython (and other "big" packages, numpy, requests and
such) on par with CPython itself with respect to "likely to implode and
become unusable."
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VFRGMKVVE3OUJUXARNEIC67GFC6H22K7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Daniel Holth]

Was it regular cffi or cffi's embedding API, which is used a bit
differently than regular cffi, that "seems to only solve a fraction of the
problem"? Was just playing around with the embedding API and was impressed.

In Python:

@ffi.def_extern()
def uwsgi_pyexample_init():
print("init called")

return 0

In C (embedded in the same plugin):

CFFI_DLLEXPORT struct uwsgi_plugin pyexample_plugin = {
.init = uwsgi_pyexample_init
};

Seems to be happily importing and exporting APIs. Interpreter starts the
first time a @ffi.def_extern() function is called.

https://cffi.readthedocs.io/en/latest/embedding.html

https://github.com/unbit/uwsgi/blob/f6ad0c6dfe431d91ffe365bed3105ed052bef6e4/plugins/pyexample/pyexample_plugin.py
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S5P524LXXNIXIB2MUATZNHYQE57MXCQ5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Daniel Holth]

It can be done exactly as passing a void* when registering a C callback,
and getting it passed back to your callback function.
https://cffi.readthedocs.io/en/latest/ref.html#ffi-new-handle-ffi-from-handle

https://bitbucket.org/dholth/kivyjoy/src/aaeab79b2891782209a1219cd65a4d9716cea669/kivyjoy/controller.py#lines-49
https://bitbucket.org/dholth/kivyjoy/src/aaeab79b2891782209a1219cd65a4d9716cea669/kivyjoy/__init__.py#lines-15
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QDHQANZJAKPTTOJHXLGFSJWKLE5NA6ZU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Chris Meyer]

> On Apr 13, 2020, at 11:26 AM, Daniel Holth  wrote:
> 
> Was it regular cffi or cffi's embedding API, which is used a bit differently 
> than regular cffi, that "seems to only solve a fraction of the problem"? Was 
> just playing around with the embedding API and was impressed.
> 
> In Python:
> 
> @ffi.def_extern()
> def uwsgi_pyexample_init():
> print("init called")
> 
> return 0
> 
> In C (embedded in the same plugin):
> 
> CFFI_DLLEXPORT struct uwsgi_plugin pyexample_plugin = {
> .init = uwsgi_pyexample_init
> };
> 
> Seems to be happily importing and exporting APIs. Interpreter starts the 
> first time a @ffi.def_extern() function is called.
> 
> https://cffi.readthedocs.io/en/latest/embedding.html 
> 
> 
> https://github.com/unbit/uwsgi/blob/f6ad0c6dfe431d91ffe365bed3105ed052bef6e4/plugins/pyexample/pyexample_plugin.py
>  
> 
I might need to understand cffi embedding more to really answer your question - 
and it’s entirely possible cffi can do this - but as a simple example:

How would I call a Python function from the C++ application that returns a 
Python object to C++ and then call a method on that Python object from C++?

My specific example is that I create Python handlers for Qt windows and then 
from the Qt/C++ I call methods on those Python objects from C++ such as “handle 
mouse event”.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6PZ7OJSPXJVQ2BLZWGOOMWZN6EINLQAV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Daniel Holth]

Sorry that this is a bit off-topic. cffi would be a user of any new C API.

I've tried to make sure ABI3 is supported in setuptools and wheel, with
varying success. Apparently virtualenvs and Windows have problems. I'm
excited about the possibility of a better C API and possibly ABI.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XFHKIXSEEZMCC7HZ3N4GP6SBUBYACI6K/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 13Apr2020 2105, Chris Meyer wrote:
How would I call a Python function from the C++ application that returns 
a Python object to C++ and then call a method on that Python object from 
C++?


My specific example is that I create Python handlers for Qt windows and 
then from the Qt/C++ I call methods on those Python objects from C++ 
such as “handle mouse event”.


You're in a bit of trouble here regardless, depending on how robust you 
need to be. If you've only got synchronous, single-threaded event 
handlers then you'll be okay. Anything more complex and you'll have some 
fun debugging sessions to look forward to.


I would definitely say look at PyBind11. A while ago I posted a sample 
using this to embed Python in a game engine at 
https://devblogs.microsoft.com/python/embedding-python-in-a-cpp-project-with-visual-studio/ 
(VS is not required, it just happened to be the hook to do the 
post/video ;) )


To jump straight to the code, go to 
https://github.com/zooba/ogre3d-python-embed/blob/master/src/PythonCharacter.cpp 
and search for "py::", and also 
https://github.com/zooba/ogre3d-python-embed/blob/master/src/ogre_module.h


PyBind11 is nice for avoiding the boilerplate and ref-counting, but has 
its own set of obscure error cases. It's also not as easy to debug as 
Cython or going straight to the Python C API, depending on what the 
issue is, as there's no straightforward generated code. Even stepping 
through the templated code interactively in VS doesn't help make it any 
easier to follow.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GVXH2AJC7Z2F5AIBIMCXEDKXLEYVCCU4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[André Malo]

Steve Dower wrote:
> On 11Apr2020 0025, Antoine Pitrou wrote:
> > On Fri, 10 Apr 2020 23:33:28 +0100
> > 
> > Steve Dower  wrote:
> >> On 10Apr2020 2055, Antoine Pitrou wrote:
> >>> On Fri, 10 Apr 2020 19:20:00 +0200
> >>> 
> >>> Victor Stinner  wrote:
>  Note: Cython and cffi should be preferred to write new C extensions.
>  This PEP is about existing C extensions which cannot be rewritten with
>  Cython.
> >>> 
> >>> Using Cython does not make the C API irrelevant.  In some
> >>> applications, the C API has to be low-level enough for performance.
> >>> Whether the application is written in Cython or not.
> >> 
> >> It does to the code author.
> >> 
> >> The point here is that we want authors who insist on coding against the
> >> C API to be aware that they have fewer compatibility guarantees [...]
> > 
> > Yeah, you missed the point of my comment here.  Cython *does* call into
> > the C API, and it's quite insistent on performance optimizations too.
> > Saying "just use Cython" doesn't make the C API unimportant - it just
> > hides it from your own sight.
> 
> It centralises the change. I have no problem giving Cython access to
> things that we discourage every developer from using, provided they
> remain responsive to change and use the special access responsibly (e.g.
> by not touching reserved fields at all).

It appears to me that this whole line of argument is contradicting the purpose 
of the whole idea. What am I missing?

For one thing, if you open up APIs for Cython, they're open for everybody 
(Cython being "just" another C extension).
More to the point: The ABIs have the same problem as they have now, regardless 
how responsive the Cython developers are. Once you compiled the extension, 
you're using the ABI and are supposedly not required to recompile to stay 
compatible.

So, where I'm getting at is: Either you open up to everybody or nobody. In C 
there's not really an in-between.

Cheers,
nd

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/S2BRL2GJGRNK5WXCSZPNLLMN4LGA5KTN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP: Modify the C API to hide implementation details

[Steve Dower]

On 13Apr2020 2308, André Malo wrote:

For one thing, if you open up APIs for Cython, they're open for everybody
(Cython being "just" another C extension).
More to the point: The ABIs have the same problem as they have now, regardless
how responsive the Cython developers are. Once you compiled the extension,
you're using the ABI and are supposedly not required to recompile to stay
compatible.

So, where I'm getting at is: Either you open up to everybody or nobody. In C
there's not really an in-between.


On a technical level, you are correct.

On a policy level, we don't make changes that would break users of the C 
API. Because we can't track everyone who's using it, we have to assume 
that everything is used and any change will cause breakage.


To make sure it's possible to keep developing CPython, we declare parts 
of the API off limits (typically by prepending them with an underscore). 
If you use these, and you break, we're sorry but we aren't going to fix it.


This line of discussion is basically saying that we would designate a 
broader section of the API that is off limits, most likely the parts 
that are only useful for increased performance (rather than increased 
functionality). We would then specifically include the Cython 
team/volunteers in discussions about how to manage changes to these 
parts of the API to avoid breaking them, and possibly do simultaneous 
releases to account for changes so that their users have more time to 
rebuild.


Effectively, when we change our APIs, we would break everyone except 
Cython because we've worked with them to avoid the breakage. Anyone else 
using it has to make their own effort to follow CPython development and 
detect any breakage themselves (just like today).


So probably the part you're missing is where we would give ourselves 
permission to break more APIs in a release, while simultaneously 
encouraging people to use Cython as an isolation layer from those breaks.


(Cython is still just a placeholder name here, btw. There are 1-2 other 
projects that could be considered instead, though I think Cython is the 
only one that also provides a usability improvement as well as API 
stability.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BPFQKMXTMVVSFVFEAJRXAPVQEZE3HMFN/
Code of Conduct: http://python.org/psf/codeofconduct/