Re: [Python-Dev] [python-committers] Winding down 3.4

2018-08-20 Thread Victor Stinner
> "shutil copy* unsafe on POSIX - they preserve setuid/setgit bits" > https://bugs.python.org/issue17180 There is no fix. A fix may break the backward compatibility. Is it really worth it for the last 3.4 release? > "XML vulnerabilities in Python" > https://bugs.python.org/issue17239 Bug inactiv

Re: [Python-Dev] Python 2.7 EOL date

2018-08-23 Thread Victor Stinner
Hi, The reference is the PEP 373 "Python 2.7 Release Schedule". See the "Update" section: https://www.python.org/dev/peps/pep-0373/#update Victor 2018-08-23 20:53 GMT+02:00 Collin Anderson : > Hi All, > > Sorry if this has been mentioned before, but I noticed the Python 2.7 EOL > date was recent

Re: [Python-Dev] [Python-checkins] bpo-34171: Prevent creating Lib/trace.cover when run the trace module. (GH-8841)

2018-08-27 Thread Victor Stinner
Jemery: would you mind to revert this commit since nobody fixed the buildbot since 2 days? https://pythondev.readthedocs.io/ci.html#revert-on-fail Victor Le sam. 25 août 2018 à 22:50, Jeremy Kloth a écrit : > > On Sat, Aug 25, 2018 at 1:28 AM Serhiy Storchaka > wrote: > > > > https://github.com/

[Python-Dev] UTF-8 Mode now also enabled by the POSIX locale

2018-08-28 Thread Victor Stinner
Hi, While working on test_utf8_mode on AIX (bpo-34347) and HP-UX (bpo-34403), I noticed that FreeBSD doesn't work properly with the POSIX locale (bpo-34527). I also noticed that my implementation of my PEP 540 "UTF-8 Mode" doesn't respect the PEP: the UTF-8 Mode should be enabled by the POSIX loca

[Python-Dev] Workflow blocked on the 3.6 because of AppVeyor; who owns the AppVeyor project?

2018-09-05 Thread Victor Stinner
Hi, It's no longer possible to merge any change in the 3.6 branch of CPython, because the AppVeyor job fails: https://bugs.python.org/issue34575 It seems like AppVeyor has a build cache and this cache is outdated. I tried to use the REST API but I'm not allowed to invalidate the cache: even the m

[Python-Dev] Schedule of the Python 3.7.1 release?

2018-09-05 Thread Victor Stinner
Hi, Someone asked somewhere (oops, I forgot where!) when is Python 3.7.1 scheduled. I wanted to reply when I saw that it was scheduled for 2 months ago: "3.7.1: 2018-07-xx" https://www.python.org/dev/peps/pep-0537/#maintenance-releases Is there any blocker for 3.7.1? I fixed dozens of bugs in th

Re: [Python-Dev] Workflow blocked on the 3.6 because of AppVeyor; who owns the AppVeyor project?

2018-09-05 Thread Victor Stinner
I wrote some notes about our CIs. Link to AppVeyor notes: https://pythondev.readthedocs.io/ci.html#appveyor Victor Le mer. 5 sept. 2018 à 12:04, Paul Moore a écrit : > > On Wed, 5 Sep 2018 at 10:55, Victor Stinner wrote: > > > > Hi, > > > > It's no longer poss

Re: [Python-Dev] Workflow blocked on the 3.6 because of AppVeyor; who owns the AppVeyor project?

2018-09-05 Thread Victor Stinner
Le mer. 5 sept. 2018 à 15:47, Zachary Ware a écrit : > For the actual issue at hand, the problem arises from doing builds on > 3.6 with both the VS2015 and VS2017 images. Apparently something > built in `/externals` by the VS2015 build gets cached, which then > breaks the VS2017 build; I haven't

[Python-Dev] We cannot fix all issues: let's close XML security issues (not fix them)

2018-09-06 Thread Victor Stinner
Hi, The Python bug tracker is full of bugs, and sadly we don't have enough people to take care of all of them. There are 3 open bugs about security issues in XML and I simply propose to close it: https://bugs.python.org/issue17318 https://bugs.python.org/issue17239 https://bugs.python.or

Re: [Python-Dev] We cannot fix all issues: let's close XML security issues (not fix them)

2018-09-06 Thread Victor Stinner
Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou a écrit : > If we consider fixing these issues to be desirable, then the issues > should be kept open. Closing issues because no-one is working on them > sounds a bit silly to me. I forgot to mention that closing these issues is my reply to Larry's ca

Re: [Python-Dev] We cannot fix all issues: let's close XML security issues (not fix them)

2018-09-06 Thread Victor Stinner
Are you volunteer to fix the XML modules? Victor Le jeu. 6 sept. 2018 à 16:50, Antoine Pitrou a écrit : > > > Le 06/09/2018 à 16:40, Victor Stinner a écrit : > > Le jeu. 6 sept. 2018 à 16:33, Antoine Pitrou a écrit : > >> If we consider fixing these issues to be de

Re: [Python-Dev] We cannot fix all issues: let's close XML security issues (not fix them)

2018-09-07 Thread Victor Stinner
Le jeu. 6 sept. 2018 à 21:10, Steve Dower a écrit : > If Christian is not able to keep maintaining the defused* packages, then > I may take a look at this next week at the sprints. The built-in XML > packages actually don't meet Microsoft's internal security requirements, > so I have some business

Re: [Python-Dev] We cannot fix all issues: let's close XML security issues (not fix them)

2018-09-07 Thread Victor Stinner
Le ven. 7 sept. 2018 à 10:23, Christian Heimes a écrit : > Back in the days, I didn't push hard for the necessary fixes, because > all fixes were breaking changes. After all I'd have to disable some > features that people may have relied upon. The XML security stuff was my > first major security t

Re: [Python-Dev] Fwd: We cannot fix all issues: let's close XML security issues (not fix them)

2018-09-07 Thread Victor Stinner
Le ven. 7 sept. 2018 à 17:02, PMS PMS a écrit : > XML support in Python is critical and desired for many sectors like banking > or telecoms, > and code base based on XML is still on rise in such world. Would it be possible to send money to the PSF? I'm sure that the PSF will be able to find you

[Python-Dev] bpo-34595: How to format a type name?

2018-09-11 Thread Victor Stinner
Hi, Last week, I opened an issue to propose to add a new %T formatter to PyUnicode_FromFormatV() and so indirectly to PyUnicode_FromFormat() and PyErr_Format(): https://bugs.python.org/issue34595 I merged my change, but then Serhiy Storchaka asked if we can add something to get the "fully qua

Re: [Python-Dev] Stop automerging

2018-09-12 Thread Victor Stinner
Hi Benjamin, https://github.com/python/cpython/commit/d13e59c1b512069d90efe7ee9b613d3913e79c56 Le mer. 12 sept. 2018 à 17:19, Benjamin Peterson a écrit : > (Just checking) Is there something wrong with this message besides the <-- > comment? Since the commit is described as a follow-up of 90fc

Re: [Python-Dev] Stop automerging

2018-09-12 Thread Victor Stinner
t. Victor Le mer. 12 sept. 2018 à 17:56, Victor Stinner a écrit : > > Hi Benjamin, > > https://github.com/python/cpython/commit/d13e59c1b512069d90efe7ee9b613d3913e79c56 > > Le mer. 12 sept. 2018 à 17:19, Benjamin Peterson a > écrit : > > (Just checking) Is there somethin

Re: [Python-Dev] bpo-34595: How to format a type name?

2018-09-12 Thread Victor Stinner
Hi, For the type name, sometimes, we only get a type (not an instance), and we want to format its FQN. IMHO we need to provide ways to format the FQN of a type for *types* and for *instances*. Here is my proposal: * Add !t conversion to format string * Add ":T" format to type.__format__() * Add "

Re: [Python-Dev] bpo-34595: How to format a type name?

2018-09-13 Thread Victor Stinner
Le jeu. 13 sept. 2018 à 16:01, Eric V. Smith a écrit : > > * Add !t conversion to format string > > I'm strongly opposed to this. This !t conversion would not be widely > applicable enough to be generally useful, and would need to be exposed > in the f-string and str.format() documentation, even t

Re: [Python-Dev] bpo-34595: How to format a type name?

2018-09-13 Thread Victor Stinner
Le ven. 14 sept. 2018 à 00:09, Eric V. Smith a écrit : > f'{type(obj)}' becomes type(obj).__format__(''), so you can return > something other than __str__ or __repr__ does. It's only by convention > that an object's __format__ returns __str__: it need not do so. What's New in Python 3.7 contains:

[Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation

2018-09-17 Thread Victor Stinner
Hi Unicode and locales lovers, tl; dr Nick, Ned, INADA-san: I modified 3.7.1 to add a new "-X coerce_c_locale=value" option and make sure that the C locale coercion cannot be when Python in embedded: are you ok with these changes? Before 3.7.0 release, during the implementation of the UTF-8 Mode

Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation

2018-09-19 Thread Victor Stinner
Le mer. 19 sept. 2018 à 09:50, Nick Coghlan a écrit : > I think the changes to both master and the 3.7 branch should be reverted. Ok, I prepared a PR to revert the 3.7 change: https://github.com/python/cpython/pull/9416 > For 3.7, I already said that I think we should just accept that that > sh

Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation

2018-09-19 Thread Victor Stinner
> IMHO the implementation is really a secondary concern here, the main > question is: what is the correct behavior? > > Nick: > > * Do we agree that we need to provide a way to disable C locale > coercion (PEP 538) even when -E is used? > * Do you agree that Py_Initialize() and Py_Main() must not e

Re: [Python-Dev] Late Python 3.7.1 changes to fix the C locale coercion (PEP 538) implementation

2018-09-19 Thread Victor Stinner
Le mardi 18 septembre 2018, Victor Stinner a écrit : > Hi Unicode and locales lovers, > > tl; dr Nick, Ned, INADA-san: I modified 3.7.1 to add a new "-X > coerce_c_locale=value" option and make sure that the C locale coercion > cannot be when Python in embedded: are

Re: [Python-Dev] Questions about signal handling.

2018-09-21 Thread Victor Stinner
Le sam. 22 sept. 2018 à 01:05, Eric Snow a écrit : > 3. Why does signal handling operate via the "pending calls" machinery > and not distinctly? Signals can be received anytime, between two instructions at the machine code level. But the Python code base is rarely reentrant. Moreover, you can get

Re: [Python-Dev] Questions about signal handling.

2018-09-25 Thread Victor Stinner
Please don't rely on this ugly API. *By design*, Py_AddPendingCall() tries 100 times to acquire the lock: if it fails to acquire the lock, it does notthing... your callback is ignored... By the way, recently, we had to fix yet another bug in signal handling. A new function has been added: void _P

Re: [Python-Dev] Documenting the private C API (was Re: Questions about signal handling.)

2018-09-25 Thread Victor Stinner
directory.) Victor Le mar. 25 sept. 2018 à 16:05, Barry Warsaw a écrit : > > On Sep 25, 2018, at 03:44, Victor Stinner wrote: > > > By the way, recently, we had to fix yet another bug in signal > > handling. A new function has been added: > > > > voi

Re: [Python-Dev] What is the purpose of the _PyThreadState_Current symbol in Python 3?

2018-09-27 Thread Victor Stinner
Hi, Le mer. 26 sept. 2018 à 23:27, Gabriele a écrit : > In trying to find the location of a valid instance of PyInterpreterState in > the virtual memory of a running Python (3.6) application (using > process_vm_read on Linux), I understand that you are writing a debugger and you can only *read

[Python-Dev] Communication channels

2018-10-01 Thread Victor Stinner
Hi, Last months, new communication channels appear. This is just a reminder that they exist: * Zulip: https://python.zulipchat.com/ (exist since 1 year?) * Discourse: http://discuss.python.org/ (I'm not sure if it's fully official yet ;-)) * IRC: #python-dev on FreeNode, only for development *of*

Re: [Python-Dev] Communication channels

2018-10-01 Thread Victor Stinner
Le lun. 1 oct. 2018 à 14:14, Chris Jerdonek a écrit : > Also, if you are trying to be complete, another communication channel > is in-person events and conferences, etc. Right, there are two main events for CPython core developers which are only in the US: * Language Summit during Pycon US (one

Re: [Python-Dev] Communication channels

2018-10-01 Thread Victor Stinner
Le lun. 1 oct. 2018 à 22:32, Tres Seaver a écrit : > I'm pretty strongly -1 on the notion that folks who subscribe python-dev, > BPO, and the github repositories should need to *also* follow an > arbitrarily-growing set of Twitter accounts: how would one know if a new > one popped into being? Ho

Re: [Python-Dev] dear core-devs

2018-10-04 Thread Victor Stinner
Hi, If IBM wants a better Python support, it would help a lot if IBM pays for this development. With money, you can easily find core dev contractors. Antoine Pitrou has been paid in the past to enhance Python support in Solaris and it worked well. Victor Le mercredi 3 octobre 2018, Michael Felt

Re: [Python-Dev] Some PRs to merge?

2018-10-19 Thread Victor Stinner
Le ven. 19 oct. 2018 à 19:01, Stephane Wirtel a écrit : > total: 49 PRs > is:open is:pr review:approved status:success label:"awaiting merge" > -label:"DO-NOT-MERGE" label:""LA signed"" I merged many PRs and closed a few (2 if I recall correctly). Your query now counts 24 PRs. Victor __

Re: [Python-Dev] Some PRs to merge?

2018-10-22 Thread Victor Stinner
Le sam. 20 oct. 2018 à 13:15, Serhiy Storchaka a écrit : > Thank you Victor! I prefer to merge my PRs and PRs assigned to me > myself, but I am not sure that I would merge all PRs that can be merged > in the nearest future. ;) Some PRs were blocked by me because I was nitpicking on something. I d

Re: [Python-Dev] The future of the wchar_t cache

2018-10-22 Thread Victor Stinner
Hi Serhiy, +1 to remove wchar_t cache. IMHO it wastes memory for no real performance gain. By the way, can we start to schedule the *removal* of the Py_UNICODE API? For example, decide when Py_DEPRECATED is used in the C API? Should we start to deprecate when Python 2 reachs its end of life? Or c

Re: [Python-Dev] The future of the wchar_t cache

2018-10-22 Thread Victor Stinner
Le sam. 20 oct. 2018 à 18:02, Steve Dower a écrit : > I don't have numbers, but my instinct says the most impacted operations > would be retrieving collections of strings from the OS (avoiding a > scan/conversion for each one), comparisons against these collections > (in-memory handling for hash/c

Re: [Python-Dev] The future of the wchar_t cache

2018-10-22 Thread Victor Stinner
Le lun. 22 oct. 2018 à 15:08, Steve Dower a écrit : > Agreed the cache is useless here, but since the listdir() result came in > as wchar_t we could keep it that way (assuming we'd only be changing it > to char), and then there wouldn't have to be a conversion when we > immediately pass it back to

Re: [Python-Dev] The future of the wchar_t cache

2018-10-22 Thread Victor Stinner
Le lun. 22 oct. 2018 à 15:24, Steve Dower a écrit : > Yes, that's true. But "should reduce ... footprint" is also an > optimisation that deserves a benchmark by that standard. pyperformance has a mode to mesure the memory usage (mostly the memory peak) if someone wants to have a look. > Also, I'

[Python-Dev] Python Language Governance Proposals

2018-10-23 Thread Victor Stinner
Hi, Last July, Guido van Rossum decided to resign from his role of BDFL. Python core developers decided to design a new governance/organization for Python. 6 governance PEPs have been proposed. It has been decided that discussions are reserved to core developers (everyone can read, but only core d

Re: [Python-Dev] ​Re: Python Language Governance Proposals

2018-10-23 Thread Victor Stinner
Hi, Le mar. 23 oct. 2018 à 18:26, eamanu15 a écrit : > Are there any list of candidates? No PEP proposes directly candidates. The process to nominate candidates is part of discussed PEPs. So first core developers must vote for the governance PEP, and then a new process will be organized to selec

Re: [Python-Dev] "Deprecation" of os.system in favor of subprocess?

2018-10-24 Thread Victor Stinner
I like os.system() and use it everyday. I prefer to write os.system("grep ^Vm /proc/%s/status" % os.getpid()) than... much more Python code to do the same thing, or subprocess.run("grep ^Vm /proc/%s/status" % os.getpid(), shell=True): os.system() is shorter and only requires "import os" :-) I'm n

[Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-28 Thread Victor Stinner
Hi, Python C API has no strict separation between the 3 levels of API: * core: Py_BUILD_CORE define * stable: Py_LIMITED_API define (it has a value) * regular: no define IMHO the current design of header files is done backward: by default, everything is public. To exclude an API from core or sta

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-28 Thread Victor Stinner
. 2018 à 17:20, Victor Stinner a écrit : > > Hi, > > Python C API has no strict separation between the 3 levels of API: > > * core: Py_BUILD_CORE define > * stable: Py_LIMITED_API define (it has a value) > * regular: no define > > IMHO the current design of header file

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-28 Thread Victor Stinner
Le dim. 28 oct. 2018 à 21:50, Benjamin Peterson a écrit : > I don't think more or less API should be magically included based on whether > Py_BUILD_CORE is defined or not. If we want to have private headers, we > should include them where needed and not install them. Really, Py_BUILD_CORE > sho

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-29 Thread Victor Stinner
Le lun. 29 oct. 2018 à 06:32, Benjamin Peterson a écrit : > > My overall approach is to make sure that we don't leak functions by > > mistakes into the public API or into the stable API anymore. For > > example, if a function is really for the core, put it in pycore/. It > > will be more explicit

Re: [Python-Dev] Standardize error message for non-pickleable types

2018-10-29 Thread Victor Stinner
Le lun. 29 oct. 2018 à 20:42, Serhiy Storchaka a écrit : > 1. "pickle" or "serialize"? serialize > 2. "can't", "Cannot", "can not" or "cannot"? cannot > 3. "object" or "objects"? object > 4. Use the "a" article or not? no: "cannot serialize xxx object" (but i'm not a native english speaker,

Re: [Python-Dev] Standardize error message for non-pickleable types

2018-10-30 Thread Victor Stinner
Le lun. 29 oct. 2018 à 22:20, MRAB a écrit : > 1. If you're pickling, then saying "pickle" is more helpful. I'm not sure that it's really possible to know if the error occurs while pickle is trying to serialize an object, or if it's a different serialization protocol. We are talking about the ver

Re: [Python-Dev] Custom AIX build last week...

2018-10-30 Thread Victor Stinner
I used a custom build to check if a fix repaired a buildbot. It's unrelated to AIX, but inlining on Visual Studio in Debug mode, so specific to Windows. Victor Le mar. 30 oct. 2018 à 07:11, Michael Felt a écrit : > > I noticed that there was a "custom" build queued for my AIX build-bot last > we

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-30 Thread Victor Stinner
Le mar. 30 oct. 2018 à 02:59, Benjamin Peterson a écrit : > > To me, it seems wrong that a function or macro defined in > > Include/objimpl.h requires an explicit #include "internal/pystate.h". > > objimpl.h should be self-sufficient. > > I agree. I would say nothing in Include/*.h should be inclu

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-31 Thread Victor Stinner
Le mer. 31 oct. 2018 à 19:22, Eric Snow a écrit : > > I propose a practical solution for that: Include/*.h files would only > > be be public API. > > As we've already discussed, I'm entirely in favor of this. :) In > fact, I was thinking along those same lines when I originally created > "Include

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-31 Thread Victor Stinner
Ok, I proposed to rename internal header files, add an "internal_" prefix, to avoid this issue: https://github.com/python/cpython/pull/10263 My PR also adds Include/internal/ to search paths of header files. Extract of the change: diff --git a/Include/internal/pystate.h b/Include/internal/intern

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-31 Thread Victor Stinner
Le mer. 31 oct. 2018 à 20:40, Eric Snow a écrit : > On Mon, Oct 29, 2018 at 6:39 AM Victor Stinner wrote: > > Le lun. 29 oct. 2018 à 06:32, Benjamin Peterson a > > écrit : > > > How does the current Include/internal/ directory fail at accomplishing > > &g

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-31 Thread Victor Stinner
Le mer. 31 oct. 2018 à 22:19, Eric Snow a écrit : > > Maybe we can keep "Include/internal/" directory name, but add > > "pycore_" rather than "internal_" to header filenames? > > this sounds good to me. thanks for chasing this down. What do you prefer? (A Include/internal/internal_pystate.h (B)

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-31 Thread Victor Stinner
" with #include "pycore_mem.h". Victor Le mer. 31 oct. 2018 à 23:38, Eric Snow a écrit : > > B > On Wed, Oct 31, 2018 at 4:28 PM Victor Stinner wrote: > > > > Le mer. 31 oct. 2018 à 22:19, Eric Snow a > > écrit : > > > > Maybe we can keep &q

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-10-31 Thread Victor Stinner
I pushed many changes. I moved most of the Py_BUILD_CORE code out of Include/ header files. Le jeu. 1 nov. 2018 à 02:35, Eric Snow a écrit : > On the one hand dropping redundancy in the filename is fine. On the other > having the names mirror the public header files is valuable. Current conten

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-11-01 Thread Victor Stinner
Le jeu. 1 nov. 2018 à 16:38, Steve Dower a écrit : > I assume the redundancy was there to handle the naming collision > problem, but a better way to do that is to have only header files that > users should ever use in "include/", and put the rest inside > "include/python/" and "include/internal/"

Re: [Python-Dev] Rename Include/internals/ to Include/pycore/

2018-11-02 Thread Victor Stinner
Le ven. 2 nov. 2018 à 18:32, Neil Schemenauer a écrit : > A simple approach would be to introduce something like > Python-internal.h. If you are a Python internal unit, you can > include both Python.h and Python-internal.h. We could, over time, > split Python-iternal.h into smaller modular inclu

Re: [Python-Dev] Get a running instance of the doc for a PR.

2018-11-04 Thread Victor Stinner
OpenStack does that on review.openstack.org PRs. If I recall correctly, the CI produces files which are online on a static web server. Nothing crazy. And it works. Old files are removed, I don't know when exactly. I don't think that it matters where the static files are hosted. Victor Le dimanch

[Python-Dev] What is the difference between Py_BUILD_CORE and Py_BUILD_CORE_BUILTIN?

2018-11-06 Thread Victor Stinner
Hi, I'm trying to cleanup the Python C API, especially strictly separate the public C API, the stable C API (Py_LIMITED_API), and the internal C API (Py_BUILD_CORE). Move internal headers to Include/internal/ : https://bugs.python.org/issue35081 Move !Py_LIMITED_API to Include/pycapi/: https://b

[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-09 Thread Victor Stinner
Hi, The current C API of Python is both a strength and a weakness of the Python ecosystem as a whole. It's a strength because it allows to quickly reuse a huge number of existing libraries by writing a glue for them. It made numpy possible and this project is a big sucess! It's a weakness because

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-09 Thread Victor Stinner
To hide all implementation details, I propose to stop using macros and use function calls instead. For example, replace: #define PyTuple_GET_ITEM(op, i) \ (((PyTupleObject *)(op))->ob_item[i]) with: # define PyTuple_GET_ITEM(op, i) PyTuple_GetItem(op, i) With this change, C extensions using

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-09 Thread Victor Stinner
Le sam. 10 nov. 2018 à 01:49, Michael Selik a écrit : >> It uses an opt-in option (Py_NEWCAPI define -- I'm not sure about the >> name) to get the new API. The current C API is unchanged. > > While one can hope that this will be the only time the C API will be revised, > it may be better to numbe

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-09 Thread Victor Stinner
Le sam. 10 nov. 2018 à 02:50, Nathaniel Smith a écrit : > Doesn't this mean that you're just making the C API larger and more > complicated, rather than simplifying it? You cite some benefits > (tagged pointers, changing the layout of PyObject, making PyPy's life > easier), but I don't see how you

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-09 Thread Victor Stinner
implementation checks arguments at runtime if compiled in debug mode. Technically, the header file still uses a macro, to implicitly cast to PyObject*, since currently the macro accepts any type, and the new C API should not change that. Victor Le sam. 10 nov. 2018 à 01:53, Victor Stinner a écrit : > &

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-11 Thread Victor Stinner
Le sam. 10 nov. 2018 à 04:02, Nathaniel Smith a écrit : > So is it fair to say that your plan is that CPython will always use > the current ("old") API internally, and the "new" API will be > essentially an abstraction layer, that's designed to let people write > C extensions that target the old A

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-13 Thread Victor Stinner
Le mar. 13 nov. 2018 à 08:13, Gregory P. Smith a écrit : > When things have only ever been macros (Py_INCREF, etc) the name can be > reused if there has never been a function of that name in an old C API. But > beware of reuse for anything where the semantics change to avoid > misunderstanding

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-13 Thread Victor Stinner
Le mar. 13 nov. 2018 à 20:32, André Malo a écrit : > As long as they are recompiled. However, they will lose a lot of performance. > Both these points have been mentioned somewhere, I'm certain, but it cannot be > stressed enough, IMHO. Somewhere is here: https://pythoncapi.readthedocs.io/perform

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-14 Thread Victor Stinner
Le mer. 14 nov. 2018 à 03:24, Nathaniel Smith a écrit : > So I think what you're saying is that your goal is to get a > new/better/shinier VM, and the plan to accomplish that is: > > 1. Define a new C API. > 2. Migrate projects to the new C API. > 3. Build a new VM that gets benefits from only sup

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-14 Thread Victor Stinner
Le mer. 14 nov. 2018 à 11:24, Antoine Pitrou a écrit : > For example in PyArrow we use PySequence_Fast_GET_ITEM() (*) Maybe PyArrow is a kind of C extension which should have one implementation for the new C API (PyPy) and one implementation for the current C API (CPython)? Cython can be used to

Re: [Python-Dev] General concerns about C API changes

2018-11-14 Thread Victor Stinner
Hi, I made many different kinds of changes to the C API last weeks. None of them should impact the backward compatibility. If it's the case, the changes causing the backward compatibility should be reverted. The most controversial changes are the conversion of macros to static inline functions, bu

Re: [Python-Dev] General concerns about C API changes

2018-11-14 Thread Victor Stinner
> First, I attempted for "force inlining" (ex: > __attribute__((always_inline)) for GCC/Clang), but it has been decided > to not do that. Please read the discussion on the issue for the > rationale. > > https://bugs.python.org/issue35059 It has been decided to use "static inline" syntax since it's

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-14 Thread Victor Stinner
Le mer. 14 nov. 2018 à 14:36, Paul Moore a écrit : > PS What percentage does "top 5" translate to? In terms of both > downloads and actual numbers of extensions? With only 5, it would be > very easy (I suspect) to get only scientific packages, and (for > example) miss out totally on database APIs,

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-14 Thread Victor Stinner
In short, you don't have to modify your C extensions and they will continue to work as before on Python 3.8. I only propose to add a new C API, don't touch the existing one in any way. Introducing backward incompatible changes to the existing C API is out of my plan. /usr/bin/python3.8 will suppo

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-14 Thread Victor Stinner
Le mer. 14 nov. 2018 à 17:28, Paul Moore a écrit : > OK, got it. Thanks for taking the time to clarify and respond to my > concerns. Much appreciated. I'm my fault. I am failing to explain my plan proplerly. It seems like I had to update my website to better explain :-) Victor __

Re: [Python-Dev] Clarify required Visual C++ compiler for binary extensions in 3.7

2018-11-14 Thread Victor Stinner
I maintain my own compatibility matrix: https://pythondev.readthedocs.io/windows.html#python-and-visual-studio-version-matrix Ned Deily asked to update the devguide from my table :-) https://github.com/python/devguide/issues/433 Victor Le mer. 14 nov. 2018 à 18:56, Maik Riechert a écrit : > > Hi

Re: [Python-Dev] General concerns about C API changes

2018-11-14 Thread Victor Stinner
Le jeu. 15 nov. 2018 à 01:06, Gregory P. Smith a écrit : > I expect the largest visible impact may be that a profiler may now attribute > CPU cycles takes by these code snippets to the function from the .h file > rather than directly to the functions the macro expanded in in the past due > to a

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-16 Thread Victor Stinner
Brett: > But otherwise I think we are making assumptions here. For me, unless we are > trying to trim the C API down to just what is syntactically supported in > Python and in such a way that it hides all C-level details I feel like we are > guessing at what's best for other VMs, both today and

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-19 Thread Victor Stinner
Le lun. 19 nov. 2018 à 10:48, Antoine Pitrou a écrit : > If the C API only provides Python-level semantics, then it will > roughly have the speed of pure Python (modulo bytecode execution). > > There are important use cases for the C API where it is desired to have > fast type-specific access to P

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-19 Thread Victor Stinner
or Le lun. 19 nov. 2018 à 12:02, M.-A. Lemburg a écrit : > > On 19.11.2018 11:53, Antoine Pitrou wrote: > > On Mon, 19 Nov 2018 11:28:46 +0100 > > Victor Stinner wrote: > >> Python internals rely on internals to implement further optimizations, > >> than modifying an &

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-19 Thread Victor Stinner
Hi Stefan, Le lun. 19 nov. 2018 à 13:18, Stefan Krah a écrit : > In practice people desperately *have* to use whatever is there, including > functions with underscores that are not even officially in the C-API. > > I have to use _PyFloat_Pack* in order to be compatible with CPython, Oh, I never

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-20 Thread Victor Stinner
Le mar. 20 nov. 2018 à 23:08, Stefan Krah a écrit : > Intuitively, it should probably not be part of a limited API, but I never > quite understood the purpose of this API, because I regularly need any > function that I can get my hands on. > (...) > Reading typed strings directly into an array wit

Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)

2018-11-21 Thread Victor Stinner
Le mer. 21 nov. 2018 à 12:11, Antoine Pitrou a écrit : > You mean the same API can compile to two different things depending on > a configuration? Yes, my current plan is to keep #include but have an opt-in define to switch to the new C API. > I expect it to be error-prone. For example, let's

Re: [Python-Dev] Missing functions [Was: Re: Experiment an opt-in new C API for Python? (leave current API unchanged)]

2018-11-21 Thread Victor Stinner
Please open a bug report once you have such issue ;-) Victor Le mer. 21 nov. 2018 à 15:56, Matěj Cepl a écrit : > > On 2018-11-19, 11:59 GMT, Stefan Krah wrote: > > In practice people desperately *have* to use whatever is > > there, including functions with underscores that are not even > > offic

Re: [Python-Dev] General concerns about C API changes

2018-11-23 Thread Victor Stinner
Le dim. 18 nov. 2018 à 17:54, Stefan Behnel a écrit : > It's also slower to compile, given that function inlining happens at a much > later point in the compiler pipeline than macro expansion. The C compiler > won't even get to see macros in fact, whereas whether to inline a function > or not is a

Re: [Python-Dev] C API changes

2018-11-27 Thread Victor Stinner
Le mar. 27 nov. 2018 à 01:13, Larry Hastings a écrit : > (...) I'm not convinced the nice-to-have of "you can't dereference the > pointer anymore" is worth this runtime overhead. About the general idea of a new C API. If you only look at CPython in release mode, there is no benefit. But you sho

Re: [Python-Dev] C API changes

2018-11-30 Thread Victor Stinner
I just would want to say that I'm very happy to read the discussions about the C API finally happening on python-dev :-) The discussion is very interesting! > C is really the lingua franca Sorry, but why not writing the API directly in french? Victor _

Re: [Python-Dev] getting merge rights back on github

2018-12-03 Thread Victor Stinner
Hi, unittest.mock definitely needs your help :-) I merged a few changes last months, but I had to rely on Mario Corchero (who authored the new seal() method) for the review, since I don't know well this module. Michael Foord (original author) doesn't seem available for reviews. More reviewers shou

[Python-Dev] Internal header files (Include/internal/*.h) are now installed

2018-12-04 Thread Victor Stinner
Hi, Since Python 3.7, "internal" C API (only declared if Py_BUILD_CORE is defined) are moving from Include/*.h to Include/internal/*.h. These API must not be used outside CPython. In Python 3.7, "make install" doesn't install them for example. I would like to move more private functions (prefixed

Re: [Python-Dev] Internal header files (Include/internal/*.h) are now installed

2018-12-04 Thread Victor Stinner
Le mar. 4 déc. 2018 à 16:35, Antoine Pitrou a écrit : > > Are you ok to install "internal" header files? If yes, should we > > modify "make install" of Python 3.7 to also install them? > > +1 to both. Ok, I reopened https://bugs.python.org/issue35296 and wrote a PR: https://github.com/python/cpy

[Python-Dev] test_urllib2net fixed to repair Travis CI

2018-12-05 Thread Victor Stinner
Hi, It seems like Travis CI changed its security or network configuration: some FTP tests of test_urllib2net started to fail with "425 Security: Bad IP connecting" yesterday: https://bugs.python.org/issue35411 The failure seems to be reproducible and so prevents to merge any change in any bra

[Python-Dev] I reverted "Add Windows App Store package" change

2018-12-07 Thread Victor Stinner
Hi, I had to revert a change since it broke buildbots. Sadly, I don't have the bandwidth to investigate the failures and try to fix the change :-( A large change has been pushed into the 3.7 and master branches to "Add Windows App Store package": "Release Windows Store app containing Python" ht

Re: [Python-Dev] I reverted "Add Windows App Store package" change

2018-12-07 Thread Victor Stinner
Le ven. 7 déc. 2018 à 16:42, Steve Dower a écrit : > As a slight aside, 8 out of 8 buildbot messages on the PR look like > false positives, and none of the true positives sent a message. What > happened there? I don't know why the PR didn't get notifications about the regression. I got something

Re: [Python-Dev] I reverted "Add Windows App Store package" change

2018-12-07 Thread Victor Stinner
Le ven. 7 déc. 2018 à 16:16, Steve Dower a écrit : > The other changes are either in Windows-only files or tests. The one > exception is venv, where they are in "if os.name=='nt'" blocks. I also > pinged our venv expert a few times with no response. Yeah, the lack of review is an issue in Python,

[Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Victor Stinner
Hi, tzickel reported a reference cycle bug in multiprocessing which keeps threads and processes alive: https://bugs.python.org/issue34172 He wrote a fix which has been merged in 3.6, 3.7 and master branches. But Pablo Galindo noticed that the fix breaks the following code (he added "I found t

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Victor Stinner
Le mar. 11 déc. 2018 à 16:14, Antoine Pitrou a écrit : > What you are proposing here starts to smell like an anti-pattern to > me. Python _is_ a garbage-collected language, so by definition, there > _are_ going to be resources that are automatically collected when an > object disappears. If I'm

Re: [Python-Dev] Usage of the multiprocessing API and object lifetime

2018-12-11 Thread Victor Stinner
Le mar. 11 déc. 2018 à 17:06, Antoine Pitrou a écrit : > > We are not talking about simple strings, but processes and threads. > > Right, but do those have an impact on the program's correctness, or > simply on its performance (or memory consumption)? Performance. I made a similar change in the

Re: [Python-Dev] [RELEASE] Python 3.7.2rc1 and 3.6.8rc1 now available for testing

2018-12-12 Thread Victor Stinner
It's not easy to find the changelog, so here you have the direct links: * 3.7: https://docs.python.org/3.7/whatsnew/changelog.html#changelog * 3.6: https://docs.python.org/3.6/whatsnew/changelog.html#changelog Victor Le mer. 12 déc. 2018 à 04:43, Ned Deily a écrit : > > https://blog.python.org/2

Re: [Python-Dev] [RELEASE] Python 3.7.2rc1 and 3.6.8rc1 now available for testing

2018-12-12 Thread Victor Stinner
Le mer. 12 déc. 2018 à 18:09, Ned Deily a écrit : > ?? It's easy to find the changelog! There is a link to a release's changelog > on the release page included in the announcement: (...) Oh. It's at the end, right. I missed it :-) > Or you can always go to the top-level page of the documentati

Re: [Python-Dev] 3.7.2rc1 and 3.6.8rc1 cutoffs ahead, last 3.6.x bugfix release!

2018-12-19 Thread Victor Stinner
Hi, I am working in the Red Hat "Python-maint" team which is maintaining Python 3.6 as the main Python interpreter in RHEL 8, which will likely be supported for at least 10 years. And we have been supporting Python 2.7 in RHEL 7. So obviously, being able to benefit of the upstream effort and infra

[Python-Dev] AMD64 Windows8.1 Refleaks 3.x buildbot is back to green!

2019-01-09 Thread Victor Stinner
Hi, The "AMD64 Windows 8.1 Refleaks 3.x" buildbot (which hunts reference leaks and memory leaks) was failing on test_asyncio for 1 year: https://bugs.python.org/issue32710 It was a leak of a single reference: test_aiosend leaked [1, 1, 1] references, sum=3 I tried multiple times since la

<    5   6   7   8   9   10   11   12   13   14   >