Re: [Python-Dev] VxWorks and cpython?

2019-01-09 Thread Victor Stinner
Le mer. 9 janv. 2019 à 18:16, Christian Heimes a écrit : > It's a PEP. The process and expectations for platform are explained in > PEP 11, https://www.python.org/dev/peps/pep-0011/ I also wrote some information in my website: https://pythondev.readthedocs.io/platforms.html > If possible it woul

Re: [Python-Dev] VxWorks and cpython?

2019-01-10 Thread Victor Stinner
Le jeu. 10 janv. 2019 à 17:54, Kuhl, Brian a écrit : > Is there a good place to document "Python on VxWorks" ? Anywhere. If you don't know, you might use https://wiki.python.org/moin/ ... But I'm not sure that the wiki is still widely used. Many pages may be outdated. > The changes to Python are

Re: [Python-Dev] pickle5 backport updated

2019-01-16 Thread Victor Stinner
I see that the PEP is still a draft. What's the status? Do you need someone to review your PEP? Do you have anyone in mind who can review it? I don't feel able to review such PEP, sorry :-( Victor Le mer. 16 janv. 2019 à 11:14, Antoine Pitrou a écrit : > > > Hello, > > For the record, the pickl

Re: [Python-Dev] Source of truth for C-API

2019-01-22 Thread Victor Stinner
I consider that frameobject.h is also part of the C API, even if it's not included by Python.h. For example, PyFrame_GetLineNumber() is part of the C API, it's just that you have to explicitly #include "frameobject.h". I'm not aware of any tool to automatically list the content of the C API. Coun

Re: [Python-Dev] Source of truth for C-API

2019-01-23 Thread Victor Stinner
e best strategy for now is to wait until first alpha-beta > releases are out, and see if anyone complains. > > -- > Ivan > > > > On Wed, 23 Jan 2019 at 06:58, Steve Dower wrote: >> >> On 22Jan.2019 1517, Victor Stinner wrote: >> > I'm not

Re: [Python-Dev] Source of truth for C-API

2019-01-23 Thread Victor Stinner
ch, see > https://github.com/python/cpython/pull/11605/files#diff-d350c56a842065575842defb8aaa9f27 > > -- > Ivan > > On Wed, 23 Jan 2019 at 16:41, Victor Stinner wrote: >> >> What is your change? Did you remove these functions? Change their parameters? >> >> V

Re: [Python-Dev] Difference between Include/internal and Include/cpython ?

2019-02-03 Thread Victor Stinner
Hi Antoine, The rules to decide what goes where have been discussed in the issues which created Include/cpython/ and the issue moving more headers to Include/internal/. In short, internal/ should not be used outside CPython codebase. In Python 3.7, these headers were even not installed. I chose t

Re: [Python-Dev] find_library and issue21622

2019-02-11 Thread Victor Stinner
Hi, Would you mind to elaborate "some issues trying to deploy in an alpine container"? What are you trying to do? What is the error message? Some more context: * https://github.com/python/cpython/pull/10460 "bpo-21622: ctypes.util find_library walk LD_LIBRARY_PATH" * https://bugs.python.org/issu

Re: [Python-Dev] Adding test.support.safe_rmpath()

2019-02-13 Thread Victor Stinner
Bikeshedding: I suggest to remove "safe_" from the name, it's hard to guarantee that removal is "safe", especially on Windows where a removal can be blocked for many reasons. Victor Le mer. 13 févr. 2019 à 13:28, Giampaolo Rodola' a écrit : > > > Hello, > after discovering os.makedirs() has no u

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-13 Thread Victor Stinner
Hi, I'm a (strong) supporter of providing a "python" command which would be the latest Python version! As php does nowadays (after previous issues with "php4" vs "php5".) I don't recall that perl had "perl4" vs "perl5", the command was always "perl", no? Same for Ruby: it was still "ruby" after f

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-13 Thread Victor Stinner
Some more context about Petr's change, Fedora, RHEL and Red Hat. At the latest Language Summit (2018), Petr detailed the state of the migration to Python 3 and how Python 2 is and will be handled at Red Hat; "Linux distributions and Python 2" talk with Matthias Klose (who works on Debian/Ubuntu):

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-13 Thread Victor Stinner
Le mer. 13 févr. 2019 à 21:26, Barry Warsaw a écrit : > I don’t think this should be conflated with PEP 394. IMHO, 3.10 is just > fine. Python 4 should be reserved for some future mythical GIL-less > interpreter or other major API breaking change. It might never happen. My point is that chan

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-14 Thread Victor Stinner
Le jeu. 14 févr. 2019 à 14:38, Matthias Klose a écrit : > Debian's concern about pointing python to python3 is that it will break > software > after an upgrade. The current state seems is still the same that Debian > doesn't > want to ship a python symlink after the Python2 removal. The other

Re: [Python-Dev] Proposed dates for Python 3.4.10 and Python 3.5.7

2019-02-15 Thread Victor Stinner
Hi, Le ven. 15 févr. 2019 à 12:07, Miro Hrončok a écrit : > I've checked Fedora CVE bugs against python 3.4 and 3.5. Here is one missing I > found: > > CVE-2018-20406 https://bugs.python.org/issue34656 > memory exhaustion in Modules/_pickle.c:1393 > Marked as resolved, but I don't see it fixed on

Re: [Python-Dev] Proposed dates for Python 3.4.10 and Python 3.5.7

2019-02-15 Thread Victor Stinner
I wrote fixes: Le ven. 15 févr. 2019 à 12:28, Victor Stinner a écrit : > https://python-security.readthedocs.io/vuln/ssl-crl-dps-dos.html 3.5: https://github.com/python/cpython/pull/11867 3.4: https://github.com/python/cpython/pull/11868 > https://python-security.readthedocs.io/vuln/

[Python-Dev] OpenSSL 1.1.1 fixes merged into Python 2.7

2019-02-15 Thread Victor Stinner
Hi, I reviewed and merged pull requests written by my colleague Charalampos Stratakis to backport OpenSSL 1.1.1 fixes into the future Python 2.7.16. Benjamin Peterson (Python 2.7 release manager) wrote me: "I would very much like to see 1.1.1 support in a Python 2.7 release." These changes are bac

Re: [Python-Dev] Making PyInterpreterState an opaque type

2019-02-19 Thread Victor Stinner
Le lun. 18 févr. 2019 à 22:34, Jeroen Demeyer a écrit : > First of all, if everybody can actually #define Py_BUILD_CORE and get > access to the complete API, I don't mind so much. But then it's > important that this actually keeps working (i.e. that those headers will > always be installed). > > S

Re: [Python-Dev] Making PyInterpreterState an opaque type

2019-02-19 Thread Victor Stinner
Le mar. 19 févr. 2019 à 11:57, INADA Naoki a écrit : > On the other hand, it makes sense to move _PyTuple_ITEMS to (3) or even (2). > PyTuple_ITEMS(t) seems better than &PyTuple_GET_ITEM(t, 0). Please don't use &PyTuple_GET_ITEM() or _PyTuple_ITEMS(). It prevents to use a more efficient storage f

Re: [Python-Dev] buildbottest on Android emulator with docker

2019-02-19 Thread Victor Stinner
This is cool! Sadly, I don't have the bandwidth right now to play with it. I may have a look later. Victor Le ven. 15 févr. 2019 à 18:27, Xavier de Gaye a écrit : > > The following command runs the buildbottest on an Android emulator with > docker (it will use a little bit more than 11 GB): > >

Re: [Python-Dev] Question - Bug Triage for 3.4 & 3.5

2019-02-20 Thread Victor Stinner
Hi, If Python 3.4 was the current version when a bug was reported, I would expect the version field of the bug set to Python 3.4. Maybe the bug has been fixed in the meanwhile, maybe not. Closing all bugs affected to 3.4 is a risk of loosing useful information on real bugs: closed bugs are ignored

Re: [Python-Dev] Making PyInterpreterState an opaque type

2019-02-21 Thread Victor Stinner
Le mar. 19 févr. 2019 à 21:15, Stefan Behnel a écrit : > What I would ask, though, and I think that's also Jeroen's request, is to > be careful what you lock up behind Py_BUILD_CORE. Any new functionality > should be available to extension modules by default, unless there is a good > reason why it

Re: [Python-Dev] Making PyInterpreterState an opaque type

2019-02-21 Thread Victor Stinner
Hi Eric, IMHO the main blocker issue for any C API change is that nobody is able to measure the risk these changes. To better control the risk, I propose to select a list of popular C extensions, and build a CI to run their test suite on top of the development version of Python. Such CI wouldn't d

Re: [Python-Dev] Making PyInterpreterState an opaque type

2019-02-21 Thread Victor Stinner
Le jeu. 21 févr. 2019 à 12:36, Antoine Pitrou a écrit : > > On Thu, 21 Feb 2019 12:13:51 +0100 > Victor Stinner wrote: > > > > Premature optimization is the root of all evil. Most C extensions use > > premature optimization > > How do you know it's prema

Re: [Python-Dev] Add minimal information with a new issue?

2019-02-21 Thread Victor Stinner
Hi, I wrote "python -m test.pythoninfo", but I wrote it to debug buildbot failures. It produces a large output. Only very few info are useful, but previously it was hard to get these info. I wouldn't suggest to start to request it. Or if you want it, it should be attached as a file to not pollute

Re: [Python-Dev] "Good first issues" on the bug tracker

2019-02-22 Thread Victor Stinner
Hi, Let me share with you (this mailing list) my experience with mentoring and what we call "easy issue". First of all, almost all "easy issues" are very hard issues: issues open for longer than one year, with many comments, and nobody succeeded to come up with a solution (well, otherwise the issu

Re: [Python-Dev] Possible performance regression

2019-02-25 Thread Victor Stinner
Hi, Le lun. 25 févr. 2019 à 05:57, Raymond Hettinger a écrit : > I'll been running benchmarks that have been stable for a while. But between > today and yesterday, there has been an almost across the board performance > regression. How do you run your benchmarks? If you use Linux, are you usi

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Hi, Le mar. 26 févr. 2019 à 05:27, Eric Snow a écrit : > I ran the "performance" suite (https://github.com/python/performance), > which has 57 different benchmarks. Ah yes, by the way: I also ran manually performance on speed.python.org yesterday: it added a new dot at Feb 25. > In the results,

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 12:33, INADA Naoki a écrit : > - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower > (+52%)> ... > ... > unpickle and unpickle_list shows massive slowdown. I suspect this slowdown > is not caused from set change. Linux perf shows many pagefault is happ

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

2019-02-26 Thread Victor Stinner
Armin Rigo released https://pypi.org/project/cffi/1.12.2/ which is compatible with Python 3.8.0a2. The issue was related to the PyInterpreterState change: https://bugs.python.org/issue35886#msg336501 Note: "[RELEASE] Python 3.8.0a1 is now available for testing" the correct version is 3.8.0a2 :-)

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
I made an attempt once and it was faster: https://faster-cpython.readthedocs.io/registervm.html But I had bugs and I didn't know how to implement correctly a compiler. Victor Le mardi 26 février 2019, Neil Schemenauer a écrit : > On 2019-02-25, Eric Snow wrote: >> So it looks like commit ef4ac9

Re: [Python-Dev] Compact ordered set

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 17:33, INADA Naoki a écrit : > My company gives me dedicated Linux machine with Core(TM) i7-6700. > So I think it's not issue of my machine. Oh great :-) > perf shows this line caused many page fault. > https://github.com/python/cpython/blob/c606a9cbd48f69d3f4a09204c781dd

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
k-based bytecode to register-based bytecode on the fly. Having both in the same code allowed to me run some benchmarks. Maybe it wasn't the best approach, but I didn't feel able to write a real compiler (AST => bytecode). Victor Le mar. 26 févr. 2019 à 21:58, Neil Schemenauer a écr

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
quot;duplicated" LOAD_GLOBAL whereas the global might be modified in the middle. I wanted to experiment such optimizations. Maybe it was a bad idea to convert stack-based bytecode to register-based bytecode and experiment these optimizations at the same time. Victor Le mar. 26 févr. 2019 à 22:42, Vic

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 22:45, Raymond Hettinger a écrit : > Victor said he generally doesn't care about 5% regressions. That makes sense > for odd corners of Python. The reason I was concerned about this one is that > it hits the eval-loop and seems to effect every single op code. The > reg

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 21:58, Neil Schemenauer a écrit : > It seems his name doesn't appear in the readme or source but I think > Rattlesnake was Skip Montanaro's project. I suppose my idea of > unifying the local variables and the registers could have came from > Rattlesnake. Very little new i

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 22:24, Gregory P. Smith a écrit : > A feature that I find missing from posix-y OSes that support #! lines is an > ability to restrict what can use a given interpreter. Fedora runs system tools (like "/usr/bin/semanage", tool to manager SELinux) with "python3 -Es": $ head

Re: [Python-Dev] Register-based VM [Was: Possible performance regression]

2019-02-26 Thread Victor Stinner
Le mar. 26 févr. 2019 à 23:40, Greg Ewing a écrit : > Victor Stinner wrote: > > LOAD_CONST_REG R0, 2 (const#2) > > LOAD_GLOBAL_REG R1, 'range' (name#0) > > CALL_FUNCTION_REG4, R1, R1, R0, 'n' > > Out of curiosity, why is the funct

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Hi, PGO compilation is very slow. I tried very hard to avoid it. I started to annotate the C code with various GCC attributes like "inline", "always_inline", "hot", etc.. I also experimented likely/unlikely Linux macros which use __builtin_expect(). At the end... my efforts were worthless. I stil

Re: [Python-Dev] Possible performance regression

2019-02-26 Thread Victor Stinner
Le mer. 27 févr. 2019 à 00:17, Victor Stinner a écrit : > My sad story with code placement: > https://vstinner.github.io/analysis-python-performance-issue.html > > tl; dr Use PGO. Hum wait, this article isn't complete. You have to see the follow-up: https://bugs.python.org/iss

Re: [Python-Dev] mmap & munmap loop (Was: Compact ordered set

2019-02-27 Thread Victor Stinner
Any idea why Python calls mmap+munmap more even in a venv? Victor Le mer. 27 févr. 2019 à 10:00, INADA Naoki a écrit : > > > > > > Ah, another interesting point, this huge slowdown happens only when > > > bm_pickle.py > > > is executed through pyperformance. When run it directly, slowdown is >

Re: [Python-Dev] mmap & munmap loop (Was: Compact ordered set

2019-02-27 Thread Victor Stinner
Sorry, I didn't get a coffee yet: more *often* in a venv. Le mer. 27 févr. 2019 à 11:32, Victor Stinner a écrit : > > Any idea why Python calls mmap+munmap more even in a venv? > > Victor > > Le mer. 27 févr. 2019 à 10:00, INADA Naoki a écrit : > > > > >

Re: [Python-Dev] mmap & munmap loop (Was: Compact ordered set

2019-02-27 Thread Victor Stinner
ts (0major+201149minor)pagefaults 0swaps > > 20 faults! > It seems two page fault / loop. (2 pools are used and returned). > > > On Wed, Feb 27, 2019 at 7:51 PM Victor Stinner wrote: >> >> Sorry, I didn't get a coffee yet: more *often* in a venv. >>

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

2019-02-28 Thread Victor Stinner
Hi, Le ven. 1 mars 2019 à 02:12, Neil Schemenauer a écrit : > I believe the correct fix is to use PEP 3121 per-interpreter module > state. I created a new issue: > > https://github.com/psycopg/psycopg2/issues/854 > > I think the fix is not trival as the psycopgmodule.c source code has > chan

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.4.10rc1 and Python 3.5.7rc1 are now available

2019-03-04 Thread Victor Stinner
FYI I check and I confirm that all known security vulnerabilities listed in the link below are fixed in these releases: https://python-security.readthedocs.io/vulnerabilities.html Victor Le lun. 4 mars 2019 à 10:24, Larry Hastings a écrit : > > > On behalf of the Python development community, I'

Re: [Python-Dev] PEPs from non-core devs now need a sponsor

2019-03-05 Thread Victor Stinner
Hi, Le mar. 5 mars 2019 à 02:53, Brett Cannon a écrit : > The steering council has implemented a new idea called sponsors to the PEP > process (...). The thinking is that to help make sure PEPs from non-core > developers receive appropriate guidance through the PEP process (...) Hum, this isn'

Re: [Python-Dev] PEP 12 updated with templates for header fields and sections

2019-03-08 Thread Victor Stinner
Hi Brett, I like to see discussions where a PEP has been disccussed. There is the "Post History" just gives dates. The problem is that PEPs are discussed on 3 mailing lists: python-ideas, python-dev and python-committers. Maybe some PEP are now also discussed on discuss.python.org. Do you have an

Re: [Python-Dev] PEP 12 updated with templates for header fields and sections

2019-03-08 Thread Victor Stinner
Yeah, such list is more convenient than just "Post-History: 14-Aug-2001, 03-Sept-2001". Victor Le ven. 8 mars 2019 à 09:40, Jeroen Demeyer a écrit : > > On 2019-03-08 09:29, Victor Stinner wrote: > > I would like to suggest to add URLs to the first messages of all &

Re: [Python-Dev] [python-committers] Last-minute request: please backport bpo-33329 fix to 3.4 and 3.5

2019-03-11 Thread Victor Stinner
I don't think that a thread about a release is the right place to discuss a bug. Please open an issue at bugs.python.org 😉 Victor Le lundi 11 mars 2019, Joni Orponen a écrit : > On Sun, Mar 10, 2019 at 7:50 AM Larry Hastings wrote: >> >> On 3/4/19 2:29 AM, Joni Orponen wrote: >> >> On Sat, Mar

Re: [Python-Dev] Initial updates to PEP 1 for Steering Council based governance

2019-03-14 Thread Victor Stinner
Hi, Le lun. 11 mars 2019 à 13:26, Nick Coghlan a écrit : > This is the smallest change to PEP 1 that we consider potentially viable: > handling all PEPs through the BDFL-Delegate model, with the Steering > Council's primary involvement being to appoint the delegates (or accept their > voluntee

[Python-Dev] Rejecting PEP 473: Adding structured data to built-in exceptions

2019-03-16 Thread Victor Stinner
Hi, While I would like to get such new attributes, I see drawbacks as blocker issues and so I am fine with this PEP rejection. Performance is too critical for most common exceptions. For me one blocker issue is the high risk of creating reference cycles. And the weak reference API isn't the most

Re: [Python-Dev] Remove tempfile.mktemp()

2019-03-19 Thread Victor Stinner
Hi, I would prefer to keep tempfile.mktemp(), remove the deprecation, but better explain the risk of race condition affecting security. Le mar. 19 mars 2019 à 14:41, Chris Angelico a écrit : > Can't you create a NamedTemporaryFile and permit the other program to > use it? I just tried that (with

Re: [Python-Dev] What is the workflow for announcing tier 2 platform support?

2019-03-19 Thread Victor Stinner
Hi, I don't think that we can say that Python supports VxWorks yet. Many PRs are not merged yet. You are free to disrribute a modified version with your changes. The PEP 11 lists conditions to fully support a platform: https://www.python.org/dev/peps/pep-0011/ My summary is: all tests must pass

Re: [Python-Dev] Can I submit more support of standard library for VxWorks after 3.8.0 beta1?

2019-03-19 Thread Victor Stinner
You have to find someone to review your PRs. It takes time. We are all volunteers. I looked and merged some VxWorks PRs. Would it be possible to pay a core dev to do the reviews? That's an open question for core devs and for WindRiver. Victor Le mardi 19 mars 2019, Xin, Peixing a écrit : > Hi,

Re: [Python-Dev] Remove tempfile.mktemp()

2019-03-20 Thread Victor Stinner
Hi, I'm not really convinced that mktemp() should be made "more secure". To be clear: mktemp() is vulnerable by design. It's not a matter of entropy. You can watch the /tmp directory using inotify and "discover" immediately the "secret" filename, it doesn't depend on the amount of entropy used to

Re: [Python-Dev] Is XML serialization output guaranteed to be bytewise identical forever?

2019-03-20 Thread Victor Stinner
Hi, Le lun. 18 mars 2019 à 23:41, Raymond Hettinger a écrit : > We're having a super interesting discussion on > https://bugs.python.org/issue34160 . It is now marked as a release blocker > and warrants a broader discussion. Thanks for starting a thread on python-dev. I'm the one who raised t

Re: [Python-Dev] Is XML serialization output guaranteed to be bytewise identical forever?

2019-03-20 Thread Victor Stinner
Le jeu. 21 mars 2019 à 01:30, Raymond Hettinger a écrit : > There's no preaching and no judgment. We can't have a conversation though if > we can't state the crux of the problem: some existing tests in third-party > modules depend on the XML serialization being byte-for-byte identical > foreve

Re: [Python-Dev] Is XML serialization output guaranteed to be bytewise identical forever?

2019-03-20 Thread Victor Stinner
Le lun. 18 mars 2019 à 23:41, Raymond Hettinger a écrit : > The code in the current 3.8 alpha differs from 3.7 in that it removes > attribute sorting and instead preserves the order the user specified when > creating an element. As far as I can tell, there is no objection to this as > a featur

Re: [Python-Dev] Removing PendingDeprecationWarning

2019-03-22 Thread Victor Stinner
Hi, I agree to make PendingDeprecationWarning an alias to DeprecationWarning. I never liked "PendingDeprecationWarning" name, it's way too long to type :-D Le ven. 22 mars 2019 à 03:45, Inada Naoki a écrit : > I want to stop using PendingDeprecationWarning for new deprecation. I'm fine with tha

Re: [Python-Dev] Replacement for array.array('u')?

2019-03-22 Thread Victor Stinner
Hi, Internally, CPython has a _PyUnicodeWriter which is an efficient way to create a string but appending substrings or characters. _PyUnicodeWriter changes the internal storage format depending on characters code points (ascii or latin1: 1 byte/character, BMP: 2 b/c, full UCS: 4 b/c). I tried onc

Re: [Python-Dev] Removing PendingDeprecationWarning

2019-03-22 Thread Victor Stinner
Le ven. 22 mars 2019 à 08:54, Inada Naoki a écrit : > Yes. It will be removed at some point, but not in near future. > > But when when backward compatibility can be kept by alias, > we can be very lazy about removing it. Honestly, I don't see the point of removing PendingDeprecationWarning anyti

Re: [Python-Dev] Removing PendingDeprecationWarning

2019-03-22 Thread Victor Stinner
Le ven. 22 mars 2019 à 09:16, Inada Naoki a écrit : > > > We have `socket.error` for long time. > > > > And it's perfectly fine, no? > > > > Yes. Waiting 10+ years to remove aliases is fine. Sure. Let me elaborate my point of view on deprecation, since we are discussing it here (and I know that

Re: [Python-Dev] Removing PendingDeprecationWarning

2019-03-22 Thread Victor Stinner
Le ven. 22 mars 2019 à 13:05, Serhiy Storchaka a écrit : > > The main problem is complexity. In other words, learning cost. > > Do you have evidences that many people have troubles with learning > PendingDeprecationWarning? I have no idea when I should use PendingDeprecationWarning rather than D

Re: [Python-Dev] introduction of __attribute__(deprecated) ?

2019-03-24 Thread Victor Stinner
Le dim. 24 mars 2019 à 20:56, Stéphane Wirtel a écrit : > So my question is, can we use/add __attribute__(deprecated) in our > "development" kit? There is already Py_DEPRECATED(VERSION_UNUSED) macro which uses __attribute__((__deprecated__)) on GCC >= 3.1. You can find the doc of a few macros in

[Python-Dev] New Python Initialization API

2019-03-27 Thread Victor Stinner
Hi, I would like to add a new C API to initialize Python. I would like your opinion on the whole API before making it public. The code is already implemented. Doc of the new API: https://pythondev.readthedocs.io/init_config.html To make the API public, _PyWstrList, _PyInitError, _PyPreConfig

[Python-Dev] PEP 587: Python Initialization Configuration

2019-03-27 Thread Victor Stinner
implement what Nick wants. Nick: please correct me if I'm wrong :-) Victor PEP: 587 Title: Python Initialization Configuration Author: Victor Stinner Discussions-To: python-dev@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2018 Python-Version: 3.8 Ab

Re: [Python-Dev] New Python Initialization API

2019-03-27 Thread Victor Stinner
Le mer. 27 mars 2019 à 19:35, Alexander Belopolsky a écrit : > Would you consider making _Py_UnixMain public as well? > > It is useful for high level embedding and not trivial for 3rd parties to > reimplement. Right, Py_Main() is causing a lot of practice issues, especially mojibake because of t

Re: [Python-Dev] New Python Initialization API

2019-03-27 Thread Victor Stinner
the priority between the different ways to configuration Python and "rules" (a parameter setting other parameters): https://github.com/python/peps/blob/master/pep-0587.rst#priority-and-rules Victor Le mer. 27 mars 2019 à 18:48, Victor Stinner a écrit : > > Hi, > > I would l

Re: [Python-Dev] New Python Initialization API

2019-03-28 Thread Victor Stinner
Le jeu. 28 mars 2019 à 05:27, Stephen J. Turnbull a écrit : > Victor Stinner writes: > > > I just fixed the mojibake issue in Python 3.8 by disabling C locale > > coercion and UTF-8 Mode by default. I'm not sure if nor how Python 3.7 > > should be fixed in a

Re: [Python-Dev] New Python Initialization API

2019-03-28 Thread Victor Stinner
Le mer. 27 mars 2019 à 21:26, Brett Cannon a écrit : > On Wed, Mar 27, 2019 at 12:39 PM Steve Dower wrote: >> I this this should be its own PEP, since as you say it is not >> implementing the only PEP we have (or alternatively, maybe you should >> collaborate to update PEP 432 so that it reflects

Re: [Python-Dev] PEP 578: Python Runtime Audit Hooks

2019-03-28 Thread Victor Stinner
Hi, I read quickly the PEP, I'm not sure that I understood it correctly, so here are some early questions more about the usage of the PEP, than its implementation. > This is not sandboxing, as this proposal does not attempt to prevent > malicious behavior (though it enables some new options to do

Re: [Python-Dev] New Python Initialization API

2019-03-29 Thread Victor Stinner
The purpose of the PEP 587 is to have a working document so everyone can look at the proposed API (stay focused to the API rather than bothering with the implementation). IMHO it's now time to get more people looking at the Python Initialization. > But there are enough of us > with fuzzy but valid

Re: [Python-Dev] New Python Initialization API

2019-04-05 Thread Victor Stinner
Le dim. 31 mars 2019 à 01:49, Steve Dower a écrit : > Here is my first review of https://www.python.org/dev/peps/pep-0587/ and > in general I think it's very good. Ah nice, that's a good start :-) Thanks for reviewing it. Your email is long, and answer makes it even longer, so I will reply in mu

Re: [Python-Dev] New Python Initialization API

2019-04-05 Thread Victor Stinner
About PyPreConfig and encodings. > The appendix is excellent, by the way. Very useful detail to have > written down. Thanks. The appendix is based on Include/cpython/coreconfig.h comments which is now my reference documentation. By the way, this header file contains more information about PyConfi

Re: [Python-Dev] New Python Initialization API

2019-04-05 Thread Victor Stinner
> > Example of Python initialization enabling the isolated mode:: > > > > PyConfig config = PyConfig_INIT; > > config.isolated = 1; > > Haven't we already used extenal values by this point that should have > been isolated? On this specific example, "config.isolated = 1;" ensures that Py_Pr

Re: [Python-Dev] New Python Initialization API

2019-04-05 Thread Victor Stinner
> I think my biggest point (about halfway down) is that I'd rather use > argv/environ/etc. to *initialize* PyConfig and then only use the config > for initializing the runtime. That way it's more transparent for users > and more difficult for us to add options that embedders can't access. I chose

Re: [Python-Dev] New Python Initialization API

2019-04-05 Thread Victor Stinner
For the PyMainConfig structure idea, I cannot comment at this point. I need more time to think about it. About the "path configuration" fields, maybe a first step to enhance the API would be to add the the following function: PyInitError PyConfig_ComputePath(PyConfig *config, const wchar *home);

Re: [Python-Dev] New Python Initialization API

2019-04-06 Thread Victor Stinner
s why Nick and me decided to disable C loclae coercion and UTF-8 Mode by default when the C API is used. Victor Le samedi 6 avril 2019, Inada Naoki a écrit : > On Sat, Apr 6, 2019 at 1:13 AM Victor Stinner wrote: >> >> > > ``PyPreConfig`` fields: >> > > >>

[Python-Dev] Need help to fix HTTP Header Injection vulnerability

2019-04-09 Thread Victor Stinner
Hi, In May 2017, user "Orange" found a vulnerability in the urllib fix for CVE-2016-5699 (HTTP Header Injection vulnerability): https://bugs.python.org/issue30458 It allows to inject arbitrary HTTP headers. Copy of their message: """ Hi, the patch in CVE-2016-5699 can be broke by an addition s

[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-09 Thread Victor Stinner
Hi, When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI. I propose to modify the debug build (Py_DEBUG) to no longer imply Py_TRACE_REFS. Antoine Pitrou proposed t

Re: [Python-Dev] Need help to fix HTTP Header Injection vulnerability

2019-04-10 Thread Victor Stinner
Hi, I dig into Python code history and the bug tracker. I would like to say that this issue is a work-in-progress since 2004. Different fixes have been pushed, but there are *A LOT* of open issues: https://bugs.python.org/issue30458#msg339846 I would suggest to discuss on https://bugs.python.org/

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-10 Thread Victor Stinner
ode related to Py_TRACE_REFS. I wrote it to see which code depends on it: commit 63509498761a0e7f72585a8cd7df325ea2abd1b2 (HEAD -> remove_trace_refs, origin/remove_trace_refs) Author: Victor Stinner Date: Thu Mar 28 23:26:58 2019 +0100 WIP: bpo-36465: Remove Py_TRACE_REFS special build

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-10 Thread Victor Stinner
Le mer. 10 avr. 2019 à 20:09, Steve Dower a écrit : > > The main question is if anyone ever used Py_TRACE_REFS? Does someone > > use sys.getobjects() or PYTHONDUMPREFS environment variable? > > > > Using PYTHONDUMPREFS=1 on a debug build (with Py_TRACE_REFS) does > > simply crash Python 3.7 at exi

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-10 Thread Victor Stinner
Le mer. 10 avr. 2019 à 21:45, Brett Cannon a écrit : >> Can anyone explain what pydebug builds are... for? Confession: I've never >> used them myself, and don't know why I would want to. > > There is a bunch of extra things done in a debug build, e.g. all freed memory > is blanked out with a kno

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-11 Thread Victor Stinner
Le jeu. 11 avr. 2019 à 07:49, Serhiy Storchaka a écrit : > 10.04.19 14:01, Victor Stinner пише: > > Disabling Py_TRACE_REFS by default in debug mode reduces the Python > > memory footprint. Py_TRACE_REFS costs 2 pointers per PyObject: 16 > > bytes on 64-bit platforms. &g

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-12 Thread Victor Stinner
Le jeu. 11 avr. 2019 à 17:33, Serhiy Storchaka a écrit : > If reducing the Python memory footprint is an argument for disabling > Py_TRACE_REFS, it is a weak argument because there is larger overhead in > the debug build. The "serialno" field of debug memory allocators is documented as: "an excel

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-12 Thread Victor Stinner
Le ven. 12 avr. 2019 à 12:57, Victor Stinner a écrit : > I wrote https://bugs.python.org/issue36611 to remove the serialno field > of debug hooks on Python memory allocators: it reduces > the memory footprint by 5% (ex: 1.2 MiB on 33.0 MiB when running > test_asyncio). I measure

Re: [Python-Dev] PEP-582 and multiple Python installations

2019-04-12 Thread Victor Stinner
Hi, Le mar. 2 avr. 2019 à 17:20, Calvin Spealman a écrit : > While the PEP does show the version number as part of the path to the actual > packages, implying support for multiple versions, this doesn't seem to be > spelled out in the actual text. Presumably __pypackages__/3.8/ might sit > bes

Re: [Python-Dev] Proposal: dict.with_values(iterable)

2019-04-12 Thread Victor Stinner
Nice optimization! I have questions on the proposed API. > with_values(self, iterable, /) > Create a new dictionary with keys from this dict and values from iterable. > > When length of iterable is different from len(self), ValueError is raised. > This method does not support dict subc

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-12 Thread Victor Stinner
Le sam. 13 avr. 2019 à 00:38, Greg Ewing a écrit : > Rather than removing features altogether, maybe the debug > build could be split into a number of separate features > that can be enabled individually? I don't propose to *remove* a feature, but just to *disable* it *by default* (when Python is

[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-12 Thread Victor Stinner
>> The very first question I asked was whether this would let us converge >> the ABIs, and the answer was "no". The answer is yes and it's my primary goal. See my first email: "This change makes the debug build ABI closer to the release build ABI". To be honest, I am now lost in this long thread

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-16 Thread Victor Stinner
Hi Michael, Do you know the tracemalloc module? Did you try it? It works on a regular Python (compiled in debug mode). I would be curious to know if tracemalloc also allows you to track the memory leak. sys.getobjects() is just a list of objects. Do you have a tool written on top of it to track

Re: [Python-Dev] No longer enable Py_TRACE_REFS by default in debug build

2019-04-16 Thread Victor Stinner
Since Python 3.6, you can use PYTHONMALLOC=malloc for Valgrind: it avoids false alarms produced by the pymalloc allocator. Victor Le mar. 16 avr. 2019 à 12:09, Nathaniel Smith a écrit : > > On Mon, Apr 15, 2019 at 8:58 PM Michael Sullivan wrote: > > > > On Mon, Apr 15, 2019 at 4:06 PM Nathaniel

Re: [Python-Dev] PEP 578: Python Runtime Audit Hooks

2019-04-16 Thread Victor Stinner
Le mar. 16 avr. 2019 à 14:35, Christian Heimes a écrit : > * Linux: readlink("/proc/self/fd/%i") That doens't work if /proc is not mounted, which can occur in a container (where /proc is not mounted nor binded to host /proc). Victor ___ Python-Dev mail

[Python-Dev] bpo-36558: Change time.mktime() return type from float to int?

2019-04-16 Thread Victor Stinner
Hi, time.mktime() looks "inconsistent" to me and I would like to change it, but I'm not sure how it impacts backward compatibility. https://bugs.python.org/issue36558 time.mktime() returns a floating point number: >>> type(time.mktime(time.localtime())) The documentation says: "It returns a f

Re: [Python-Dev] bpo-36558: Change time.mktime() return type from float to int?

2019-04-16 Thread Victor Stinner
Le mar. 16 avr. 2019 à 16:44, Paul Ganssle a écrit : > 2. Mainly for Victor, but anyone can answer: I agree that the natural output > of `mktime()` would be `int` if I were designing it today, but would there be > any practical benefits for making this change? It's just for the consistency of t

Re: [Python-Dev] bpo-36558: Change time.mktime() return type from float to int?

2019-04-16 Thread Victor Stinner
Le mar. 16 avr. 2019 à 17:46, Guido van Rossum a écrit : > Consistency with C should not be the issue -- consistency between the time > functions is important. > (...) > So let's drop the idea. Ok, I'm fine with that. It was just an idea ;-) I closed the issue. Victor _

Re: [Python-Dev] PEP 7: Adding anonymous union / struct

2019-04-17 Thread Victor Stinner
AIX is somehow supported and uses xlc compiler: does xlc support this C11 feature? Do you want to use it in Python 3.8 and newer only? Victor Le mer. 17 avr. 2019 à 13:14, Inada Naoki a écrit : > > Hi, all. > > PEP 7 includes some C99 features. > I propose to add include anonymous union and str

[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Victor Stinner
Hi, Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by default in debug build", but I lost myself in details, I forgot the main purpose of my proposal... Let me retry from scratch with a more explicit title: I would like to be able to run C extensions compiled in release mode on

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Victor Stinner
Le mer. 24 avr. 2019 à 01:44, Victor Stinner a écrit : > The first requirement for the use case is that a Python debug build > supports the ABI of a release build. (...) I > propose to no longer imply Py_TRACE_REFS (...) > > Apart of Py_TRACE_REFS, I'm not aware of othe

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

2019-04-23 Thread Victor Stinner
Le mer. 24 avr. 2019 à 01:44, Victor Stinner a écrit : > The current blocker issue is that > the Py_DEBUG define imply the Py_TRACE_REFS define (...): > > https://bugs.python.org/issue36465 > https://github.com/python/cpython/pull/12615 I updated my PR: """ Release

<    6   7   8   9   10   11   12   13   14   15   >