[Python-Dev] Re: Need help to debug a ssl crash on Windows which prevents merging PRs

2021-06-01 Thread Victor Stinner
On Fri, May 28, 2021 at 6:40 PM Victor Stinner wrote: > In the 3.10 branch, it became really hard to merge PRs because the > following ssl crashs on Windows: > https://bugs.python.org/issue44252 Update on this bug which blocked the Python 3.10 beta 2 release. It's now fully fi

[Python-Dev] New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support

2021-06-02 Thread Victor Stinner
Hi, What do you think of promoting the pythoncapi_compat project that I'm introducing below in the "C API: Porting to Python 3.10" section of What's New In Python 3.10? Should this project be moved under the GitHub psf organization to have a more "future proof" URL? I would like to promote this

[Python-Dev] Re: New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support

2021-06-02 Thread Victor Stinner
On Thu, Jun 3, 2021 at 2:30 AM Joannah Nanjekye wrote: > Is HPy ready yet given IIRC, there is no even first release yet? I stand to > be corrected. > I reckon it will still go through a period of incompatible changes for some > time/months too. HPy is a great project, but even if 90% of top 40

[Python-Dev] Re: Proposal: declare "unstable APIs"

2021-06-03 Thread Victor Stinner
Hi Guido, It seems like you are talking about the Python API. In the C API, there is the internal C API which fits with your description. To access it, you have to declare the Py_BUILD_CORE_MODULE macro. It's not usable directly on purpose. It's an user agreement: I know what I am doing, and I kn

[Python-Dev] Re: Proposal: declare "unstable APIs"

2021-06-04 Thread Victor Stinner
On Fri, Jun 4, 2021 at 12:29 AM Guido van Rossum wrote: >> In the C API, there is the internal C API which fits with your >> description. To access it, you have to declare the >> Py_BUILD_CORE_MODULE macro. It's not usable directly on purpose. It's >> an user agreement: I know what I am doing, and

[Python-Dev] Re: Making PY_SSIZE_T_CLEAN not mandatory.

2021-06-09 Thread Victor Stinner
On Wed, Jun 9, 2021 at 10:32 AM Ronald Oussoren via Python-Dev wrote: > Its a bit late to complain (and I’m not affected by this myself), but those > functions are part of the stable ABI. The change in 3.10 will break any > extensions that use the stable ABI, use these functions and don’t use >

[Python-Dev] Re: Difficulty of testing beta releases now available

2021-06-11 Thread Victor Stinner
Hi, On Wed, May 26, 2021 at 2:05 AM Neil Schemenauer wrote: > - Cython doesn't work because of _PyGen_Send change [1] My team fixed the Python 3.10 compatibility in the Cython 0.29.x branch, but the latest 0.29.x release (0.29.23, April 14, 2021) doesn't include these fixes yet. A new Cython rel

[Python-Dev] Re: Difficulty of testing beta releases now available

2021-06-11 Thread Victor Stinner
ep in sync with tools/cythonize.py ] Victor On Fri, Jun 11, 2021 at 4:30 PM Victor Stinner wrote: > > Hi, > > On Wed, May 26, 2021 at 2:05 AM Neil Schemenauer > wrote: > > - Cython doesn't work because of _PyGen_Send change [1] > > My team fixed the Pytho

[Python-Dev] Re: Roundup to GitHub Issues migration

2021-06-21 Thread Victor Stinner
Hi, As a bug triager, it's very frustrating when I ask for more information about a bug, and I get no reply. Usually, such bug is closed as "out of date" (lack enough information to be debugged). The requirement for a GitHub account was well known when PEP 581 was accepted. The PEP was approved.

[Python-Dev] Re: Is the Python review process flawed?

2021-07-01 Thread Victor Stinner
Thanks Brian for your nice summary! Let me complete it. Warning: I didn't look at https://github.com/python/cpython/pull/4819 at all. My email is a really general remark on the Python workflow. You in this email is not the OP, but "any contributor" ;-) What contributors don't see is that regressi

[Python-Dev] Does anyone use threading debug PYTHONTHREADDEBUG=1 env var? Can I remove it?

2021-07-07 Thread Victor Stinner
Hi, Does anyone use threading debug PYTHONTHREADDEBUG=1 env var on a Python debug build? If not, can I just remove it? -- To fix a race condition at thread exit on Linux using the glibc, I removed calls to pthread_exit() (PyThread_exit_thread()) in the _thread module: https://bugs.python.org

[Python-Dev] Re: Does anyone use threading debug PYTHONTHREADDEBUG=1 env var? Can I remove it?

2021-07-08 Thread Victor Stinner
ry itself. (Evidenced by it only working on --with-pydebug, > which is for when you're debugging CPython itself, not any user apps.) > > On Wed, Jul 7, 2021 at 9:33 AM Gregory P. Smith wrote: >> >> >> >> On Wed, Jul 7, 2021 at 2:28 AM Victor Stinner wrote: >

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-08-04 Thread Victor Stinner
On Tue, Aug 3, 2021 at 7:54 PM Ethan Furman wrote: > I would rather keep `bchr` and lose the `.fromint()` methods. I would prefer to only have a bytes.byte(65) method, no bchr() built-in function. I would prefer to keep builtins namespace as small as possible. bytes.byte() name is similar to byt

[Python-Dev] Re: Windows buildbots may be broken

2021-08-04 Thread Victor Stinner
Hi Jason, One month ago, I changed the Buildbot configuration: --- Use Git "fresh" method Git "clean" method keeps most files created by a previous build. Use Git "fresh" method instead to ignores .gitignore rules and so remove all generated files (like ".o" files): run a fresh build rather than

[Python-Dev] Re: Repealing PEP 509 (Add a private version to dict)

2021-08-04 Thread Victor Stinner
Hi, I wrote the PEP 509 as part of my abandonned "FAT Python" project which was a ahead-of-time optimizer using runtime guards to deoptimize code. I planed to abandon this PEP as well, but the dictionary version was used by LOAD_GLOBAL opcode cache which made the version useful and so the PEP was

[Python-Dev] Re: Python3 OSF/1 support?

2021-08-04 Thread Victor Stinner
Hi, I suggest you to start by forking the python/cpython repository and keep your changes in a branch. You can share it on a website, maybe with a tarball including your patches. If it gets enough popularity, maybe we can consider later to include these changes. Since Alpha hardware is not longer

[Python-Dev] Re: [python-committers] Roundup to GitHub Issues migration

2021-08-05 Thread Victor Stinner
Hi Ezio, What is the status of the migration of Python issues from bugs.python.org (Roundup) to GitHub? Is it still a work-in-progress or is it stalled? Victor On Mon, Jun 21, 2021 at 4:20 AM Ezio Melotti wrote: > > As you might know, PEP 581 (Using GitHub Issues for CPython) has been > approve

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

2021-08-17 Thread Victor Stinner
se they have > different signatures) called simply Py_TRASHCAN_BEGIN and Py_TRASHCAN_END. > - by that time there was already a follow-up PR open (GH-12607) to improve > backwards compatibility of the macros, as well as introduce tests for them; > this was never merged. > - in Fe

[Python-Dev] Re: Deprecate Py_TRASHCAN_SAFE_BEGIN/END in 3.10?

2021-08-17 Thread Victor Stinner
See also: * https://bugs.python.org/issue44881 "Consider integration of PyObject_GC_UnTrack() with the trashcan C API". * https://bugs.python.org/issue44897 "Integrate trashcan mechanism into _Py_Dealloc" The issue44897 would make all these macros useless ;-) Victor _

[Python-Dev] Re: Making code object APIs unstable

2021-08-17 Thread Victor Stinner
Since Cython is a common consumer of this C API, can somone please dig into Cython to see exactly what it needs in terms of API? How does Cython create all arguments of the __Pyx_PyCode_New() macro? Does it copy an existing function to only override some fields, something like CodeType.replace(fiel

[Python-Dev] Re: My apologies to the list

2021-08-27 Thread Victor Stinner
Hi Antoine, I have an alternative to GMane. I'm using a different email address for anything related to Python (and OSS projects in general): python-dev list, bug tracker, buildbots, GitHub notifications, etc. Some days, I don't read this email account at all, but I can still read my personal and

[Python-Dev] Re: A ban from Core Developer spaces.

2021-08-27 Thread Victor Stinner
Hi, I collected past events about bans and code of conduct incidents in the Python community: https://pythondev.readthedocs.io/diversity.html#python-code-of-conduct-bans It may help some people to see how bans and incidents are handled in Python. See also the https://pythondev.readthedocs.io/com

[Python-Dev] Re: Is anyone relying on new-bugs-announce/python-bugs-list/bugs.python.org summaries

2021-08-30 Thread Victor Stinner
Hi Ammar, On Tue, Aug 24, 2021 at 12:17 AM Ammar Askar wrote: > 1. Weekly summary emails with bug counts and issues from the week, > example: > https://mail.python.org/archives/list/python-dev@python.org/thread/JRFJ4QH7TR35HFRQWOYPPCGOYRFAXK24/ These emails annoy me. I learnt to mark them as re

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Victor Stinner
If creating a fake frame is a common use case, we can maybe write a public C API for that. For example, I saw parser injecting frames to show the file name and line number of the parsed file in the traceback. Victor On Wed, Sep 1, 2021 at 4:07 AM Stefan Behnel wrote: > > Guido van Rossum schrieb

[Python-Dev] Re: Making code object APIs unstable

2021-09-01 Thread Victor Stinner
I saw Python projects injecting fake frames for XML and JSON parsers, maybe also configuration file (.ini?) parsers. So text files which have line numbers ;-) On Wed, Sep 1, 2021 at 7:33 PM Pablo Galindo Salgado wrote: > I don't think we should think on those terms. We certainly don't want to be

[Python-Dev] Re: Making code object APIs unstable

2021-09-03 Thread Victor Stinner
On Thu, Sep 2, 2021 at 11:15 PM Guido van Rossum wrote: > FWIW I've applied for an exception from the two-release deprecation policy > from the SC: > https://github.com/python/steering-council/issues/75 On the PyPI top 5000 packages, 136 contain "PyCode" in the source. I didn't check how many ar

[Python-Dev] Re: Making code object APIs unstable

2021-09-06 Thread Victor Stinner
Oh, I didn't know this *existing* C API function: PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) So Cython could be modified to use it, no? Victor On Tue, Sep 7, 2021 at 12:44 AM Guido van Rossum wrote: > > On Fri, Sep 3, 2021 at 4:12 PM Victor Sti

[Python-Dev] Re: deprecated-removed

2021-09-08 Thread Victor Stinner
Hi Serhiy, For me, "deprecated" should be used if the removal is *not* planned: the feature is too popular, and removing it would break too much 3rd party code. "deprecated-removal" should be used if the feature removal *is* planned. IMO it's ok if we forget to remove the feature and keep it long

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-08 Thread Victor Stinner
On Wed, Sep 8, 2021 at 7:46 AM Steven D'Aprano wrote: > >>> bytes.from_int(121404708502361365413651784, 'little') > # should return b'Hello world' Really? I don't know anyone serializing strings as a "bigint" number. Did you already see such code pattern in the wild? Usually, bytes are se

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-09 Thread Victor Stinner
Hum, it seems like this is a confusion between converting a whole bytes *string* to/from an integer, and converting a single *character* to/from an integer. I propose to rename PEP 467 method bytes.fromint(n) to => bytes.fromchar(n) <= to convert an integer to a single *character*: it fails if n i

[Python-Dev] Re: PEP 467 feedback from the Steering Council

2021-09-09 Thread Victor Stinner
I proposed bytes.byte earlier in this thread: https://mail.python.org/archives/list/python-dev@python.org/message/KBVVBJL2PHI55Y26Z4FMSCJPER242LFA/ Gregory dislikes the name: "I don't *like* to argue over names (the last stage of anything) but I do need to point out how that sounds to read". https

[Python-Dev] Spam from Daniel Scott

2021-09-10 Thread Victor Stinner
Hi, For 1 or 2 years, I'm getting spam messages from "Daniel Scott". It started with empty replies to multiple Python mailing lists. Mailing list owners: please block his email address. I never ever saw any useless message from this name (sorry if a real Daniel Scott plans to contribute to Python

[Python-Dev] Bug fixed in the bugs.python.org OAuth-based authentication: user logged as the wrong account

2021-09-13 Thread Victor Stinner
Hi, A bug has been identified and *fixed* in the OAuth-based authentication code used on the Python bug tracker bugs.python.org (BPO) to log in with GitHub, Launchpad or Google. Under some conditions, it was possible to be logged as another person account. We are only aware of a single user affect

[Python-Dev] Re: python3.10rc2 compilation on android/termux/clang12.0.1 fails

2021-09-16 Thread Victor Stinner
Hi, On Wed, Sep 15, 2021 at 7:16 PM Sandeep Gupta wrote: > I get following warnings and errors: > Python/pytime.c:398:10: warning: implicit conversion from 'long' to 'double' > changes value from 9223372036854775807 to 9223372036854775808 > [-Wimplicit-const-int-float-conver > sion] >if (!_

[Python-Dev] Re: How to decipher Windows errors on builds?

2021-09-16 Thread Victor Stinner
Hi Mark, https://bugs.python.org/issue45220 now tracks these Windows build errors. Victor On Thu, Sep 16, 2021 at 3:25 PM Mark Shannon wrote: > > Hi, > > I'm getting build errors on Windows for > https://github.com/python/cpython/pull/28386 > > The error message is: > > C:\Program Files (x86)\W

[Python-Dev] Regressions caused the recent work on ceval.c and frame objects

2021-09-19 Thread Victor Stinner
Hi, The recent optimization work on ceval.c and frame objects introduced regressions and the situation is stuck for 4 months (bpo-43760). Right now, maybe the best would be to revert the 2 commits in the 3.10 branch, to get more time in Python 3.11 development cycle to solve these issues. (1) "

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-27 Thread Victor Stinner
Hi Eric, Which stdlib modules are currently frozen? If I really want to hack site.py or os.py for whatever reason, I just have to use "python3 -X frozen_modules=off"? > 1. always default to "on" (the annoyance for contributors isn't big enough?) What is the annoyance? What is different between f

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-27 Thread Victor Stinner
On Tue, Sep 28, 2021 at 12:52 AM Eric Snow wrote: > On Mon, Sep 27, 2021 at 3:31 PM Victor Stinner wrote: > > Which stdlib modules are currently frozen? If I really want to hack > > site.py or os.py for whatever reason, I just have to use "python3 -X > > frozen_mo

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Victor Stinner
On Mon, Sep 27, 2021 at 6:58 PM Eric Snow wrote: > We've frozen most of the stdlib modules imported during "python -c > pass" [1][2], to make startup a bit faster. Import of those modules > is controlled by "-X frozen_modules=[on|off]". Currently it defaults > to "off" but we'd like to default t

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Victor Stinner
t; loader and repr. Also, frozen modules do not > have __file__ set (and __path__ is always []).) is that frozen modules don't > have a `__file__` attribute IIRC and therefore > tracebacks won't include the source. > > > On Mon, 27 Sept 2021 at 22:31, Victor Stinner wr

[Python-Dev] Re: The Default for python -X frozen_modules.

2021-09-28 Thread Victor Stinner
The Python Debug Build document lists changes compared to a release build: https://docs.python.org/dev/using/configure.html#python-debug-build Sometimes, I'm confused that "./python" (Python built locally in debug mode) displays warnings, whereas "python" (Fedora package) doesn't. See also the P

[Python-Dev] PEP: Taking the Python C API to the Next Level

2021-09-28 Thread Victor Stinner
+ :: PEP: xxx Title: Taking the Python C API to the Next Level Author: Victor Stinner Status: Draft Type: Informational Content-Type: text/x-rst Created: 28-Sep-2021 Python-Version: 3.11 While the C API is a key of the Python

[Python-Dev] Re: Should I care what -3.14 // inf produces?

2021-09-30 Thread Victor Stinner
Hi Jeff, The decimal module docstring starts with: """ This is an implementation of decimal floating point arithmetic based on the General Decimal Arithmetic Specification: http://speleotrove.com/decimal/decarith.html and IEEE standard 854-1987: http://en.wikipedia.org/wiki/IEEE_854-19

[Python-Dev] Re: PEP 654 except* formatting

2021-10-04 Thread Victor Stinner
To stay consistent with PEP 8, exception groups should use 4 spaces. Victor On Sun, Oct 3, 2021 at 5:54 PM Irit Katriel via Python-Dev wrote: > > > We wonder if people have a view on which of the following is clearer/better: > > 1. except *E as e: // except *(E1, E2) as e: > 2. except* E as e:

[Python-Dev] svn.python.org require an HTTP authentication: r77062 links at bugs.python.org

2021-10-08 Thread Victor Stinner
Hi, I'm digging into the Python bug tracker history and I found links to Subversion commits: "Fixed in r77062 (trunk), r77063 (py3k)." https://bugs.python.org/issue1811#msg96910 Roundup adds links which are redirected: * https://hg.python.org/lookup/r77062 -> https://svn.python.org/view?view=rev

Re: [Python-Dev] Disabling string interning for null and single-char causes segfaults

2013-03-04 Thread Victor Stinner
Hi, 2013/3/4 Amaury Forgeot d'Arc : > The example above is obviously from python2.7; there is a similar example > with python3.2: x = b'\xe9\xe9'.decode('ascii', 'ignore') x == '', x is '' > (True, False) > > ...but this bug has been fixed in 3.3: PyUnicode_Resize() always returns the >

Re: [Python-Dev] Difference in RE between 3.2 and 3.3 (or Aaron Swartz memorial)

2013-03-06 Thread Victor Stinner
n) or even on a different CPython version. Anyway, you spotted a missed optimization: it's now "fixed" in Python 3.3 and 3.4 by the following commits. Copy/paste of the CIA IRC bot: 19:30 < irker555> cpython: Victor Stinner 3.3 * 82517:3dd2fa78fb89 / Objects/unicodeobject.c:

Re: [Python-Dev] Difference in RE between 3.2 and 3.3 (or Aaron Swartz memorial)

2013-03-07 Thread Victor Stinner
You should try to write a simple test not using your library (just copy/paste code) reproducing the issue. If you can do that, please fill an issue on bugs.python.org. Victor 2013/3/7 Matej Cepl : > On 2013-03-06, 18:34 GMT, Victor Stinner wrote: >> In short, Unicode was rewritten in P

[Python-Dev] pytracemalloc 0.7: new tool to track memory leaks in Python

2013-03-07 Thread Victor Stinner
Hi, See below for a copy of my email posted to python-list and python-announce mailing lists. pytracemalloc tool requires a patch on Python to hook memory allocation functions. I posted the patch there: http://bugs.python.org/issue3329 Thanks to this patch, it would also be possible to enable or

[Python-Dev] Can we triple quoted string as a comment?

2013-03-25 Thread Victor Stinner
Hi, I just realized that the Python peephole optimizer removes useless instructions like numbers and strings between other instructions, without raising an error nor emiting an error. Example: $ python -Wd -c 'print "Hello"; "World"' Hello As part of my astoptimizer project, I wrote a function t

Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1

2013-03-26 Thread Victor Stinner
2013/3/26 Lennart Regebro : > On Tue, Mar 26, 2013 at 11:54 AM, Matthias Klose wrote: >> Am 25.03.2013 01:30, schrieb Benjamin Peterson: >>> 2.7.4 will be the latest maintenance release in the Python 2.7 series. >> >> I hope it's not (and in the IDLE thread you say so otherwise too). > > It most c

[Python-Dev] astoptimizer: static optimizer working on the AST

2013-03-26 Thread Victor Stinner
Hi, I made progress since last August on my astoptimizer project (read the Changelog). Previous email thread: http://mail.python.org/pipermail/python-dev/2012-August/121286.html The astoptimizer project is an optimizer rewriting Python AST. It executes as much code as possible during the compilat

Re: [Python-Dev] cpython: Add fast-path in PyUnicode_DecodeCharmap() for pure 8 bit encodings:

2013-04-11 Thread Victor Stinner
2013/4/11 Serhiy Storchaka : > On 09.04.13 23:29, victor.stinner wrote: >> >> http://hg.python.org/cpython/rev/53879d380313 >> changeset: 83216:53879d380313 >> parent: 83214:b7f2d28260b4 >> user:Victor Stinner >> date:Tue Apr 09 21

Re: [Python-Dev] [Python-checkins] cpython: Issue #17693: CJK encoders now use the new Unicode API (PEP 393)

2013-04-14 Thread Victor Stinner
Hi, 2013/4/14 Nick Coghlan : > On Sun, Apr 14, 2013 at 10:06 AM, victor.stinner > wrote: >> http://hg.python.org/cpython/rev/d621bdaed7c3 >> changeset: 83317:d621bdaed7c3 >> user:Victor Stinner >> date:Sun Apr 14 02:06:32 2013 +0200 >> summ

Re: [Python-Dev] Why can't I encode/decode base64 without importing a module?

2013-04-22 Thread Victor Stinner
Hi, Your question is discussed since 4 years in the following issue: http://bugs.python.org/issue7475 The last proposition is to add transform() and untransform() methods to bytes and str types. But nobody implemented the idea. If I remember correctly, the missing point is how to define which typ

Re: [Python-Dev] PEP 435 -- Adding an Enum type to the Python standard library

2013-04-27 Thread Victor Stinner
Why not defining new methods/changing the behaviour using a different metaclass? Victor Le 27 avr. 2013 05:12, "Nikolaus Rath" a écrit : > Steven D'Aprano writes: > > On 26/04/13 13:22, Greg wrote: > >> On 26/04/2013 3:12 p.m., Glenn Linderman wrote: > >>> On 4/25/2013 7:49 PM, Nick Coghlan wro

Re: [Python-Dev] PEP 435 - requesting pronouncement

2013-05-04 Thread Victor Stinner
Great job guys. Victor Le 5 mai 2013 00:06, "Eli Bendersky" a écrit : > Hello pydev, > > PEP 435 is ready for final review. A lot of the feedback from the last few > weeks of discussions has been incorporated. Naturally, not everything could > go in because some minor (mostly preference-based) i

Re: [Python-Dev] PEP 435 - requesting pronouncement

2013-05-05 Thread Victor Stinner
I'm unhappy with this API. I never used it. It is also more verbose than the functional API. Victor Le dimanche 5 mai 2013, Antoine Pitrou a écrit : > On Sat, 4 May 2013 15:04:49 -0700 > Eli Bendersky > wrote: > > Hello pydev, > > > > PEP 435 is ready for final review. A lot of the feedback from

[Python-Dev] All 3.x stable buildbots are red

2013-05-07 Thread Victor Stinner
http://buildbot.python.org/all/waterfall?category=3.x.stable x86 Windows Server 2003 [SB] 3.x: 3 tests failed, test___all__ test_gc test_ssl x86 Windows7 3.x: 3 tests failed, test___all__ test_gc test_ssl x86 Gentoo Non-Debug 3.x: 3 tests failed, test_logging test_multiprocessing test_urllib2net x

Re: [Python-Dev] Issue 11406: adding os.scandir(), a directory iterator returning stat-like info

2013-05-12 Thread Victor Stinner
2013/5/13 Ben Hoyt : > class DirEntry: > def __init__(self, name, dirent, lstat, path='.'): > # User shouldn't need to call this, but called internally by scandir() > self.name = name > self.dirent = dirent > self._lstat = lstat # non-public attributes >

Re: [Python-Dev] How to debug python crashes

2013-05-14 Thread Victor Stinner
Hi, I don't know if it can help, but if you really don't know where your programcrash/hang occurs, you can use the faulthandler module: https://pypi.python.org/pypi/faulthandler It can be used to display te backtrace of all threads on an event like a signal or a timeout. It works with Python, b

[Python-Dev] Enhancement of Python memory allocators

2013-06-12 Thread Victor Stinner
Hi, I would like to improve memory allocators of Python. My two use cases are replacing memory allocators with custom allocators in embedded system and hooking allocators to track usage of memory. I wrote a patch for this, I'm going to commit it if nobody complains: http://bugs.python.org/issue33

Re: [Python-Dev] Enhancement of Python memory allocators

2013-06-12 Thread Victor Stinner
2013/6/13 Nick Coghlan : > On 13 Jun 2013 09:09, "Victor Stinner" wrote: >> Using this patch, detecting memory corruptions (buffer underflow and >> overflow) can be done without recompilation. We may add an environment >> variable to enable Python debu

Re: [Python-Dev] Enhancement of Python memory allocators

2013-06-13 Thread Victor Stinner
2013/6/13 Nick Coghlan : > Yes, that sounds better. One of the biggest problems with the current > startup sequence is the way it relies on environment variables for > configuration, which makes life hard for other applications that want to > embed the CPython runtime. I wrote a new patch (attache

[Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

2013-06-13 Thread Victor Stinner
Hi, I would like to remove the "GIL must be held" restriction from PyMem_Malloc(). In my opinion, the restriction was motived by a bug in Python, bug fixed by the issue #3329. Let me explain why. The PyMem_Malloc() function is a thin wrapper to malloc(). It returns NULL if the size is lager than

Re: [Python-Dev] Allow calling PyMem_Malloc() without the GIL held in Python 3.4

2013-06-14 Thread Victor Stinner
I commited the new API (little bit different than my last patch on issue #3329): http://hg.python.org/cpython/rev/6661a8154eb3 The documentation will be available in a few minutes at: http://docs.python.org/3/c-api/memory.html 2013/6/14 Kristján Valur Jónsson : >> Removing the GIL restriction wou

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Victor Stinner
2013/6/15 Antoine Pitrou : >> http://hg.python.org/cpython/rev/6661a8154eb3 >> ... >> Issue #3329: Add new APIs to customize memory allocators >> >> * Add a new PyMemAllocators structure >> * New functions: >> >> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory >> a

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
Le 15 juin 2013 03:54, "Victor Stinner" > a écrit : > Ok, I reverted my commit. > > I will work on a PEP to explain all these new functions and their use cases. I created the PEP 445 to reserve the number. It is ready for a review but already contains some explanation

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Christian Heimes : > Am 15.06.2013 14:22, schrieb Nick Coghlan: >> However, it's still desirable to be able to monitor those direct >> allocations in debug mode, thus it makes sense to have a GIL protected >> direct allocation API as well. You could try to hide the existence of >> the lat

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Antoine Pitrou : > On Sat, 15 Jun 2013 03:54:50 +0200 > Victor Stinner wrote: >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be >> able to setup a custom allocator for

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Nick Coghlan : > The only reason for the small object allocator to exist is because > operating system allocators generally aren't optimised for frequent > allocation and deallocation of small objects. You can gain a *lot* of > speed from handling those inside the application. As the allo

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Antoine Pitrou : > Moreover, I think you are conflating two issues: the ability to add > memory allocation hooks (for tracing/debugging purposes), and the > adaptation to "non-traditional" memory models (whatever that means). > Those concerns don't necessarily come together. In my implem

Re: [Python-Dev] Backports galore

2013-06-16 Thread Victor Stinner
There is also faulthandler on PyPI. It is not really a backport since the project developement started on PyPI. Victor Le dimanche 16 juin 2013, Łukasz Langa a écrit : > Now we have (at least) the following libraries backported from 3.2+ to > older versions of Python by members of the core team:

Re: [Python-Dev] eval and triple quoted strings

2013-06-17 Thread Victor Stinner
It may be possible to implement parsing the codec cookie as a Python codec :-) Victor 2013/6/18 Guido van Rossum : > On Mon, Jun 17, 2013 at 5:02 PM, Benjamin Peterson > wrote: >> 2013/6/17 Guido van Rossum : >>> On Mon, Jun 17, 2013 at 4:40 PM, Benjamin Peterson >>> wrote: 2013/6/17 Gre

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-17 Thread Victor Stinner
2013/6/16 Antoine Pitrou : > On Sun, 16 Jun 2013 01:48:06 +0200 > Victor Stinner wrote: >> I just create the issue http://bugs.python.org/issue18227: "Use Python >> memory allocators in external libraries like zlib or OpenSSL". >> >> Is it possible to

Re: [Python-Dev] cpython (3.3): ctypes: AIX needs an explicit #include to get alloca()

2013-06-18 Thread Victor Stinner
2013/6/18 Christian Heimes : > Am 18.06.2013 12:56, schrieb Jeremy Kloth: >> On Mon, Jun 17, 2013 at 2:02 PM, victor.stinner >>> +#include >> >> This header is not present on Windows, thus breaking all the Windows >> buildbots. Perhaps it should be wrapped in an AIX-specific #ifdef? Oh really? P

[Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Victor Stinner
If you prefer the HTML version: http://www.python.org/dev/peps/pep-0445/ PEP: 445 Title: Add new APIs to customize Python memory allocators Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-june-2013 Python

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Victor Stinner
typedef struct { /* user context passed as the first argument to the 2 functions */ void *ctx; /* allocate a memory mapping */ void* (*alloc) (void *ctx, size_t size); /* release a memory mapping */ void (*free) (void *ctx, void *ptr,

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-18 Thread Victor Stinner
Le mercredi 19 juin 2013, Scott Dial a écrit : > On 6/18/2013 4:40 PM, Victor Stinner wrote: > > No context argument > > I think there is a lack of justification for the extra argument, and the > extra argument is not free. The typical use-case for doing this > continuation-

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
2013/6/19 Antoine Pitrou : > Le Tue, 18 Jun 2013 22:40:49 +0200, > Victor Stinner a écrit : >> >> Other changes >> - >> > [...] >> >> * Configure external libraries like zlib or OpenSSL to allocate memory >> using ``PyMem_RawMalloc

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
2013/6/19 Antoine Pitrou : > On Wed, 19 Jun 2013 17:24:21 +0200 >> >> Drawback: the caller has to check if the result is 0, or handle the >> >> error. >> > >> > Or you can just call Py_FatalError() if the domain is invalid. >> >> I don't like Py_FatalError(), especially when Python is embedded. It'

Re: [Python-Dev] pyparallel and new memory API discussions...

2013-06-19 Thread Victor Stinner
> 1. All memory allocated in a parallel context is localized to a > private heap. How do you allocate memory in this "private" heap? Did you add new functions to allocate memory? Victor ___ Python-Dev mailing list Python-Dev@python.org http

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
2013/6/19 Kristján Valur Jónsson : > Oh, it should be public, in my opinion. Ok. And do you think that the PyMemMappingAllocator structure is complete, or that we should add something to be future-proof? At least, PyMemMappingAllocator is enough for pymalloc usage :-) Is PyMemMappingAllocator com

Re: [Python-Dev] pyparallel and new memory API discussions...

2013-06-19 Thread Victor Stinner
am I still going to be able to do that with your new API in place? """ 2013/6/19 Trent Nelson : > On Wed, Jun 19, 2013 at 08:45:55AM -0700, Victor Stinner wrote: >> > 1. All memory allocated in a parallel context is localized to a >> > private hea

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
PyMem_RawAlloc()/Realloc/Free should be part of the stable ABI. I agree that all other new fumctions ans structures should not. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-19 Thread Victor Stinner
Le jeudi 20 juin 2013, Nick Coghlan a écrit : > > > Is PyMemMappingAllocator complete enough for your usage at CCP Games? > > Can we go back to calling this the "Arena" allocator? Or at least > "Mapped"? When I see "Mapping" in the context of Python I think of the > container API, not a memory allo

Re: [Python-Dev] RFC: PEP 445: Add new APIs to customize Python memory allocators

2013-06-20 Thread Victor Stinner
> * Add new GIL-free (no need to hold the GIL) memory allocator functions: > > - ``void* PyMem_RawMalloc(size_t size)`` > - ``void* PyMem_RawRealloc(void *ptr, size_t new_size)`` > - ``void PyMem_RawFree(void *ptr)`` > - the behaviour of requesting zero bytes is not defined: return *NULL* >

[Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)

2013-06-20 Thread Victor Stinner
Hi, I changed the PEP 445 according to the discussing on python-dev. Read it online: http://www.python.org/dev/peps/pep-0445/ Changes: * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or PYALLOC_PYOBJECT * rename: - PyMemBlockAllocator structure => PyMemAllocator - PyM

Re: [Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)

2013-06-20 Thread Victor Stinner
> I also updated the implementation attached to: > http://bugs.python.org/issue3329 I excluded new enums, structures and functions from the stable ABI, except PyMem_Raw*() functions. Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.p

[Python-Dev] stat module in C -- what to do with stat.py?

2013-06-20 Thread Victor Stinner
2013/6/20 Eric V. Smith >: > But isn't the real problem with this module in Python the fact that the > constants might be wrong? I'm not sure what, if anything, we can do > about that. Python is providing a stat module implemented in Python since 10 years, or maybe 20 years, and I don't remember t

[Python-Dev] PEP 445: Add new APIs to customize Python memory allocators (second round)

2013-06-20 Thread Victor Stinner
2013/6/20 Antoine Pitrou >: > Victor Stinner > a écrit : >> Changes: >> >> * add PyMemAllocatorDomain enum: PYALLOC_PYMEM_RAW, PYALLOC_PYMEM or >> PYALLOC_PYOBJECT > > PYMEM_DOMAIN_RAW? I prefer your suggestion because it shares the PYMEM/PyMem prefix w

Re: [Python-Dev] stat module in C -- what to do with stat.py?

2013-06-20 Thread Victor Stinner
2013/6/20 Eric V. Smith : > This is serious, not argumentative: If there's really no concern that > the values be correct, then why implement it in C? I read again the issue. The problem is to add new flags. Current flags (type: block device/symlink/..., file mode) are well defined and portable, w

Re: [Python-Dev] stat module in C -- what to do with stat.py?

2013-06-20 Thread Victor Stinner
2013/6/20 Serhiy Storchaka : > Now with enumerations in the stdlib the stat module constants are candidates > for flag enumerations. How easy will be implement it on C? Numerical values are less important than S_ISxxx() macros. Example: #define S_ISDOOR(mode) (((mode)&0xF000) == 0xd000) 0

Re: [Python-Dev] stat module in C -- what to do with stat.py?

2013-06-21 Thread Victor Stinner
2013/6/21 Nick Coghlan : > Because practicality beats purity. This "wrong" Python code has been > good enough for all Python version up until 3.4, it makes sense to > keep it as a fallback instead of throwing it away. How do you plan to handle the following case in Python? "Looking in more detail

Re: [Python-Dev] PEP 445 delegate

2013-06-21 Thread Victor Stinner
Hi, 2013/6/21 Antoine Pitrou : > I've been appointed PEP 445 delegate by Nick and Guido. I would like to > know if there are still pending changes to the PEP. Cool. Hum, there is maybe something. In my pytracemalloc project, I added another API to track usage of free lists: PyAPI_FUNC(int) _PyF

Re: [Python-Dev] PEP 445 delegate

2013-06-23 Thread Victor Stinner
Hi, 2013/6/21 Antoine Pitrou : > I've been appointed PEP 445 delegate by Nick and Guido. I would like to > know if there are still pending changes to the PEP. If not, I expect to > give it a review in the coming days or weeks, and then make a final > pronouncement (which will probably be positive

Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-24 Thread Victor Stinner
Hi Raymond, Thank you for your long explanation, it is exactly what Antoine asked for :-) I like micro-optimization even if I know that some other developers only see such work as noise, not providing an real speed up :-) So it was interesting to read your email! I'm sorry that you was injured by

Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-24 Thread Victor Stinner
2013/6/24 Raymond Hettinger : > Lastly, there was a change I just put in to Py 3.4 replacing > the memcpy() with a simple loop and replacing the > "deque->" references with local variables. Besides > giving a small speed-up, it made the code more clear > and less at the mercy of various implementa

Re: [Python-Dev] cpython (2.7): Fix comment blocks. Adjust blocksize to a power-of-two for better divmod

2013-06-24 Thread Victor Stinner
2013/6/24 Raymond Hettinger : > Changing the BLOCKLEN from 62 to 64 is debatable. > It measureably improved deque_index without an > observable negative effect on the other operations. Out of curiosity, do you know (remember) how was the number 62 chosen? Is it a compromise between memory usage an

Re: [Python-Dev] [Python-checkins] cpython: Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64

2013-06-24 Thread Victor Stinner
4d279508a3 >> user:Victor Stinner >> date:Mon Jun 24 23:31:48 2013 +0200 >> summary: >> Issue #9566: Fix a compiler warning in tupleiter_setstate() on Windows x64 >> ... >> static PyObject * >> tupleiter_setstate(tupleiterobject *it, PyO

<    9   10   11   12   13   14   15   16   17   18   >