Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-17 Thread Victor Stinner
2013/6/16 Antoine Pitrou : > On Sun, 16 Jun 2013 01:48:06 +0200 > Victor Stinner wrote: >> I just create the issue http://bugs.python.org/issue18227: "Use Python >> memory allocators in external libraries like zlib or OpenSSL". >> >> Is it possible to detect if Python is used as a standalone appli

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-16 Thread Nick Coghlan
On 16 June 2013 19:50, Antoine Pitrou wrote: > On Sun, 16 Jun 2013 01:48:06 +0200 > Victor Stinner wrote: > >> 2013/6/15 Christian Heimes : >> > Am 15.06.2013 14:22, schrieb Nick Coghlan: >> >> However, it's still desirable to be able to monitor those direct >> >> allocations in debug mode, thus

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-16 Thread Antoine Pitrou
On Sun, 16 Jun 2013 02:18:32 +0200 Victor Stinner wrote: > 2013/6/15 Antoine Pitrou : > > On Sat, 15 Jun 2013 03:54:50 +0200 > > Victor Stinner wrote: > >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 > >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be >

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-16 Thread Antoine Pitrou
On Sun, 16 Jun 2013 01:48:06 +0200 Victor Stinner wrote: > 2013/6/15 Christian Heimes : > > Am 15.06.2013 14:22, schrieb Nick Coghlan: > >> However, it's still desirable to be able to monitor those direct > >> allocations in debug mode, thus it makes sense to have a GIL protected > >> direct allo

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Nick Coghlan
On 16 Jun 2013 10:54, "Victor Stinner" wrote: > > 2013/6/15 Antoine Pitrou : > > Moreover, I think you are conflating two issues: the ability to add > > memory allocation hooks (for tracing/debugging purposes), and the > > adaptation to "non-traditional" memory models (whatever that means). > > Th

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Antoine Pitrou : > Moreover, I think you are conflating two issues: the ability to add > memory allocation hooks (for tracing/debugging purposes), and the > adaptation to "non-traditional" memory models (whatever that means). > Those concerns don't necessarily come together. In my implem

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Nick Coghlan : > The only reason for the small object allocator to exist is because > operating system allocators generally aren't optimised for frequent > allocation and deallocation of small objects. You can gain a *lot* of > speed from handling those inside the application. As the allo

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Antoine Pitrou : > On Sat, 15 Jun 2013 03:54:50 +0200 > Victor Stinner wrote: >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be >> able to setup a custom allocator for *all* allocation made by Pytho

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
2013/6/15 Christian Heimes : > Am 15.06.2013 14:22, schrieb Nick Coghlan: >> However, it's still desirable to be able to monitor those direct >> allocations in debug mode, thus it makes sense to have a GIL protected >> direct allocation API as well. You could try to hide the existence of >> the lat

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Christian Heimes
Am 15.06.2013 14:57, schrieb Victor Stinner: > Le 15 juin 2013 03:54, "Victor Stinner" > a écrit : > >> Ok, I reverted my commit. >> >> I will work on a PEP to explain all these new functions and their use > cases. > > I created the PEP 445 to reserve the number. It is ready for a review > but a

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Christian Heimes
Am 15.06.2013 14:22, schrieb Nick Coghlan: > However, it's still desirable to be able to monitor those direct > allocations in debug mode, thus it makes sense to have a GIL protected > direct allocation API as well. You could try to hide the existence of > the latter behaviour and treat it as a pri

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Antoine Pitrou
On Sun, 16 Jun 2013 00:12:02 +1000 Nick Coghlan wrote: > On 15 June 2013 22:41, Antoine Pitrou wrote: > > On Sat, 15 Jun 2013 22:22:33 +1000 > > Nick Coghlan wrote: > >> For > >> custom allocators, it's useful to be able to *ensure* you can bypass > >> CPython's small object allocator, rather th

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Nick Coghlan
On 15 June 2013 22:41, Antoine Pitrou wrote: > On Sat, 15 Jun 2013 22:22:33 +1000 > Nick Coghlan wrote: >> For >> custom allocators, it's useful to be able to *ensure* you can bypass >> CPython's small object allocator, rather than having to rely on it >> being bypassed for allocations above a ce

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Victor Stinner
Le 15 juin 2013 03:54, "Victor Stinner" > a écrit : > Ok, I reverted my commit. > > I will work on a PEP to explain all these new functions and their use cases. I created the PEP 445 to reserve the number. It is ready for a review but already contains some explanation of the new API. http://www.

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Antoine Pitrou
On Sat, 15 Jun 2013 22:22:33 +1000 Nick Coghlan wrote: > On 15 June 2013 21:01, Antoine Pitrou wrote: > > On Sat, 15 Jun 2013 03:54:50 +0200 > > Victor Stinner wrote: > >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 > >> (Replace calls to malloc() with PyMem_Malloc()). The

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Nick Coghlan
On 15 June 2013 21:01, Antoine Pitrou wrote: > On Sat, 15 Jun 2013 03:54:50 +0200 > Victor Stinner wrote: >> The addition of PyMem_RawMalloc() is motivated by the issue #18203 >> (Replace calls to malloc() with PyMem_Malloc()). The goal is to be >> able to setup a custom allocator for *all* alloc

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Antoine Pitrou
On Sat, 15 Jun 2013 03:54:50 +0200 Victor Stinner wrote: > The addition of PyMem_RawMalloc() is motivated by the issue #18203 > (Replace calls to malloc() with PyMem_Malloc()). The goal is to be > able to setup a custom allocator for *all* allocation made by Python, > so malloc() should not be cal

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-15 Thread Gregory P. Smith
fwiw i'm also supportive of adding these apis. Lets PEP away to iron out any details or document disagreements but overall I'd also like to see something a lot like these go in. -gps On Fri, Jun 14, 2013 at 10:50 PM, Nick Coghlan wrote: > On 15 June 2013 11:54, Victor Stinner wrote: > > 2013

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Nick Coghlan
On 15 June 2013 11:54, Victor Stinner wrote: > 2013/6/15 Antoine Pitrou : >>> http://hg.python.org/cpython/rev/6661a8154eb3 >>> ... >>> Issue #3329: Add new APIs to customize memory allocators >>> >>> * Add a new PyMemAllocators structure >>> * New functions: >>> >>> - PyMem_RawMalloc(), PyMem

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Victor Stinner
2013/6/15 Antoine Pitrou : >> http://hg.python.org/cpython/rev/6661a8154eb3 >> ... >> Issue #3329: Add new APIs to customize memory allocators >> >> * Add a new PyMemAllocators structure >> * New functions: >> >> - PyMem_RawMalloc(), PyMem_RawRealloc(), PyMem_RawFree(): GIL-free memory >> a

Re: [Python-Dev] cpython: Issue #3329: Add new APIs to customize memory allocators

2013-06-14 Thread Antoine Pitrou
On Sat, 15 Jun 2013 00:44:11 +0200 (CEST) victor.stinner wrote: > http://hg.python.org/cpython/rev/6661a8154eb3 > changeset: 84127:6661a8154eb3 > user:Victor Stinner > date:Sat Jun 15 00:37:46 2013 +0200 > summary: > Issue #3329: Add new APIs to customize memory allocators >