Kristján Valur Jónsson added the comment:
At ccp we have something similar. We are embedding python in the UnrealEngine
on the PS3 and need to get everything through their allocators. For the
purpose of flexibility, we added an api similar to the OPs, but more flexible:
/* Support for custom allocators */
typedef void *(*PyCCP_Malloc_t)(size_t size, void *arg, const char *file, int
line, const char *msg);
typedef void *(*PyCCP_Realloc_t)(void *ptr, size_t size, void *arg, const char
*file, int line, const char *msg);
typedef void (*PyCCP_Free_t)(void *ptr, void *arg, const char *file, int line,
const char *msg);
typedef size_t (*PyCCP_Msize_t)(void *ptr, void *arg);
typedef struct PyCCP_CustomAllocator_t
{
PyCCP_Malloc_t pMalloc;
PyCCP_Realloc_t pRealloc;
PyCCP_Free_t pFree;
PyCCP_Msize_t pMsize; /* can be NULL, or return -1 if no size info is
avail. */
void *arg; /* opaque argument for the functions */
} PyCCP_CustomAllocator_t;
/* To set an allocator! use 0 for the regular allocator, 1 for the block
allocator.
* pass a null pointer to reset to internal default
*/
PyAPI_FUNC(void) PyCCP_SetAllocator(int which, const PyCCP_CustomAllocator_t *);
For a module to install itself as a "hook" at runtime, this approach can be
extended by querying the current allocator, so that such a hook can the
delegate the previous calls.
The "block" allocator here, is intended as the underlying allocator to be used
by obmalloc.c. Depending on platforms, this can then allocate aligned virtual
memory directly, which is more efficient than layering that on-top of a
malloc-like allocator.
There are areas in cPython that use malloc() directly. Those are actually not
needed in all cases, but to cope with them we change them all to new RAW api
calls (using preprocessor macros).
Essentially, malloc() maps to PyCCP_RawMalloc() or PyMem_MALLOC_INNER() (both
local additions) based on whether the particular site using malloc() requires
truly gil free malloc or not.
For this reason, the custom allocators mentioned canot be assumed to be called
with the GIL. However, it is easily possible to extend the system above so
that there is a GIL and non-GIL version for the 'regular' allocator.
I'll put details of the stuff we have done for EVE Online / Dust 514 on my
blog. It is this, but much much more too.
Hopefully we can arrive at a way to abstract memory allocation away from Python
in a flexible and extendible manner :)
----------
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
