Hi, I would like to improve memory allocators of Python. My two use cases are replacing memory allocators with custom allocators in embedded system and hooking allocators to track usage of memory.
I wrote a patch for this, I'm going to commit it if nobody complains: http://bugs.python.org/issue3329 Using this patch, detecting memory corruptions (buffer underflow and overflow) can be done without recompilation. We may add an environment variable to enable Python debug functions at runtime, example: PYDEBUGMALLOC=1. There is just a restriction: the environment variable would not be ignored with -E command line option, because command line options are parsed after the first memory allocation. What do you think? ***** The patch adds the following functions: void PyMem_GetAllocators( void **ctx_p, void* (**malloc_p) (void *ctx, size_t size), void* (**realloc_p) (void *ctx, void *ptr, size_t size), void (**free_p) (void *ctx, void *ptr)); void PyMem_SetAllocators( void *ctx, void* (*malloc) (void *ctx, size_t size), void* (*realloc) (void *ctx, void *ptr, size_t size), void (*free) (void *ctx, void *ptr)); It adds 4 similar functions (get/set) for PyObject_Malloc() and allocators of pymalloc arenas. ***** For the "track usage of memory" use case, see the following project which hooks memory allocators using PyMem_SetAllocators() and PyObject_SetAllocators() to get allocated bytes per filename and line number. https://pypi.python.org/pypi/pytracemalloc ***** Another issue proposes to use VirtualAlloc() and VirtualFree() for pymalloc arenas, see: http://bugs.python.org/issue13483 I don't know if it would be interesting, but it would now possible to choose the memory allocator (malloc, mmap, HeapAlloc, VirtualAlloc, ...) at runtime, with an environment variable for example. Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
