The memory allocators don't have any context of tracing, they just allocate. Tracemalloc is a trampoline based allocator that also trace what's going on, bit from the point of view of the python allocator system is just another allocator.
There is no concept of "notify the python allocator" because the python allocator system doesn't have the concept of external allocation events. You can override it entirely to do something extra, but it is designed (as many other allocators) to not be aware of the existence of other systems running in parallel. In other words, the allocators are concerned about memory, not tracing or anything else that can be done by overriding them. That's why there is no "notify allocator" APIs in the python allocators. Regards from rainy London, Pablo Galindo Salgado On Tue, 15 Feb 2022, 13:57 Sümer Cip, <sum...@gmail.com> wrote: > Hi everyone, > > I would like to ask a question about an issue that we faced regarding > profiling memory usage: > > We have implemented a custom memory profiler using > `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now, > we are facing an issue with numpy: numpy seems to have its own memory > allocator and they use `PyTraceMalloc_Track` APIs to notify tracemalloc > about the allocation. A simple search on GitHub reveals there are more > projects using this approach: > https://github.com/search?q=PyTraceMalloc_Track&type=code which is fine. > I believe it is common to use custom memory allocators for scientific > computation which makes perfect sense. > > However, I would have expected to have at least some form of > `PyMem_NotifyAllocator` type of API instead of a one that is specific to > `tracemalloc`? I might be missing some context here. > > WDYT? > > Best, > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/BHOIDGRUWPM5WEOB3EIDPOJLDMU4WQ4F/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ATULLVYZMDGNOE2NYC53ZUZXBBAHLNSO/ Code of Conduct: http://python.org/psf/codeofconduct/
