The memory allocators don't have any context of tracing, they just
allocate. Tracemalloc is a trampoline based allocator that also trace
what's going on, bit from the point of view of the python allocator system
is just another allocator.

There is no concept of "notify the python allocator" because the python
allocator system doesn't have the concept of external allocation events.
You can override it entirely to do something extra, but it is designed (as
many other allocators) to not be aware of the existence of other systems
running in parallel. In other words, the allocators are concerned about
memory, not tracing or anything else that can be done by overriding them.

That's why there is no "notify allocator" APIs in the python allocators.

Regards from rainy London,
Pablo Galindo Salgado

On Tue, 15 Feb 2022, 13:57 Sümer Cip, <sum...@gmail.com> wrote:

> Hi everyone,
>
> I would like to ask a question about an issue that we faced regarding
> profiling memory usage:
>
> We have implemented a custom memory profiler using
> `PyMem_GetAllocator/PyMem_SetAllocator` APIs like `tracemalloc`. Right now,
> we are facing an issue with numpy: numpy seems to have its own memory
> allocator and they use `PyTraceMalloc_Track` APIs to notify tracemalloc
> about the allocation. A simple search on GitHub reveals there are more
> projects using this approach:
> https://github.com/search?q=PyTraceMalloc_Track&type=code which is fine.
> I believe it is common to use custom memory allocators for scientific
> computation which makes perfect sense.
>
> However, I would have expected to have at least some form of
> `PyMem_NotifyAllocator` type of API instead of a one that is specific to
> `tracemalloc`? I might be missing some context here.
>
> WDYT?
>
> Best,
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BHOIDGRUWPM5WEOB3EIDPOJLDMU4WQ4F/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ATULLVYZMDGNOE2NYC53ZUZXBBAHLNSO/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to