Hi Julien, Isn't this a similar problem that you have with regular malloc? When you call malloc() with some size, malloc actually will reserve more than that for alignment purposes and for bookkeeping and apart from some platform-specific APIs like malloc_usable_size() you cannot query that value.
Unless I am missing something here, there is also some problem of what you propose: the Python allocators deal with memory and they do not know what that memory is going to be used so when you say: > This is a bummer as there then no safe way that I can think of to know ifan allocated memory space is gc-tracked or gc-untracked. My answer would be that that's because memory itself cannot be gc tracked, only objects can and those belonging to different categories. For example, notice that the tracemalloc module does not report objects, it only reports memory blocks and you cannot ask tracemalloc to give you a list of all the objects that were allocated because it does not have the notion of what an object is. Adding the functionality that you suggest would couple the allocator with other parts of the VM and it also won't give you the the answer that you need to because what's underneath (pymalloc or libc malloc or whatever else) may actually allocate even more than what you asked. Regards from cloudy London, Pablo Galindo Salgado On Thu, 14 Jan 2021 at 16:10, Julien Danjou <jul...@danjou.info> wrote: > Hi there, > > I've spent quite some time on memory profiling for Python now. I'm > struggling to get more information from the allocated memory right now > for what looks like a sad reason. :( > > Supposedly PyObject_Malloc() returns some memory space to store a > PyObject. If that was true all the time, that would allow anyone to > introspect the allocated memory and understand why it's being used. > > Unfortunately, this is not the case. Objects whose types are tracked by > the GC go through _PyObject_GC_Alloc() which changes the underlying > memory structure to be (PyGC_HEAD + PyObject). > > This is a bummer as there then no safe way that I can think of to know > if an allocated memory space is gc-tracked or gc-untracked. It makes it > therefore impossible to introspect the memory allocated by > PyObject_Malloc(). > > There are multiple ways to solve this, but I don't see any way right now > of doing this without slightly changing CPython. > > Has anyone any idea on how to workaround this or the kind of change that > could be acceptable to mitigate the issue? > > Thanks! > > Cheers, > -- > Julien Danjou > # Free Software hacker > # https://julien.danjou.info > _______________________________________________ > Python-Dev mailing list -- python-dev@python.org > To unsubscribe send an email to python-dev-le...@python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/QRX6U5XBXMHMT6YKIXERS3UT64ALYV27/ > Code of Conduct: http://python.org/psf/codeofconduct/ >
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ISXWWDMPY2N4UAAEHC4VYLCRW2XJLWDM/ Code of Conduct: http://python.org/psf/codeofconduct/
