[Python-Dev] tp_dealloc
My tp_dealloc method (of non-subtypable type) calls the freeMem-method of a memory manager (this manager was also used for the corresponding allocation). This freeMem-method deallocates and modifies the memory, which is a valid action, because after free, the memory-manager has ownership of the freed memory. Several memory managers do this (for example the Memory Manager in Delphi during debug mode, in order to track invalid memory access after free). The python31.dll calls tp_alloc and later (after return of tp-alloc) the python31.dll is still awaiting valid content in the deallocated memory. I don't know where this happens, I'm not a developer of CPython, but at this point the python31.dll causes an access violation. IMO the python31.dll assumes that freeMem never modifies the memory (pyobject header), this is valid for many memory managers, but not for all. And from my perspective, this assumption a bug, which can cause access violations in many applications (for example, applications which use the PythonForDelphi-package; PyScripter is one of them, but also many others) Please, could some CPython-developer take a look, thank you! -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] tp_dealloc
Sorry, I wrote tp_alloc in last post, it should be always tp_dealloc: My tp_dealloc method (of non-subtypable type) calls the freeMem-method of a memory manager (this manager was also used for the corresponding allocation). This freeMem-method deallocates and modifies the memory, which is a valid action, because after free, the memory-manager has ownership of the freed memory. Several memory managers do this (for example the Memory Manager in Delphi during debug mode, in order to track invalid memory access after free). The python31.dll calls tp_dealloc and later (after return of tp_dealloc) the python31.dll is still awaiting valid content in the deallocated memory. I don't know where this happens, I'm not a developer of CPython, but at this point the python31.dll causes an access violation. IMO the python31.dll assumes that freeMem never modifies the memory (pyobject header), this is valid for many memory managers, but not for all. And from my perspective, this assumption a bug, which can cause access violations in many applications (for example, applications which use the PythonForDelphi-package; PyScripter is one of them, but also many others) Please, could some CPython-developer take a look, thank you! -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
2010/6/1 : > Sorry, I wrote tp_alloc in last post, it should be always tp_dealloc: > > My tp_dealloc method (of non-subtypable type) calls the freeMem-method > of a memory manager (this manager was also used for the corresponding > allocation). > This freeMem-method deallocates and modifies the memory, > which is a valid action, because after free, the memory-manager > has ownership of the freed memory. > Several memory managers do this (for example the Memory Manager in > Delphi during debug mode, in order to track invalid memory access after free). > > The python31.dll calls tp_dealloc and later (after return of tp_dealloc) > the python31.dll is still awaiting valid content in the deallocated memory. > I don't know where this happens, I'm not a developer of CPython, > but at this point the python31.dll causes an access violation. > IMO the python31.dll assumes that freeMem never modifies the memory > (pyobject header), this is valid for many memory managers, but not for all. > And from my perspective, this assumption a bug, which can cause access > violations in many applications (for example, applications which use the > PythonForDelphi-package; PyScripter is one of them, but also many others) > > Please, could some CPython-developer take a look, thank you! CPython does not access memory after the call to tp_dealloc. There is even a mode (--without-pymalloc) where tp_dealloc calls free() at the end, and would cause crashes if the memory was read afterwards. This said, there may be a bug somewhere, but what do you want us to look at? Do you have a case that we could reproduce and investigate? -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
> This said, there may be a bug somewhere, but what do you want us to look > at? > Do you have a case that we could reproduce and investigate? > > -- > Amaury Forgeot d'Arc Thank you, I'm not a C-Developer, but still I have one more detail: I call py_decRef( pyObj) of dll (version 3.1.1), ( which calls tp_dealloc, which calls my freeMem() method)) No problem is reported here. Now, the freed memory should not be accessed anymore by python31.dll. You may fill the freed pyObjectHead with invalid values, in my case it's: ob_refcnt= 7851148, ob_type = $80808080 But later, when I call Py_Finalize, there inside is some access to the same freed memory; this causes an AV, more precisely, when the value $80808080 is checked. My Delphi-Debugger shows the following byte-sequence inside python31.dll: 5EC3568B7424088B4604F7405400407504 5E - pop esi C3 - ret 56 - push esi 8B742408- mov esi, [esp+$08] 8B4604 - mov eax, [esi+$04] // eax = $80808080 // F740540040 - test [eax+$54], $4000 // AV exception by read of address $808080D4 // 7504- jnz $1e03681b Maybe this can help someone, thank you! -- Marvin GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
2010/6/1 : >> This said, there may be a bug somewhere, but what do you want us to look >> at? >> Do you have a case that we could reproduce and investigate? >> >> -- >> Amaury Forgeot d'Arc > > Thank you, I'm not a C-Developer, > but still I have one more detail: > > I call py_decRef( pyObj) of dll (version 3.1.1), > ( which calls tp_dealloc, which calls my freeMem() method)) > No problem is reported here. > Now, the freed memory should not be accessed anymore by python31.dll. > You may fill the freed pyObjectHead with invalid values, > in my case it's: ob_refcnt= 7851148, ob_type = $80808080 > > But later, when I call Py_Finalize, > there inside is some access to the same freed memory; > this causes an AV, more precisely, > when the value $80808080 is checked. > > My Delphi-Debugger shows the following byte-sequence inside python31.dll: > 5EC3568B7424088B4604F7405400407504 > > 5E - pop esi > C3 - ret > 56 - push esi > 8B742408 - mov esi, [esp+$08] > 8B4604 - mov eax, [esi+$04] > // eax = $80808080 // > > F740540040 - test [eax+$54], $4000 > // AV exception by read of address $808080D4 // > > 7504 - jnz $1e03681b > > > Maybe this can help someone, thank you! I'm sorry but this kind of issue is difficult to investigate without the source code. Normally I would compile everything (python & your program) in debug mode, and try to see why the object is used after tp_dealloc. For example, it's possible that your code does not handle reference counts correctly A call to Py_INCREF() may be missing somewhere, for example. This is a common error. tp_dealloc() is called when the reference count falls to zero, but if the object is still referenced elsewhere, memory will be accessed again! Without further information, I cannot consider this as a problem in Python. I know other extension modules that manage memory in their own way, and work. It's more probably an issue in the code of your type. -- Amaury Forgeot d'Arc ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
> Without further information, I cannot consider this as a problem in > Python. > I know other extension modules that manage memory in their own way, and > work. > It's more probably an issue in the code of your type. > > -- > Amaury Forgeot d'Arc Ok, thank you, but I'm still hoping, someone could test this. I'm very sure, my app is not the cause; only the python31.dll (py_finalize) is accessing the freed memory. Inside py_finalize there is really no call to my hosting app (or reverse), I even tested this in my debugger. In most applications this python-problem remains hidden, because their freeMem() leaves the freed memory unmodified. (And that's why very good debuggers modify the freed memory to reveal such hidden errors). You could simply test this by setting pyObject.ob_type = $80808080 after freeMem( pyObject). Then later, call py_finalize, and you will see the same problem (Access violation by trying to use ob_type) -- GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT! Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] tp_dealloc
2010/6/1 :
>> Without further information, I cannot consider this as a problem in
>> Python.
>> I know other extension modules that manage memory in their own way, and
>> work.
>> It's more probably an issue in the code of your type.
>>
>> --
>> Amaury Forgeot d'Arc
>
> Ok, thank you, but I'm still hoping, someone could test this.
> I'm very sure, my app is not the cause;
> only the python31.dll (py_finalize) is accessing the freed memory.
> Inside py_finalize there is really no call to my hosting app (or reverse),
> I even tested this in my debugger.
To be clear:
- you did not provide anything for us to test.
- the fact that the crash is inside python31.dll does not indicates a
bug in python.
Consider this (bogus) code:
FILE *fp = fopen("c:/temp/t", "w");
free(fp);
This will lead to a crash at program exit (when fcloseall() is called
by the system)
but the issue is really in the code - it should not free(fp).
Without knowing what your code really do, we won't be able to help.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
