[issue43351] `RecursionError` during deallocation

[Andrew V. Jones]

Andrew V. Jones  added the comment:

For us, this issue was resolved with moving to 3.9.2.

I have closed it as it seems it was an "accidentally fixed" bug.

--
stage:  -> resolved
status: open -> closed

___
Python tracker 
<https://bugs.python.org/issue43351>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43816] Missing 'extern "C"' for _Py_ctype_table

[Andrew V. Jones]

New submission from Andrew V. Jones :

With Python 3.9.4, and when compiling with Visual Studio 2019, we have noticed 
that the variable `_Py_ctype_table` is *not* scoped with in an `extern "C"` 
block, and where the Python library (`python39.lib`) *has* been compiled with a 
C compiler.

This causes an issue when trying to refer to `_Py_ctype_table` from a C++ file, 
as the compiler tries to name-mangle the _use_ of `_Py_ctype_table`, but the 
linker cannot then tie the mangled name to non-mangled named from  
`python39.lib`.

Example:

```
#include "Python.h"
int main() { return _Py_ctype_table[0]; }
```

Compilation:

```
cl.exe /Fe:test.exe /TP /I include test.cpp /link libs/python39.lib
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29336 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

test.cpp
Microsoft (R) Incremental Linker Version 14.28.29336.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test.exe
libs/python39.lib
test.obj
test.obj : error LNK2019: unresolved external symbol "__declspec(dllimport) 
unsigned int const * const _Py_ctype_table" (__imp_?_Py_ctype_table@@3QBIB) 
referenced in function main
test.exe : fatal error LNK1120: 1 unresolved externals
```

With `cl.exe`:

```
cl.exe /Bv  
Microsoft (R) C/C++ Optimizing Compiler Version 19.28.29336 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

Compiler Passes:
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\cl.exe:
Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c1.dll:
Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c1xx.dll:  
Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c2.dll:
Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\c1xx.dll:  
Version 19.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\link.exe:  
Version 14.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\mspdb140.dll:  
Version 14.28.29336.0
 Z:\home\avj\visual_studio\MSVC\14.28.29333\bin\HostX64\x64\1033\clui.dll: 
Version 19.28.29336.0
```

A naïve check of Python.h (e126547c07) seems to suggest that:

* There are 82 includes

* 64 of these contain `extern "C"`

* 8 do not contain `extern "C"`

* The remaining 10 are either system includes or pyconfig.h

For the 8 that *do not* contain `extern "C"`, none of these use `PyAPI_DATA`. 
This leads me to believe that it is an oversight that `pyctype.h` does not have 
`extern "C"`

--
messages: 390855
nosy: andrewvaughanj
priority: normal
severity: normal
status: open
title: Missing 'extern "C"' for _Py_ctype_table
versions: Python 3.9

___
Python tracker 
<https://bugs.python.org/issue43816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43816] Missing 'extern "C"' for _Py_ctype_table

[Andrew V. Jones]

Change by Andrew V. Jones :


--
keywords: +patch
pull_requests: +24099
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/25365

___
Python tracker 
<https://bugs.python.org/issue43816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43816] Missing 'extern "C"' for _Py_ctype_table

[Andrew V. Jones]

Andrew V. Jones  added the comment:

> In fact, _Py_ctype_table is limited to the internal parts of the interpreter. 
> So in this case, this one could not be used in an external tool.
>

Hmm, so why is this "exposed" by the "world-facing" `Python.h` file?

I should say: we found this bug via Cython; and it was Cython that was 
accessing/referring to `_Py_ctype_table` -- our Cythonated code pulls in C++ 
headers, so we need to compile these files as C++.

I am happy to re-assign this as a Cython bug, but the fact it is fixed with an 
`extern "C"` in Python.h, really makes it feel like it is a Python-proper issue 
and not a "user" issue.

--

___
Python tracker 
<https://bugs.python.org/issue43816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43816] Missing 'extern "C"' for _Py_ctype_table

[Andrew V. Jones]

Andrew V. Jones  added the comment:

> I am happy to re-assign this as a Cython bug, but the fact it is fixed with 
> an `extern "C"` in Python.h, really makes it feel like it is a Python-proper 
> issue and not a "user" issue.
>

Just to extend on this:

1) The Cython-generated code uses `Py_ISSPACE` (and not `_Py_ctype_table`), 
but the expansion of the macro `Py_ISSPACE` then adds `_Py_ctype_table` to the 
user's code

2) The "user-fix" is to wrap `#include ` in `extern "C"` -- 
however, given other parts of Python.h already do this, it seems extraneous to 
expect a C++ user to wrap Python.h in this way

--

___
Python tracker 
<https://bugs.python.org/issue43816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43816] Missing 'extern "C"' for _Py_ctype_table

[Andrew V. Jones]

Andrew V. Jones  added the comment:

> 1) The Cython-generated code uses `Py_ISSPACE` (and not `_Py_ctype_table`), 
> but the expansion of the macro `Py_ISSPACE` then adds `_Py_ctype_table` to 
> the user's code
>

I wrote this up as a Cython bug here (just to see if the Cython team consider 
this "their" bug): https://github.com/cython/cython/issues/4111

--

___
Python tracker 
<https://bugs.python.org/issue43816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43816] Missing 'extern "C"' for _Py_ctype_table

[Andrew V. Jones]

Andrew V. Jones  added the comment:

> I think the fact that they've been moved to Include/cpython means that user
> code shouldn't be using them.
>

I think it is fine to say that they shouldn't be used, but then we get this 
from Victor's blog:

> It was decided that internal header files must not be included implicitly by
> the generic #include , but included explicitly. 
>

So, is it the case that we have two issues here:

1) Cython is using stuff it shouldn't (I can do a PR against Cython)

2) Python.h is exposing more than it should (so, if Python "core" wants 
something from pyctype.h, it should be explicitly including pyctype.h and not 
getting via Python.h)

?

--

___
Python tracker 
<https://bugs.python.org/issue43816>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43351] `RecursionError` during deallocation

[Andrew V. Jones]

New submission from Andrew V. Jones :

I am currently working with "porting" some code from Python 2.7.14 to Python 
3.7.5, but the process running the Python code seems to terminate in the 
following way:

```
#0  0x2ef63337 in raise () from /lib64/libc.so.6
#1  0x2ef64a28 in abort () from /lib64/libc.so.6
#2  0x2e726e18 in fatal_error (prefix=0x0, msg=0x2e8091f0 "Cannot 
recover from stack overflow.", status=-1) at Python/pylifecycle.c:2187
#3  0x2e727603 in Py_FatalError (msg=0x9bf0 ) at Python/pylifecycle.c:2197
#4  0x2e6ede2b in _Py_CheckRecursiveCall (where=) at 
Python/ceval.c:489
#5  0x2e62b61d in _PyMethodDef_RawFastCallDict (method=0x2eae2740 
, self=0x2aaabb1d4d70, args=0x0, nargs=0, 
kwargs=0x0) at Objects/call.c:464
#6  0x2e62b6a9 in _PyCFunction_FastCallDict (func=0x2aaabeaa5690, 
args=0x6, nargs=0, kwargs=0x0) at Objects/call.c:586
#7  0x2e62c56c in _PyObject_CallFunctionVa (callable=0x9bf0, 
format=, va=, is_size_t=) at 
Objects/call.c:935
#8  0x2e62cc80 in callmethod (is_size_t=, va=, format=, callable=) at Objects/call.c:1031
#9  _PyObject_CallMethodId (obj=, name=, 
format=0x0) at Objects/call.c:1100
#10 0x2e724c51 in flush_std_files () at Python/pylifecycle.c:1083
#11 0x2e72704f in fatal_error (prefix=0x0, msg=, 
status=-1) at Python/pylifecycle.c:2175
#12 0x2e727603 in Py_FatalError (msg=0x9bf0 ) at Python/pylifecycle.c:2197
#13 0x2e6ede2b in _Py_CheckRecursiveCall (where=) at 
Python/ceval.c:489
#14 0x2e62ba3d in _PyObject_FastCallDict (callable=0x2aaabeab8790, 
args=, nargs=, kwargs=0x0) at Objects/call.c:120
#15 0x2e62c2f0 in object_vacall (callable=0x2aaabeab8790, 
vargs=0x7ff54d40) at Objects/call.c:1202
#16 0x2e62c3fd in PyObject_CallFunctionObjArgs (callable=0x9bf0) at 
Objects/call.c:1267
#17 0x2e6c1bf0 in PyObject_ClearWeakRefs (object=) at 
Objects/weakrefobject.c:872
#18 0x2e4b26f6 in instance_dealloc () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
#19 0x2e67c3e0 in subtype_dealloc (self=0x2aaabeab9e40) at 
Objects/typeobject.c:1176
#20 0x2e4ba63f in life_support_call () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
#21 0x2e62b9c4 in _PyObject_FastCallDict (callable=0x2aaabeab87b0, 
args=, nargs=, kwargs=0x0) at Objects/call.c:125
#22 0x2e62c2f0 in object_vacall (callable=0x2aaabeab87b0, 
vargs=0x7ff54fd0) at Objects/call.c:1202
#23 0x2e62c3fd in PyObject_CallFunctionObjArgs (callable=0x9bf0) at 
Objects/call.c:1267
#24 0x2e6c1bf0 in PyObject_ClearWeakRefs (object=) at 
Objects/weakrefobject.c:872
#25 0x2e4b26f6 in instance_dealloc () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
#26 0x2e67c3e0 in subtype_dealloc (self=0x2aaabeab9e90) at 
Objects/typeobject.c:1176
#27 0x2e4ba63f in life_support_call () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
#28 0x2e62b9c4 in _PyObject_FastCallDict (callable=0x2aaabeab87d0, 
args=, nargs=, kwargs=0x0) at Objects/call.c:125
#29 0x2e62c2f0 in object_vacall (callable=0x2aaabeab87d0, 
vargs=0x7ff55260) at Objects/call.c:1202
#30 0x2e62c3fd in PyObject_CallFunctionObjArgs (callable=0x9bf0) at 
Objects/call.c:1267
#31 0x2e6c1bf0 in PyObject_ClearWeakRefs (object=) at 
Objects/weakrefobject.c:872
#32 0x2e4b26f6 in instance_dealloc () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
#33 0x2e67c3e0 in subtype_dealloc (self=0x2aaabeab9ee0) at 
Objects/typeobject.c:1176
#34 0x2e4ba63f in life_support_call () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
```

This is only the inner most 35 frames -- the actual back-trace is 7375 frames 
deep, and ends with:

```
#7358 0x2e4b26f6 in instance_dealloc () from 
/home/LOCAL/avj/build/vc21__90601_pyedg_improvements/vc/lib64/libboost_python37.so.1.69.0
#7359 0x2e67c3e0 in subtype_dealloc (self=0x2aaabeaefdf0) at 
Objects/typeobject.c:1176
#7360 0x2e6f2f46 in _PyEval_EvalFrameDefault (f=0x2aaabce48b30, 
throwflag=39920) at Python/ceval.c:1098
#7361 0x2e62a959 in function_code_fastcall (co=, 
args=0x7fffd088, nargs=1, globals=) at Objects/call.c:283
#7362 0x2e62ae44 in _PyFunction_FastCallDict (func=0x2aaabda07950, 
args=0x7fffd080, nargs=1, kwargs=0x0) at Objects/call.c:322
#7363 0x2e62bbea in _PyObject_Call_Prepend (callable=0x2aaabda07950, 
obj=0x2aaabea92590, args=0x2aaabb193050, kwargs=0x0) at Objects/call.c:908
#7364 0x2e62b9c4 in _PyObject_FastCallDict (callable=0x2aaabb253a50, 
args=, nargs=, kwargs=0x0) at Objects/call.c:125
#7365 0

[issue43351] `RecursionError` during deallocation

[Andrew V. Jones]

Andrew V. Jones  added the comment:

Here's some representative code that triggers the issue:

```
def loop():
a_node = boost_python_library.get_linked_list()
all_elems = []
while a_node is not None:
#
# Uncomment the below to make the crash disappear
#
# all_elems.append(a_node)
#
a_node = a_node.next
```

The *really* interesting bit is that if we save what comes off the Boost.Python 
linked list into a Python-proper list (`all_elems` as above), then the crash 
goes away.

--

___
Python tracker 
<https://bugs.python.org/issue43351>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue43351] `RecursionError` during deallocation

[Andrew V. Jones]

Andrew V. Jones  added the comment:

Same logic, but this crashes:

```
def loop():
a_node = boost_python_library.get_linked_list()
temp = []
while True:
assert a_node is not None
temp.append(a_node)
prev = a_node   # <-- comment this out to make the crash go away
a_node = a_node.next
if not a_node:
break
```

--

___
Python tracker 
<https://bugs.python.org/issue43351>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com