dylanetaft opened a new issue, #50715:
URL: https://github.com/apache/arrow/issues/50715
### Describe the bug, including details regarding any error messages,
version, and platform.
System-wide clang libc++ use has compatibility issues with pyarrow.
Disclosure - I investigated this with Grok 4.5, but made an effort to
understand the issue and limited changes it wanted to make. I will submit a PR
shortly that addresses this. It is very small, two lines, moves array dtor to
cpp file.
python -c "import pyarrow as pa
import pyarrow.parquet as pq
arr = pa.array([1], type=pa.int64())
table = pa.table([arr], names=['a'])
pq.write_table(table, '/tmp/t1.parquet')
print('1 OK')
"
libc++abi: terminating due to uncaught exception of type std::bad_cast:
std::bad_cast
Aborted (core dumped) python -c "import pyarrow as pa
tldr; Python RTLD_LOCAL + Arrow’s header-inline virtual destructor results
in duplicate array typeinfo in several DSOs; without NDEBUG, parquet’s
dynamic_cast fails under libc++ when used in pyarrow.
Python defaults to loading shared libraries with RTLD_LOCAL. Several Apache
Arrow headers contain inline defined virtuals in the headers instead of the
source files, which leads to technically separate class footprints for each
translation unit that includes the header, RTTI is not unified across DSOs.
libc++ is very particular with dynamic_cast, where libstdc++ is more relaxed.
When apache arrow is compiled without DNDEBUG, a code path in VisitArrayInline
is enabled that uses c++ dynamic casts. Pyarrow has multiple shared objects
loaded that use arrow data types, combined with RTLD_LOCAL - this leads to
objects that actually cannot dynamic_cast under clang's libc++ - dynamic_cast
will see different type identities.
This is reproducible under a Gentoo system compiled entirely with clang +
libc++, plus all libraries compiled against libc++, with pyarrow.
Other workarounds:
This will basically put libarrow symbols in the global namespace
circumventing the issue as a workaround
LD_PRELOAD=/usr/lib64/libarrow.so.2500:/usr/lib64/libparquet.so.2500 python
yourapp.py
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
