L3o-pold opened a new issue, #50471:
URL: https://github.com/apache/arrow/issues/50471
### Describe the bug, including details regarding any error messages,
version, and platform
pyarrow 25.0.0 segfaults deterministically when `pyarrow` is first imported
on a **non-main thread that later exits**, and a different thread then performs
its first Arrow allocation. The crash is inside the bundled mimalloc's thread
initialization:
```
Exception Type: EXC_BAD_ACCESS (SIGSEGV) KERN_INVALID_ADDRESS at
0x0000000000000018
libarrow.2500.dylib mi_heap_main
libarrow.2500.dylib mi_thread_init
libarrow.2500.dylib _mi_malloc_generic
libarrow.2500.dylib mi_theap_malloc_zero_aligned_at_generic
libarrow.2500.dylib arrow::(anonymous
namespace)::MimallocAllocator::AllocateAligned(long long, long long, unsigned
char**)
libarrow.2500.dylib arrow::BaseMemoryPoolImpl<arrow::(anonymous
namespace)::MimallocAllocator>::Allocate(...)
libarrow.2500.dylib arrow::PoolBuffer::Reserve(long long)
libarrow.2500.dylib arrow::PoolBuffer::Resize(long long, bool)
libarrow_python.2500.dylib arrow::py::ConvertPySequence /
arrow::py::NdarrayToArrow
```
#### Minimal reproduction
```python
import threading
def imp():
import pyarrow # dlopens libarrow -> bundled mimalloc initializes on
this thread
def alloc():
import pyarrow as pa
pa.array(["x"] * 5000)
a = threading.Thread(target=imp); a.start(); a.join()
b = threading.Thread(target=alloc); b.start(); b.join()
```
Result: SIGSEGV (exit code 139). The same script:
- does **not** crash if `import pyarrow` happens on the main thread first;
- does **not** crash with `ARROW_DEFAULT_MEMORY_POOL=system`;
- does **not** crash on pyarrow 24.0.0.
#### Root cause
This is upstream mimalloc bug microsoft/mimalloc#1287 ("mimalloc >= 3.3.0
causes segmentation faults when used from multiple threads", closed
2026-06-20): mimalloc treats the first thread that initializes it as the
process main thread; when that thread exits, subsequent threads crash in
`mi_thread_init`/`_mi_malloc_generic`. mimalloc 3.2.x is unaffected.
Arrow 25.0.0 is the first release affected because GH-49772 / PR #49801
bumped `ARROW_MIMALLOC_BUILD_VERSION` from v3.2.7 to v3.3.1 (confirmed in
`cpp/thirdparty/versions.txt` at tag `apache-arrow-25.0.0`).
The upstream fix exists only on mimalloc's `dev3` branch (commit
microsoft/mimalloc@b92c116b67d0, "allow initial main thread to terminate before
the process terminates (see issue #1287)"); the latest tagged mimalloc release
(v3.3.2, 2026-04-29) predates it, so there is currently no tagged mimalloc
version Arrow could bump to.
#### Real-world impact
This import pattern is not exotic — it is structurally guaranteed by
Streamlit + pandas 3:
- pandas 3.x imports pyarrow eagerly (default pyarrow-backed `str` dtype);
- Streamlit executes user scripts (including their imports) on short-lived
`ScriptRunner` threads, never on the main thread.
So in any Streamlit app using pandas 3 + pyarrow 25, libarrow first loads on
script-run thread #1, and the next script run (a simple page reload) segfaults
the whole server on its first string-column conversion (`pandas
maybe_convert_objects` → `arrow::py::NdarrayToArrow`). We observed 7+ identical
crashes per day on one such app before diagnosing.
#### Workarounds
- `ARROW_DEFAULT_MEMORY_POOL=system` (verified);
- `import pyarrow` on the main thread before any worker thread touches it
(verified);
- pin `pyarrow<=24.0.0` (bundles unaffected mimalloc v3.2.7).
#### Suggested resolution
Consider reverting the bundled mimalloc to the v3.2.x line until a tagged
mimalloc release contains the #1287 fix, or picking up that fix as a patch on
top of v3.3.1.
Related but distinct: #50428 (interpreter-teardown crash when co-loading
another mimalloc-v3-bundling extension; the `ARROW_DEFAULT_MEMORY_POOL=system`
workaround reportedly does not help there, whereas it does here).
#### Environment
- pyarrow 25.0.0 (PyPI wheel), pandas 3.0.3, streamlit 1.59.1
- CPython 3.14.5 (Homebrew), macOS 26 (Darwin 25.5.0), Apple Silicon
- Note: mimalloc#1287 was reproduced upstream on Linux and Windows as well,
so this is unlikely to be macOS-specific.
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
