jbonofre opened a new issue, #1455: URL: https://github.com/apache/arrow-java/issues/1455
### Describe the bug, including details regarding any error messages, version, and platform. The RC workflow's Binaries job intermittently fails because the JVM running the arrow-dataset tests crashes. Surefire reports `TestAllTypes` as the crashed test (`Process Exit Code: 134`, "The forked VM terminated without properly saying goodbye"), and sometimes `TestAllTypes` fails with a memory leak instead. It has failed about 1 in 8 push runs on `main` since at least Aug 25. Recent examples: - `main`: https://github.com/apache/arrow-java/actions/runs/42614743276/job/129174599773 - #1453: https://github.com/apache/arrow-java/actions/runs/42576225131/job/129026485158 - #1452 (leak variant): https://github.com/apache/arrow-java/actions/runs/42576170397/job/129025668681 The crash comes from `TestReservationListener#testErrorThrownFromReservationListener`, which runs just before `TestAllTypes` in the same JVM. The three `hs_err` files uploaded so far (#1428) show the same stack on an Arrow C++ thread-pool thread. Symbolized against the job's `libarrow_dataset_jni.so`: ``` SIGSEGV (0xb) ... si_code: 2 (SEGV_ACCERR) <- call to a heap address arrow::PoolBuffer::Reserve(long)+0x2d3 arrow::PoolBuffer::Resize(long, bool)+0x5f arrow::PoolBuffer::MakeUnique(arrow::MemoryPool*, long)+0x67 arrow::AllocateBuffer(long, long, arrow::MemoryPool*)+0x3b ``` `pool->Allocate()` calls through the vtable of a `MemoryPool` that has been freed. In each file, the last JVM event before the crash, 2 to 5 ms earlier, is the `IllegalArgumentException` that the test's listener throws, rethrown through JNI. What happens: 1. The listener throws from `reserve()`, so the scan fails. 2. `TestDataset#collectResultFromFactory` rethrows without closing the scanner or the dataset. 3. The test closes the `NativeMemoryPool`, and `releaseMemoryPool` deletes the native `ReservationListenableMemoryPool`. 4. Arrow C++ is still running the failed scan's async work (Parquet file opens, read-ahead) on its thread pools. That work holds a raw pointer to the pool and allocates from it or frees to it after it's deleted. I reproduced this on macOS arm64 with the `jni-macos-aarch_64` libraries from the RC workflow, looping the test's steps up to 1,000 times per JVM: - Pool closed after each scan, as the test does now: 5 of 6 runs failed. The JVM crashed (in `PoolBuffer::Resize`, or an abort in `ReservationListenableMemoryPool::Impl::Free` while a Parquet reader was destroyed on an IO thread), or a scan whose listener always throws completed without an error. - Scanner and dataset closed before the pool: 6 of 8 runs failed the same ways, including a crash with the `PoolBuffer` stack above. Closing the scanner doesn't wait for that work. - Pool left open, with or without closing the scanner: 10 of 10 runs passed. This issue covers the test. A listenable pool being freed while native scan work still uses it is a library problem, and I'll add the native-side details to #246, which reports sporadic failures and crashes in this test class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
