comphead commented on issue #5023: URL: https://github.com/apache/datafusion-comet/issues/5023#issuecomment-5609752380
Adding a **Linux/x86-64/JDK 11** occurrence. This issue is titled and evidenced as macOS, but the same crash reproduces on `ubuntu-24.04`, which I think widens the scope rather than adding a duplicate. Run: [34404348583](https://github.com/apache/datafusion-comet/actions/runs/34404348583/job/102654024260), job `PR Build (Linux) / Spark 3.4, JDK 11, Scala 2.12 [scans]`, on an unrelated CI-config PR (#5782). ### Same signature ``` SIGSEGV (0xb) at pc=0x0000000000000000, pid=1317, tid=4739 JRE: Zulu11.90+19-CA (11.0.32+9), linux-amd64 (vs Zulu17 / bsd-aarch64 above) Problematic frame: C 0x0000000000000000 Current thread is native thread siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000 ``` Consistent with @peterxcli's `lr = hdfsThreadDestructor+80`, `x8 = 0` on aarch64: - `RIP = 0x0`, `si_addr = 0x0` — jump to address zero. - `RDI == R15 == 0x00007f6d4003c350`. On the SysV x86-64 ABI `RDI` is the first argument, which is the `JNIEnv*` position for `(*env)->GetJavaVM(env, ...)`. - Stack used is ~4.8 KB of a 4 MB thread stack, i.e. a handful of frames from thread entry. That is what a pthread TLS destructor at thread exit looks like. - The return address on top of stack, `0x00007f6ccbc978df`, and a deeper entry, `0x00007f6cd0b6f47e`, both resolve into the `r-xp` text segment of `/spark/target/tmp/libcomet-4125068328974464746.so` (base `0x7f6cca000000`, so file offsets `0x1c978df` and `0x6b6f47e`). One caveat: I could **not** symbolize those offsets, so the `hdfsThreadDestructor` attribution here is by signature and register shape, not by symbol. The dump is also missing its thread list entirely, because error reporting itself SIGSEGV'd twice inside `libjvm.so`: ``` [error occurred during error reporting (printing register info), id 0xb, SIGSEGV (0xb) at pc=0x00007f6d90c22238] [error occurred during error reporting (inspecting top of stack), id 0xb, SIGSEGV (0xb) at pc=0x00007f6d90c22238] [thread 17129 also had an error] ``` Not resource exhaustion: G1 heap was 727 MB used of 1432 MB committed against `-Xmx4g`, host had 10.1 GB `MemAvailable` and completely untouched swap. ### Why it is not macOS-specific `native/core/Cargo.toml`: ```toml default = ["hdfs-opendal"] hdfs-opendal = ["opendal", "object_store_opendal", "hdfs-sys"] ``` `hdfs-sys` (with `hdfs_3_3`, vendoring the libhdfs C that contains the destructor) is a **default feature on every platform**. Only `hdrs = { features = ["vendored"] }` is macOS-gated. So every default `libcomet` build carries the vulnerable `hdfsThreadDestructor`, and `ParquetReadFromFakeHadoopFsSuite` is in the `[scans]` bucket of both `pr_build_linux.yml` and `pr_build_macos.yml`. ### The destructor fires in a later suite than the one that arms it This may be the more actionable part. In the macOS reports the crash lands right at the `ParquetReadFromFakeHadoopFsSuite` to `ParquetTimestampLtzAsNtzSuite` boundary. Here it landed **4m03s into the fork, during `CometIcebergNativeSuite`**, long after the fake-fs suite finished. The `hs_err` command line confirms both suites were in the same JVM, and `SparkTestSuite.txt` shows the order: ``` CometParquetWriterSuite: → ParquetReadV1Suite: → ParquetReadFromFakeHadoopFsSuite: → ParquetTimestampLtzAsNtzSuite: → CometNativeReaderSuite: → CometIcebergNativeSuite: ← crash here ``` Last test to complete was `runtime filtering - join with dynamic partition pruning`. So the failure is not bounded to the suite that registers the `fake` libhdfs scheme. That suite attaches libhdfs to whatever pooled threads it touches, and the TLS destructor only runs when one of those threads eventually exits, which can be minutes later in an unrelated suite. Any suite sharing the fork is a candidate crash site, which is probably why this reads as a random `[scans]` flake rather than something pinned to one test. Incidental, and I do not think it is causal, but recording it: the final 9 seconds before the crash show 1269 `CodecPool: Got brand-new compressor [.zstd]` at ~100/s on a single executor task, meaning compressors are never being returned to the pool. Possibly a separate minor leak. ### Repro artifacts The run's `crash-logs-*`, `unit-tests-*` and `java-test-reports-*` artifacts for that job carry `hs_err_pid1317.log`, `unit-tests.log` and the surefire output, and `native-lib-linux` from the same run is the exact binary that crashed, so the offsets above should symbolize directly: ```bash gh run download 34404348583 --repo apache/datafusion-comet -p '*3.4*scans*' gh run download 34404348583 --repo apache/datafusion-comet -n native-lib-linux addr2line -f -C -e libcomet.so 0x1c978df 0x6b6f47e ``` Nothing here changes the recommended fix; it just argues the title should lose "macOS" and that the priority is a bit higher than a single-platform flake. _Analysis assisted by LLM._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
