peterxcli opened a new pull request, #5036: URL: https://github.com/apache/datafusion-comet/pull/5036
## Which issue does this PR close? Closes #5023. ## Rationale for this change Vendored libhdfs detaches every thread with a cached `JNIEnv`, including threads that were already attached by the JVM. During Spark worker teardown, that `JNIEnv` can already be freed when `hdfsThreadDestructor` calls through it, causing a jump to address zero. HDFS-16021 describes the same ownership bug, but no released `hdfs-sys` version contains the fix. ## What changes are included in this PR? Patch crates.io resolution at the Cargo workspace level to pin `hdfs-sys` to an immutable commit carrying the ownership fix. The workspace patch makes both Comet's direct dependency and the OpenDAL/HDFS path through `hdrs` use the same fixed crate. ## How are these changes tested? - `cargo build --locked --profile ci` on macOS ARM64 with JDK 17 - `cargo tree --locked -p datafusion-comet -i hdfs-sys -e features` confirms one Git-sourced `hdfs-sys` instance with `hdfs_3_3` and `vendored` - 100 fresh-JVM runs of `ParquetReadFromFakeHadoopFsSuite` against the patch, followed by 10 fresh-JVM runs against the exact GitHub-pinned artifact - Disassembly confirms `hdfsThreadDestructor` checks libhdfs ownership before dereferencing `JNIEnv` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
