pingzh opened a new pull request, #5531: URL: https://github.com/apache/datafusion-comet/pull/5531
## Which issue does this PR close? Part of #5352. This is the seventh foundational PR and does not close the issue. Previous PRs: - https://github.com/apache/datafusion-comet/pull/5473 - https://github.com/apache/datafusion-comet/pull/5476 - https://github.com/apache/datafusion-comet/pull/5481 - https://github.com/apache/datafusion-comet/pull/5491 - https://github.com/apache/datafusion-comet/pull/5501 - https://github.com/apache/datafusion-comet/pull/5513 ## Rationale for this change The preceding PRs added native Celeborn map-side shuffle writes. Comet also needs a reduce-side reader that consumes native frames without Celeborn's row decompression, preserves AQE read ranges, and participates in Spark's fetch-failure recovery. ## What changes are included in this PR? - Add a shared local/remote reader interface and route native Celeborn handles through the application-owned client. - Read reducers lazily using one file-group snapshot, preserving shuffle generations, mapper ranges, attempt identities, and shuffle metrics. Support the stock Celeborn 0.6/0.7 raw-reader API and the newer coalesced-metadata overload without adding a Celeborn dependency. - Validate frame boundaries before concatenating reducers and preserve `FetchFailedException` across native codec/IPC decoding. Keep metadata timeouts, cancellation, and cleanup failures distinct from corrupt shuffle data. - Validate Arrow payloads on the remote path while retaining the existing local decoder entry point and fast path. - Release prefetched batches and unconsumed Arrow structures on early termination, EOF, and failure, preserving suppressed cleanup errors. Planner enablement remains for 8/n; this PR does not enable end-to-end Celeborn query planning. ## How are these changes tested? - 177 JVM tests passed across the Celeborn reader/pusher/manager/writer, native shuffle, configuration, and Arrow utility suites (Spark 4.1.3 / Scala 2.13). - 299 Rust unit tests passed across the core, shuffle, and JNI bridge crates. Four existing HDFS-dependent tests remain ignored. - Added regressions using real native frames and Arrow IPC for reducer boundaries, corrupt offsets, schema mismatches, JNI fetch-failure propagation, cancellation, and cleanup. - Seven isolated compatibility probes passed using published Celeborn 0.6.3/0.7.0 binaries. Reader/decoder source was also type-checked against Spark 3.4.3/3.5.9 and Scala 2.12 APIs; these probes are not complete legacy Maven builds or live Celeborn cluster tests. - Spotless, Scalastyle, Rust formatting, workspace/all-target Clippy with warnings denied, and Scala 2.12 parsing passed. - Three independent review passes completed with no outstanding P1/P2 findings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
