andygrove opened a new pull request, #6275: URL: https://github.com/apache/datafusion-comet/pull/6275
Backport of #5265 to `branch-1.0`. Cherry-picked from `73e8dacbc7b52763645cb31ddef924ddf8198790` without conflicts. No adaptations were needed. ## Which issue does this PR close? Closes #5264 on `branch-1.0`. #5265 is in the 1.0.1 milestone. Listed in #6201. ## Rationale for this change The bug ships in 1.0.0. On `branch-1.0`, `CometNativeExec` reports scan input metrics only when one of its Spark plans is a `CometNativeScanExec`. `CometIcebergNativeScanExec` tracks `bytes_scanned`, but that flag gates the reporting. So a native Iceberg scan that is fused into a larger native block, or that feeds a native shuffle writer, leaves Spark's task input metrics at zero. The Input column on the Spark UI's Stages and Executors tabs then reads 0.0 B. ## What changes are included in this PR? The fix is the original one; see #5265 for the details. `CometNativeExec` now checks for any `CometLeafExec` rather than only `CometNativeScanExec`. `reportScanInputMetrics` already skips leaves that have no `bytes_scanned` metric, such as `CometCsvNativeScanExec`, and it is the same on `branch-1.0` as on `main`. The input-metrics tests in `CometIcebergNativeSuite` move onto shared helpers and gain two cases. ## How are these changes tested? The tests in `CometIcebergNativeSuite`, run locally on `branch-1.0` with the default Spark 4.1 profile, Iceberg 1.11.0 and JDK 17: - The three input-metrics tests pass, including the two new ones for a fused scan and for a scan feeding a native shuffle. - The bug is present on `branch-1.0`, and the tests catch it. With `operators.scala` reverted and the tests kept, both new tests fail with `bytesRead should be > 0, got 0`. - The full `CometIcebergNativeSuite` passes, 99 tests. - Scalastyle, through `test-compile` on the default profile, Spotless, and scalafix in CHECK mode on Spark 3.5 pass. I ran them once with all six `branch-1.0` backports from this round applied together. ## Are there any user-facing changes? The Input column in the Spark UI now shows the bytes read by native Iceberg scans, including fused ones and ones that feed a native shuffle. There are no config or API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
