sunchao commented on PR #5370: URL: https://github.com/apache/datafusion-comet/pull/5370#issuecomment-5309712567
Thanks for the detailed review, @andygrove — addressed all five points: 1. Moved the task-metric bridge into `CometMetricNode.reportSpillMetrics`, preserving completion-listener ordering for both successful and failed attempts. 2. Renamed the SQL UI label to `disk spilled bytes` while keeping the native `spilled_bytes` metric key unchanged. 3. Updated the metrics guide to cover partition interleaving, disk-versus-memory spill metrics, Spark task-metric semantics, and compression-disabled shuffles. 4. Opened #5382 to track spill accounting for native operators inlined beneath the shuffle writer. 5. Removed the unused `COMET_SHUFFLE_JVM_BATCH_SIZE` setting from the failed-attempt regression. The focused successful- and failed-attempt spill regressions pass against both Spark 3.5 and Spark 4.0. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
