sunchao opened a new pull request, #5370:
URL: https://github.com/apache/datafusion-comet/pull/5370
## Why are the changes needed?
Native shuffle write metrics were incomplete and could misrepresent spill
behavior in Spark's SQL and task UIs:
- Native partition interleaving time was not exposed as a SQL metric.
- Spill bytes emitted only when buffered Arrow batches were flushed were
omitted from the disk-spill counter.
- The same compressed disk byte count was reported as both memory bytes
spilled and disk bytes spilled.
- Repeated spills of slices sharing one Arrow allocation could overstate
in-memory spill volume.
- Failed shuffle attempts lost task-level spill metrics because they never
reached the successful writer accounting path.
Part of #3996 (native shuffle write direction only).
## What changes were proposed in this PR?
- Expose native `interleave_time` and a new `memory_spilled_bytes` metric in
the Spark SQL shuffle exchange.
- Measure physical spill-file growth after flushing buffered Arrow batches
so compressed disk-spill metrics include the complete bytes written.
- Track in-memory spill bytes separately from compressed disk bytes,
including partition-index allocations and batches whose memory reservation was
rejected.
- Deduplicate Arrow backing allocations across repeated spills within one
outer input batch while preserving cumulative accounting for separate input
batches.
- Transfer the final native memory and disk spill totals into their
corresponding Spark task metrics from an ordered task-completion listener,
covering both successful and failed shuffle attempts.
- Add focused Rust regressions for forced spills, rejected reservations,
no-spill behavior, repeated shared buffers, and physical spill-file sizes; add
Spark SQL/task UI regressions for successful and failed native shuffle writes.
## How was this PR tested?
```bash
cd native
cargo fmt --all -- --check
cargo clippy -p datafusion-comet-shuffle --lib -- -D warnings
cargo test -p datafusion-comet-shuffle --lib spill -- --nocapture
cargo test -p datafusion-comet-shuffle --lib
shared_backing_once_per_input_batch -- --nocapture
cd ..
make core
./mvnw -Pspark-3.5 test -Dtest=none \
'-Dsuites=org.apache.spark.sql.comet.CometTaskMetricsSuite memory and disk
spill metrics'
./mvnw -Pspark-4.0 test -Dtest=none \
'-Dsuites=org.apache.spark.sql.comet.CometTaskMetricsSuite memory and disk
spill metrics'
```
Both Spark profiles passed the successful-write and failed-attempt shuffle
spill metric regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
