CadyLinn opened a new pull request, #23490:
URL: https://github.com/apache/datafusion/pull/23490
## Which issue does this PR close?
- Closes #20644.
## Rationale for this change
CSV and JSON sinks do not currently expose write metrics through
`DataSinkExec`, while the Parquet sink exposes `rows_written`, `bytes_written`,
and `elapsed_compute`.
This change provides consistent metrics for CSV and JSON sinks and
centralizes the common file sink metrics wiring to reduce duplication across
file formats.
## What changes are included in this PR?
- Add a shared `FileSinkMetrics` implementation for `rows_written`,
`bytes_written`, and `elapsed_compute`.
- Use the shared metrics implementation for Parquet, CSV, and JSON sinks.
- Track rows and serialized bytes in the common stateless write
orchestration.
- Preserve the existing `spawn_writer_tasks_and_join` API and add an opt-in
metrics variant.
- Add SQL logic test coverage for `EXPLAIN ANALYZE COPY` with CSV and JSON
output.
## Are these changes tested?
Yes. The affected datasource unit tests and the `copy.slt` SQL logic test
pass.
```bash
cargo test -p datafusion-datasource \
-p datafusion-datasource-csv \
-p datafusion-datasource-json \
-p datafusion-datasource-parquet --lib
```
```bash
cargo test -p datafusion-sqllogictest \
--test sqllogictests -- copy.slt --complete
```
The affected crates also pass targeted Clippy checks with warnings denied.
## Are there any user-facing changes?
Yes. `EXPLAIN ANALYZE COPY` now reports `rows_written`, `bytes_written`, and
`elapsed_compute` for CSV and JSON sinks, consistent with the existing Parquet
sink metrics.
There are no breaking public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
