sunchao opened a new issue, #5508: URL: https://github.com/apache/datafusion-comet/issues/5508
### What is the problem the feature request solves? Native Parquet scans expose `bytes_scanned`, but that counter does not distinguish projected data from footer, page-index, or Bloom-filter reads, and it does not reveal the larger GET ranges produced by object-store coalescing. A metadata-only scan and a warm metadata-cache hit can therefore be difficult to distinguish in the Spark SQL UI. For example, two logical 64-byte reads can be serviced by one coalesced 524,416-byte GET. Both paths can report the same 128 logical bytes while placing very different demands on storage. ### Describe the potential solution Expose counters at the Parquet reader, remote object-store API, and metadata-cache boundaries, and propagate them to Spark SQL metrics. Document which byte counters overlap, the difference between requested and consumed bytes, encrypted-footer behavior, backend scope, and the limitations of cancellation snapshots. Preserve the existing `bytes_scanned` meaning and correct documentation that describes it as complete filesystem traffic. Tests should cover coalescing, projection and pruning, cache hits and misses, encrypted and malformed footers, partial response consumption, and real delegation to local/custom range readers. Measure the additional SQL accumulator work on a job with many tasks. ### Additional context This tracks the metrics-only scope of #5453. Object-store registration isolation is tracked separately in #5502 and implemented by prerequisite #5503. Producer cancellation and final-snapshot waiting are tracked separately in #5504 and implemented by #5505; they are not part of this metrics request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
