matanper opened a new issue, #70406: URL: https://github.com/apache/doris/issues/70406
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description In cloud mode, `CachedRemoteFileReader` fetches whole file-cache blocks from S3 (or from a peer compute group). A small request can therefore move much more data than it asked for. Block fills, read-ahead, dry runs, warm-up and the peer/S3 race all add to it. Today there is no way to see how much remote traffic each of these causes. The existing bvars count peer-race wins and losses (`peer_race_peer_win`, `peer_race_s3_win`, `peer_race_both_fail`, `peer_race_s3_skipped_after_queue`) but not bytes, and nothing splits S3 traffic by why it was read. When we measured this with ad-hoc counters, successful S3 payload was about two orders of magnitude above the bytes callers requested from the remote path. That held even while the file cache served almost all requested bytes, and the overhead was almost entirely block fill. Without per-purpose numbers you cannot tell whether to tune block size, read-ahead, warm-up or the peer race, or check that a change helped. ### Solution Add fixed-cardinality bvars (no per-tablet or per-query labels) in `CachedRemoteFileReader`, counted only for successful remote reads: 1. **S3 bytes and reads by purpose.** These partition every successful S3 read: `cached_remote_reader_s3_{demand,read_ahead,dryrun,warmup,bypass_peer}_{bytes,reads}`. 2. **S3 bytes by file kind.** An independent partition of the same bytes: `cached_remote_reader_s3_{segment_metadata,inverted_index,other}_bytes`. 3. **Peer race bytes**, which overlap the partitions above on purpose: `peer_race_{peer,s3}_{winner,loser}_bytes` (bytes fetched by the winning and losing side) and `peer_race_s3_skipped_queued_bytes` (bytes not fetched because a queued S3 hedge saw the peer had already won). Read counts exclude failed calls, and SDK-internal HTTP retries count once per call. The cost is two or three thread-local `bvar::Adder` adds per successful remote read. The result: requested vs fetched bytes per purpose and per kind, and the race overhead in bytes rather than wins, all from the standard bvar endpoint. ### Use case Diagnosing and tuning file-cache read amplification in cloud mode: block size, read-ahead, warm-up, peer read and hedging, plus checking any cache-policy change before and after. ### Related issues The peer/S3 race and the queued-hedge skip from #69620. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
