yujun777 commented on PR #69456: URL: https://github.com/apache/doris/pull/69456#issuecomment-6490977271
Reference data: what the query profile and the audit log report for the same queries, next to the RSS numbers from the previous comment. These are not the numbers the change is justified with — the profile does not see the per-slot payload the change removes — but they are useful as an independent view. Setup: release build, single BE, `enable_profile=true` / default audit logging, `enable_sql_cache=false`, 10,000,000 rows. **Query profile, per-operator `MemoryUsagePeak` (after the change)** | value type | rows per group | groups | operator | `MIN(v)` | `min_topk(v, 8)` | | --- | --- | --- | --- | --- | --- | | BIGINT | 1 | 10M | `AGGREGATION_SINK_OPERATOR` | 774.53 MB | 1.50 GB | | BIGINT | 8 | 1.25M | `BUCKETED_AGGREGATION_SINK_OPERATOR` | 95.99 MB | 159.99 MB | | VARCHAR | 8 | 1.25M | `BUCKETED_AGGREGATION_SINK_OPERATOR` | 159.99 MB | 287.99 MB | **Audit log `PeakMemoryBytes`**: the 1.25M-group BIGINT `min_topk(v, 8)` query logs `32768128` bytes (32 MB), while its RSS peak delta is ~460 MB, so the audit figure does not cover the aggregation state. **Why the profile number is only a partial view.** At 1.25M groups the same query reports exactly `159.99 MB` for both `min_topk(v, 1)` and `min_topk(v, 8)`, even though the second state holds eight slots per group instead of one. The operator counter therefore captures the arena-allocated state record but not the slot vector's payload, which is exactly what this change shrinks. In the 10M-group case, where the state record dominates, the profile and the RSS figures agree (774 MB vs 841 MB for `MIN`, 1.50 GB vs 1.59 GB for `min_topk`). For comparison, the same queries under RSS sampling (from the previous comment, second of two rounds): | value type | rows per group | `MIN(v)` | `min_topk(v, 8)` | | --- | --- | --- | --- | | BIGINT | 1 | 841 MB | 1594 MB | | BIGINT | 8 | 89 MB | 459 MB | | VARCHAR | 1 | 1426 MB | 2847 MB | | VARCHAR | 8 | 153 MB | 833 MB | RSS additionally responds to `k` where the profile does not: at 1.25M BIGINT groups, `min_topk(v, 1)` peaks at 298/302 MB and `min_topk(v, 8)` at 455/455 MB across two rounds, which is the expected cost of the extra slots. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
