yujun777 commented on PR #69456: URL: https://github.com/apache/doris/pull/69456#issuecomment-6494658210
Memory of the final top-K state, for `min_topk` against `MIN`. Setup: release build, single BE restarted before every query (cold page cache on both sides), `enable_sql_cache=false`, `exec_mem_limit` raised, RSS peak delta of one query. Two value shapes per type: 10M groups with 1 row each (10M rows) and 10M groups with 8 rows each (80M rows). | value type | rows per group | total rows | `MIN(v)` | `min_topk(v, 8)` | state per group | | --- | --- | --- | --- | --- | --- | | BIGINT | 1 | 10M | 999 MB | 1687 MB | ~69 B | | BIGINT | 8 | 80M | 1886 MB | 5038 MB | ~315 B | | VARCHAR | 1 | 10M | 1501 MB | 2987 MB | ~149 B | | VARCHAR | 8 | 80M | 2742 MB | 8214 MB | ~547 B | "state per group" is the `min_topk` query minus the `MIN` query of the same shape, divided by the group count. Query profile cross-check (10M groups, 1 row per group, per-operator `MemoryUsagePeak`): the aggregation sink peaks at 774 MB for `MIN(v)` and 1.25 GB for `min_topk(v, 8)` on BIGINT, and at 2.35 GB for `min_topk(v, 8)` on VARCHAR. The profile counts the state record but not the slot vector's payload, so it reads lower than RSS at 8 values per group. What the state costs: 56 bytes of fixed part per group (k, the slot vector header, the cached worst slot, the live/dead counters and the priority seed) plus 32 bytes per retained value for BIGINT and 40 bytes for VARCHAR (count, both treap links, the priority and the value or its span). A `MIN` state is a single 8-byte value, so the comparison is dominated by that fixed part at one value per group and by the slots at eight. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
