yujun777 commented on PR #69456: URL: https://github.com/apache/doris/pull/69456#issuecomment-6487922191
Updated the top-K state layout: a group's value now lives in its slot instead of in a second vector, so a state owns one slot array (72 bytes and one heap allocation per group for fixed-width types, was 96 bytes and two allocations). The string case loses its separate span vector the same way. The serialized state format is unchanged. Memory of `min_topk` vs `MIN` on a release build: single BE, 10,000,000 rows, RSS peak delta of one query (`enable_sql_cache=false`), second of two consecutive rounds in the same session. BIGINT (`v` BIGINT, 10M rows): | rows per group | groups | query | before | after | | --- | --- | --- | --- | --- | | 1 | 10M | `MIN(v)` | 863 MB | 841 MB | | 1 | 10M | `min_topk(v, 8)` | 1896 MB | 1594 MB | | 8 | 1.25M | `MIN(v)` | 93 MB | 89 MB | | 8 | 1.25M | `min_topk(v, 8)` | 487 MB | 459 MB | VARCHAR (`v` VARCHAR, 10M rows): | rows per group | groups | query | before | after | | --- | --- | --- | --- | --- | | 1 | 10M | `MIN(v)` | 1403 MB | 1426 MB | | 1 | 10M | `min_topk(v, 8)` | 3069 MB | 2847 MB | | 8 | 1.25M | `MIN(v)` | 153 MB | 153 MB | | 8 | 1.25M | `min_topk(v, 8)` | 889 MB | 833 MB | The state kept on top of `MIN` (query minus `MIN`, divided by the group count): | value type | rows per group | before | after | | --- | --- | --- | --- | | BIGINT | 1 | ~103 B/group | ~75 B/group | | BIGINT | 8 | ~315 B/group | ~296 B/group | | VARCHAR | 1 | ~167 B/group | ~142 B/group | | VARCHAR | 8 | ~589 B/group | ~544 B/group | Reading these numbers: at one value per group the per-group overhead is what changes, so the state drops by a quarter; at eight values per group the slots themselves dominate and the saving is the removed allocation and the 24-byte smaller state, roughly 5-8%. VARCHAR pays more than BIGINT per group at the same value count because a slot carries a 16-byte span instead of an 8-byte value and the state carries the arena header. Tests on the updated code: `be/test/exprs/aggregate/agg_topk_test.cpp` (18 cases) and `regression-test/suites/query_p0/aggregate/min_max_topk.groovy` pass, on both the debug/ASAN and the release build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
