BiteTheDDDDt opened a new pull request, #61104: URL: https://github.com/apache/doris/pull/61104
This pull request introduces an optimization for streaming aggregation operators by detecting when a query is assigned to a single backend and applying more aggressive hash table expansion thresholds in that case. The change propagates a new `single_backend_query` flag from the frontend to the backend, and both the `streaming_aggregation_operator` and `distinct_streaming_aggregation_operator` use this flag to select different hash table reduction thresholds, potentially improving performance for single-backend queries. **Single-backend query detection and propagation:** - Added a `single_backend_query` field to `TQueryOptions` and set it in the FE (`CoordinatorContext`, `Coordinator`, and `ThriftPlansBuilder`) when all fragments are assigned to a single backend. [[1]](diffhunk://#diff-e23e53cda5dd2228558b7c3d07ed85b76911e3962c1b6eb46108340966179a5eR443-R447) [[2]](diffhunk://#diff-1e3445c79eb4d715c96a128cb34afc75943f324edf3a28eeffe791c5f72db7a3R106) [[3]](diffhunk://#diff-1e3445c79eb4d715c96a128cb34afc75943f324edf3a28eeffe791c5f72db7a3R451-R454) [[4]](diffhunk://#diff-45871b12ae79fb05592420c1f6399b3e5a5836873690278ec917403451605b94R109-R113) [[5]](diffhunk://#diff-1065d3f88b9a3b4bb8f151cac8514ebd39085c9dc2f0081b0eec43fd67ad7b74R838) [[6]](diffhunk://#diff-1065d3f88b9a3b4bb8f151cac8514ebd39085c9dc2f0081b0eec43fd67ad7b74R875) - The BE receives and stores this flag in `QueryContext`, with new getter and setter methods and a backing field. [[1]](diffhunk://#diff-3be0bcdb0c3c5fe6415556cf49c7270d4fc2e2f071c0240145ce175a32a484e2R926-R928) [[2]](diffhunk://#diff-b6e358e9c765ec5d14aa33dfeab3f20288e26bf636e4da72161aac4a42c82923R112-R117) [[3]](diffhunk://#diff-b6e358e9c765ec5d14aa33dfeab3f20288e26bf636e4da72161aac4a42c82923R397-R398) **Streaming aggregation operator optimization:** - Defined new, more aggressive hash table expansion thresholds for single-backend queries (`SINGLE_BE_STREAMING_HT_MIN_REDUCTION`) in both `streaming_aggregation_operator.cpp` and `distinct_streaming_aggregation_operator.cpp`. [[1]](diffhunk://#diff-5f2882c1f711fc0954459c6b98a1dcde9b688bb5634be71ce6f585332d8b6497L69-R86) [[2]](diffhunk://#diff-e78adff1d9b4effff7657f3966e776bf9e8acd9dee58f69df3aa2692a450501aR56-R66) - The local state for both streaming aggregation operators now checks the `single_backend_query` flag and uses the appropriate threshold table to decide when to expand hash tables. [[1]](diffhunk://#diff-5f2882c1f711fc0954459c6b98a1dcde9b688bb5634be71ce6f585332d8b6497R95) [[2]](diffhunk://#diff-81893b8013f0ade85a4aca0bcf807655fc08a57e7a6cc8531278cdc5330fae9eR119) [[3]](diffhunk://#diff-e78adff1d9b4effff7657f3966e776bf9e8acd9dee58f69df3aa2692a450501aL65-R77) [[4]](diffhunk://#diff-0098f1d5348adf3114a99bf8e273b95e68e137e3d45a1c89e5db06134a63adb2R90-R91) - Logic in the hash table expansion decision method selects the correct reduction table based on the flag, ensuring the optimization is applied only for single-backend queries. [[1]](diffhunk://#diff-5f2882c1f711fc0954459c6b98a1dcde9b688bb5634be71ce6f585332d8b6497R251-R258) [[2]](diffhunk://#diff-5f2882c1f711fc0954459c6b98a1dcde9b688bb5634be71ce6f585332d8b6497L271-R287) [[3]](diffhunk://#diff-e78adff1d9b4effff7657f3966e776bf9e8acd9dee58f69df3aa2692a450501aR128-R135) [[4]](diffhunk://#diff-e78adff1d9b4effff7657f3966e776bf9e8acd9dee58f69df3aa2692a450501aL148-R164) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
