924060929 opened a new pull request, #67177:
URL: https://github.com/apache/doris/pull/67177
### What problem does this PR solve?
Issue Number: None
Related PR: #65835 addresses a different local-shuffle distribution mismatch
and is not a dependency.
Problem Summary:
Scalar DISTINCT aggregation can make a pipeline serial while its join and
remote Exchange subtree remains parallel. FE still addresses every remote
receiver task, but BE previously inherited the serial task count into that
subtree. On a four-BE cluster with parallel_pipeline_task_num=3, FE created
twelve receiver destinations while BE created only four live receivers, so rows
sent to the remaining receiver IDs were lost and RQG queries returned incorrect
results.
This change makes the parallel-to-serial transition explicit. FE inserts
PASS_TO_ONE when a non-serial child feeds a serial parent. BE keeps all
upstream receiver tasks alive while retaining one downstream serial task,
producing the required N-to-one topology. PASS_TO_ONE now always creates
PassToOneExchanger; broadcast joins explicitly request BROADCAST when shared
hash tables are disabled.
The five reported RQG queries now return their expected results. Their
runtime profile shows twelve PASS_TO_ONE sink instances feeding four source
instances, which is three upstream tasks and one downstream task per BE.
### Release note
Fix incorrect results from local shuffle queries when a parallel join or
exchange feeds a serial operator.
### Check List (For Author)
- Test:
- Regression test: five reported RQG queries passed on a four-BE cluster
with parallel_pipeline_task_num=3 and enable_sql_cache=false
- Regression test: nereids_p0/local_shuffle passed, 7 suites, 0 failed
- Regression test: pipeline_p0/local_shuffle passed, 1 suite, 0 failed
- Unit test: LocalExchangePlannerTest and
DistributePlannerReceiverDestinationTest passed, 40 tests
- Manual test: runtime profile verified 12 upstream PASS_TO_ONE sink
instances to 4 downstream source instances
- Build: ./build.sh --be -j6 passed
- Style: FE Checkstyle and BE clang-format/check-format passed
- Static analysis: clang-tidy could not analyze the macOS PCH because
the generated AST and clang-tidy builtin-system-module settings were
incompatible; no clang-tidy result is claimed
- Behavior changed: Yes. Parallel-to-serial pipeline transitions now use an
explicit PASS_TO_ONE local exchange, and broadcast joins explicitly choose
BROADCAST when shared hash tables are disabled.
- Does this need documentation: No
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
