mrhhsg opened a new pull request, #70419:
URL: https://github.com/apache/doris/pull/70419
### Background
A fragment prepares a local task for each pipeline and instance, except for
serial pipelines. Preparation must finish before the fragment-dispatch RPC
budget expires.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Bugfix
Problem Summary:
The parallel-prepare decision counts only local instances, so multiple
pipelines can multiply initialization work while preparation remains serial.
Use the actual task work to choose the existing parallel preparation path.
Before widening that path, make thread-pool rejection atomic so failed worker
creation cannot leave callbacks referring to a completed prepare call. Most
of
the diff adds boundary and failure-path tests; existing users keep the same
threshold value and RPC budget but get task-aware scheduling.
### Details
- **Change map:** 81 production-source changed lines (34 for
scheduling/variable description,
47 for thread-pool ownership and BE_TEST-only hooks), 318 test lines, 50
runner-generated
expected-output lines, and no standalone documentation changes.
- **Frequency:** Multi-pipeline queries with moderate instance counts; the
callback
lifetime failure additionally requires worker creation to fail with no
executor.
- **Root cause:** The threshold ignores pipeline fan-out; 32 instances with
two
32-task pipelines still take the serial path at the default threshold of
32.
- **Why existing tests missed it:** No boundary test covered multiple
pipelines
with an instance count equal to the threshold. Thread-pool tests did not
inject
first-worker creation failure followed by recovery or overlapping
submission.
- **Fix:** Count work using the task builder's serial-pipeline predicate and
reuse the existing per-instance thread-pool jobs and status/latch handling.
Make thread-pool bootstrap failure reject before queue ownership transfers,
do not report failed pool growth as rejection after acceptance, and wake
shutdown waiters when the final pending start fails. Retain the last live
worker during idle/positive-limit shrink while growth is pending; shutdown
still discards queued work. Fault injection uses BE_TEST-only SyncPoint
hooks,
not unsynchronized global configuration writes; callbacks filter by
captured
pool identity before accessing test-local state.
Keep explicitly unset presence flags, singleton instances, negative values
and strict
threshold boundaries unchanged; describe the task-based threshold in FE.
- **Other paths and branches:** Initial prepare and recursive fragment
rebuild
share the helper. Regression coverage includes FE- and BE-planned
exchanges.
The pool prerequisite also protects other callers that treat non-OK
submission
as rejection. No protocol, storage format or transaction changes.
### Testing
- `BUILD_TYPE=ASAN ./build.sh --be --fe -j 24` and subsequent ASAN BE
rebuild passed locally; FE Checkstyle reported zero violations.
- `./run-be-ut.sh --run
--filter='ThreadPoolCreationFailureTest.*:ThreadPoolTest.*:Tokens/ThreadPoolTestTokenTypes.*:PipelineTest.*'
-j 24`: 43/43 passed.
- `GTEST_REPEAT=100 GTEST_SHUFFLE=1 ./run-be-ut.sh --run
--filter='ThreadPoolCreationFailureTest.*' -j 24`: all four failure tests
passed in all 100 iterations (400 checks).
- `./run-regression-test.sh --conf <isolated-conf> --run -d pipeline_p0 -s
test_parallel_prepare_task_threshold`: five variants passed. Output was
initially
generated through the runner and then verified by ordinary reruns; the
final
production logic passed again after the thread-pool prerequisite.
- clang-format v16, format check, build hygiene, git diff --check.
- Changed-line clang-tidy passed for all six C++ source/header/test files,
using
the toolchain resource directory and UT compilation database.
- Isolated four-BE ASAN pressure check with unrelated CTE inlining disabled
(`cte_inline_mode=-1`): original binary timed out twice during fragment
dispatch;
the fixed scheduling/submission logic completed three fresh-start trials
with identical
1,044-row result multisets and the default RPC budget. That binary predates
the final BE_TEST-only hook conversion and comment cleanup; normal
scheduling
and submission logic is unchanged, but this is not a final-version-stamp
test.
Not a production, cloud or Release benchmark.
- A supplemental SQL rerun of the earlier test/comment follow-up was blocked
before
SQL by an occupied FE Arrow port. It does not replace the earlier passing
SQL
checks; the final changed paths were rebuilt and passed the focused
UT/static checks.
### Notes for Reviewers
The existing threshold now measures task work rather than instance count.
Parallel preparation can use more transient CPU/memory concurrency, bounded
by the existing thread-count cap. First-worker bootstrap creates under the
pool
lock before accepting work; established pools still grow without holding that
lock. Non-OK submissions cannot leave runnable callbacks behind. No new
timeout
or cancellation mechanism is added. Existing dynamic max=0 pause/resume
progress
is not redesigned; submission ownership is separate from that management
boundary.
### Release note
Reduce fragment-dispatch timeouts for complex queries by accounting for
pipeline tasks when choosing preparation parallelism. The existing
parallel_prepare_threshold applies to task work rather than only instances.
Prevent rejected thread-pool submissions from running later after
worker-creation
failures.
### Check List (For Author)
- Behavior changed: Yes (preparation scheduling and worker-creation failure
handling).
- Does this need documentation: No (existing session-variable description
updated).
- Pre-submission status: Partially verified: local ASAN focused tests and SQL
checks passed; no cloud, mixed-version, full test run or remote CI
validation.
### AI Disclosure
- Tool: Codex (authoring model not exposed); local review: GPT-6 Astra
(xhigh).
- Generated: Code, tests and this description.
- Human review: Not yet performed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
