HappenLee opened a new pull request, #67762:
URL: https://github.com/apache/doris/pull/67762
### What problem does this PR solve?
Problem Summary:
Internal-table late materialization through `PMultiGetRequestV2` reads
independent segment groups serially. A fetch spanning several segments
therefore waits for each group's storage reads in sequence.
Add opt-in parallel column-store reads. `rowid_fetch_parallel_batch_rows` is
forwarded from FE through `TQueryOptions` into the V2 request and specifies the
number of segment groups per bthread task.
`rowid_fetch_parallel_max_concurrency` limits concurrent tasks per request
(default 8).
Each task owns its readers, schema, slots, and statistics. Tasks write
disjoint scan blocks and scatter-map entries; results retain the existing
row-ID sorting, deduplication, original request order, and file-cache
accounting. Worker exceptions are converted to errors, started tasks are joined
before returning, and synchronous fallback preserves the caller's
memory-tracking context. Row-store reads remain serial; external reads retain
their existing scheduler.
### Release note
Add `rowid_fetch_parallel_batch_rows` to enable parallel internal-table
column-store rowid fetch. The default value, 0, preserves serial execution;
positive values specify segment groups per task. The mutable BE configuration
`rowid_fetch_parallel_max_concurrency` limits concurrency per request and
defaults to 8.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test
- [ ] No need to test or manual test.
Validation:
- On the Apache master-based branch: BE clang-format 16.0.6 checks,
build-hygiene checks, FE Checkstyle (0 violations), and `git diff --check`
passed.
- The FE session-variable test passed before rebasing onto current master:
1 test, 0 failures, 0 errors. It covers defaults and forwarding into Thrift
query options.
- Six BE unit tests cover task partitioning, serial fallbacks, empty
input, concurrency bounds, error completion, and exception propagation.
- Added `parallel_rowid_fetch_v2` regression coverage comparing
materialized results with ordinary reads, including duplicate projections,
repeated row IDs, nullable/nested values, row-store reads, and empty results.
- BE unit tests and SQL regression have not been run successfully. The
original local BE test build was blocked by unrelated uncommitted column
changes; no such changes are included in this PR.
- No performance speedup is claimed without a dedicated benchmark.
- Behavior changed:
- [ ] No.
- [x] Yes. Positive `rowid_fetch_parallel_batch_rows` values enable
parallel internal column-store fetch; default behavior remains serial.
- Does this need documentation?
- [ ] No.
- [x] Yes. Session-variable and BE-configuration descriptions are
included; a separate website documentation PR is not included.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
