eldenmoon opened a new pull request, #66709:
URL: https://github.com/apache/doris/pull/66709
### What problem does this PR solve?
Issue Number: None
Related PRs: #66204, #66705, #66668
Variant V2 mode-0 ingestion repeatedly decoded object metadata, rebuilt
canonical paths,
inferred stable scalar types, and retained an auxiliary `{plan,value}` entry
for every
sparse/doc cell during the path-to-row transpose. These costs made V2
ingestion materially
slower than V1 for `Sparse16` and `Doc16` workloads and caused excessive
finalize memory use
at large row counts.
This PR contains only the ingest/finalize performance optimization: one
commit, eight files.
It does not contain the native shredded execution feature from #66705.
### What is changed?
- Cache object container metadata through `VariantRef::ObjectView` and use a
dense root
field-id lookup with the existing nested-path fallback.
- Skip repeated inference/LCS work only when a scalar's full physical type
exactly matches
the established path type; widening, arrays, decimal range/scale changes,
and conflicts
retain the original slow path and JSONB fallback.
- Cache the per-path binary SerDe and invalidate it on promotion/conversion.
- Reserve sparse/doc output buffers and transpose in bounded cell chunks,
recovering value
indexes from per-path cursors instead of keeping a 16-byte entry per cell.
- Add focused correctness tests for object views, path reuse, stable scalar
boundaries,
SerDe invalidation, native shredded ownership/conflicts, and
chunk-boundary equivalence.
No benchmark-specific production branch, deferred LEGACY owner, or
whole-read fast path is
included.
### Validation
- Clang-format 16 and `git diff --check`: passed before commit and rechecked
after the split.
- Targeted ASAN BE tests through `run-be-ut.sh` with
`GLIBC_COMPATIBILITY=OFF` and default
build concurrency: 34/34 passed across `VariantPathBuilderTest`,
`VariantShredderTest`,
and `VariantValueTest`.
- Release benchmark target built and linked with default build concurrency
(no explicit `-j`).
- Default 1M ingestion, five samples per representation:
| Layout | V1 median (ms) | V2 median (ms) | V2/V1 | V2 improvement vs
pre-change baseline |
|---|---:|---:|---:|---:|
| Sparse16 | 9,212.349 | 8,202.836 | 0.8904 | 23.47% |
| Doc16 | 7,917.702 | 8,286.466 | 1.0466 | 23.68% |
| Full | 10,231.216 | 9,195.444 | 0.8988 | 13.60% |
- 10M ingestion, one sample per representation with ten 1M-row segments:
`Sparse16=0.8728`, `Doc16=1.0509`, and `Full=0.9065` V2/V1; all row,
layout, and route
checks passed.
- Independent final review of the benchmark framework and retained
implementation:
P0=0, P1=0.
#### Test host and method
- Alibaba Cloud VM, Ubuntu 22.04.4 / Linux 5.15.
- 8 vCPU (Intel Xeon 6982P-C), 30 GiB RAM, no swap, 300 GiB NVMe EBS/ext4.
- Release process pinned to CPU 4; startup load `0.09/0.02/0.01`.
- Source prevalidated and read through page cache; correctness/layout
validation paused
outside measured stages.
The companion benchmark and raw scan-and-rewrite results are maintained in
#66668. Those
results show that whole-column query/assembly and append remain follow-up
work; this PR does
not claim to optimize whole-column read/rewrite.
### Dependency and merge order
This optimization applies independently to current `master`. #66705 is a
separate feature PR.
The two PRs intentionally overlap in several Variant builder/shredder files,
so whichever PR
lands second must be rebased and rerun through CI; the scopes remain
separate for review.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (Release benchmarks above)
- Behavior changed:
- [ ] No user-visible behavior change; this PR optimizes existing mode-0
shredding.
- Does this need documentation?
- [x] No.
### Release note
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
