eldenmoon opened a new pull request, #66705:
URL: https://github.com/apache/doris/pull/66705
### What problem does this PR solve?
Issue Number: None
Related PR: #66204
Variant V2 mode-0 ingestion repeatedly decoded object metadata, rebuilt
canonical paths, inferred stable scalar types, and retained an auxiliary
`{plan,value}` entry for every sparse/doc cell during the path-to-row
transpose. These costs made V2 ingestion materially slower than V1 for
`Sparse16` and `Doc16` workloads and caused excessive finalize memory use at
large row counts.
This clean PR supersedes #66674. It deliberately excludes the later deferred
`ColumnVariant`/LEGACY representation and whole-column-read optimization: V1
`ColumnVariant` is not used as a new V2 runtime state.
### What is changed?
- Add the native shredded Variant V2 representation and preserve residual
encoded values, presence, and typed children across supported operators and
SerDe paths.
- Reuse Doris's existing Variant path/type machinery for writer, reader,
cast, element extraction, and fallback behavior.
- Cache object container metadata through `VariantRef::ObjectView` and use a
dense root field-id lookup with the existing nested-path fallback.
- Skip repeated inference/LCS work only when a scalar's full physical type
exactly matches the established path type; widening, arrays, decimal
range/scale changes, and conflicts retain the original slow path and JSONB
fallback.
- Cache the per-path binary SerDe and invalidate it on promotion/conversion.
- Reserve sparse/doc output buffers and transpose in bounded cell chunks,
recovering value indexes from per-path cursors instead of keeping a 16-byte
entry per cell.
- Add focused correctness tests for object views, path reuse, stable scalar
boundaries, SerDe invalidation, native shredded ownership, conflicts, and
chunk-boundary equivalence.
No benchmark-specific production branch, deferred LEGACY owner, or
whole-read fast path is included.
### Validation
- Clang-format 16 and `git diff --check`: passed before commit.
- Targeted ASAN BE tests through `run-be-ut.sh` with
`GLIBC_COMPATIBILITY=OFF` and default build concurrency: 34/34 passed across
`VariantPathBuilderTest`, `VariantShredderTest`, and `VariantValueTest`.
- Release benchmark target built and linked with default build concurrency
(no explicit `-j`).
- Default 1M ingestion, five samples per representation:
| Layout | V1 median (ms) | V2 median (ms) | V2/V1 | V2 improvement vs
pre-change baseline |
|---|---:|---:|---:|---:|
| Sparse16 | 9,212.349 | 8,202.836 | 0.8904 | 23.47% |
| Doc16 | 7,917.702 | 8,286.466 | 1.0466 | 23.68% |
| Full | 10,231.216 | 9,195.444 | 0.8988 | 13.60% |
- 10M ingestion, one sample per representation with ten 1M-row segments:
`Sparse16=0.8728`, `Doc16=1.0509`, `Full=0.9065` V2/V1; all row/layout/route
checks passed. The run used the immediately preceding benchmark-only counter
revision; production ingestion code is identical.
- Final Release exact-path checks also passed, but whole-column read
performance is intentionally not a target or gate for this PR.
- Independent final review of the benchmark framework and retained
implementation: P0=0, P1=0.
### Scope notes
- The current production compactor still materializes its persisted Variant
data plane as `ColumnVariant`; this PR does not introduce or claim a persisted
V2 compaction mode.
- The benchmark and its compaction fixture are maintained separately in
#66668.
- The old draft #66674 is superseded because its scope includes design
directions intentionally removed from this version.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (Release benchmarks above)
- Behavior changed:
- [x] Yes. Variant V2 can retain and write native shredded state; encoded
fallback remains available for unsupported/conflicting cases.
- Does this need documentation?
- [x] No.
### Release note
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
