eldenmoon opened a new pull request, #66966:
URL: https://github.com/apache/doris/pull/66966
### What problem does this PR solve?
Issue Number: None
Related PR: #66910
Problem Summary:
This PR is based on the exact head of #66910 (`fb41a26081f`) and extends it
with the Q5/Q6 fix. It contains two complementary optimizations for
Spark-written unshredded Iceberg Variant queries.
First, the original #66910 change avoids materializing the complete root
`ColumnVariantV2` for path extraction from the exact complete `metadata +
value` layout. It directly seeks requested scalar paths, reuses prefixes and
cached results, deduplicates metadata dictionaries, and preserves SQL NULL,
missing path, Variant JSON null, integer-width, corrupt-layout, and
shredded-overlay semantics.
Second, Q5 and Q6 remained much slower than StarRocks because the benchmark
creates a refreshed 480-row MTMV, but Doris rejected every external-table MTMV
by default. Doris consequently scanned all 180 million Iceberg rows. The
existing gate could not distinguish an external table with an exact
statement-pinned snapshot from data-unaware external tables.
The added FE change lets snapshot-id-aware external tables such as Iceberg
participate in MTMV rewrite freshness checks under the default setting. Query
planning and MTMV validation use the same pinned statement snapshot id.
Timestamp-based, unpinned, unsupported, and unresolved external tables remain
behind the existing explicit data-unawareness opt-in.
On the 180-million-row reproduction environment, with
`materialized_view_rewrite_enable_contain_external_table=false`, both Q5 and Q6
now choose `mv_truewatch_spark_unshredded`, scan 480 rows, and preserve stable
result hashes. Hot engine-side latencies were 46/49 ms for Q5 and 45/42 ms for
Q6. A complete matched Release Q1-Q10 workflow is being run for the final
comparison.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- Existing Iceberg MTMV rewrite case now validates rewrite with the
data-unawareness opt-in disabled.
- [x] Unit Test
- `./run-be-ut.sh --run
--filter='VariantValueTest.*:VariantColumnReaderTest.*'` (59 passed, ASAN;
inherited from #66910)
- `./run-fe-ut.sh --run
'org.apache.doris.mtmv.MTMVUtilTest,org.apache.doris.mtmv.MTMVRewriteUtilTest'`
(16 passed)
- [x] Manual test
- `build-support/clang-format.sh` and
`build-support/check-format.sh` for the C++ change in #66910
- `BUILD_TYPE=RELEASE ./build.sh --fe` (Checkstyle passed)
- Manual EXPLAIN, profile, result-hash, and Q5/Q6 timing against the
180-million-row Spark unshredded Iceberg data
- Full Release c-benchmark Spark unshredded Iceberg filecache Q1-Q10
workflow pending
- [ ] No need to test or manual test.
- Behavior changed:
- [ ] No.
- [x] Yes. Complete unshredded Parquet Variant extraction avoids
redundant materialization, and snapshot-id-aware external tables can use fresh
MTMVs without enabling the data-unawareness opt-in.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
