hubgeter opened a new pull request, #68121:
URL: https://github.com/apache/doris/pull/68121
### What problem does this PR solve?
Issue Number: None
Related PR: #66538
Problem Summary:
Since the Iceberg V3 default-value backport (#66538), reading a subpath of
an Iceberg `VARIANT` column fails during planning whenever
`enable_prune_nested_column` is on, which is the default:
```sql
SELECT CAST(message['mainDomain'] AS STRING) FROM iceberg_tbl;
-- ERROR 1105 (HY000): errCode = 2, detailMessage = Iceberg access path
continues below primitive column message
```
Plain queries fail as well as `EXPLAIN`. So do Variants nested in structs,
map values, and array elements (`info.payload['kind']`,
`attrs['object']['kind']`, `events[1]['kind']`). The existing
`test_iceberg_variant_read` suite hits this at its first nested Variant subpath
query.
Root cause: `IcebergScanNode` walks every projected access path to decide
whether the scan needs current backend semantics (recursive initial defaults,
required-field rejection, name-mapping aliases). The walk assumes each
component below the root names an Iceberg struct, list, or map child. Nested
column pruning keeps Variant object keys in the path, for example `[<field id>,
mainDomain]`. A Variant column has no Iceberg schema children, so the walk
trips its `getChildren() != null` precondition. A query that also projects a
nested field with an initial default can hide the failure, because the walk
returns early.
Fix: check the Variant field itself as before, then stop. The remaining
components are keys or indexes inside the encoded value, not Iceberg field IDs
or access tokens. Struct, map, and array traversal and their preconditions are
unchanged.
Master is not affected. Its Iceberg scans go through `PluginDrivenScanNode`,
whose access-path handling already stops at Variant columns (#66413).
Workaround before this fix: `SET enable_prune_nested_column = false`.
### Release note
Fix the planning error "Iceberg access path continues below primitive
column" when querying subpaths of Iceberg `VARIANT` columns.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Unit test:
`IcebergScanNodeTest#testVariantAccessPathTerminatesIcebergFieldTraversal`
covers Variant keys at the top level, in a struct, and in a map value,
including keys spelled like a field ID or `*`. It checks that a sibling initial
default is still detected and that a required Variant added later still
requires missing-field rejection. It failed with the reported error before the
fix; all 94 `IcebergScanNodeTest` cases pass after it.
Regression test: `test_iceberg_variant_access_path_defaults` creates a
format-version 3 table with top-level, struct, map, and array Variants. Spark
writes unshredded rows, then the Iceberg API adds `info.added` with
initial-default 7 and appends a file with shredded Variant columns. With nested
column pruning on, the suite checks:
- `EXPLAIN` access paths (`message(2).mainDomain`,
`info(3).payload(7).kind`);
- leaf projection, `GROUP BY`, filter, and a key that matches another
field's ID;
- struct, map, and array Variant subpaths;
- Variant subpaths together with the nested initial default (7 for old
rows, explicit NULL preserved, 100 preserved);
- identical results with pruning on and off;
- Spark results for the top-level and struct fields.
Locally, with an isolated Iceberg REST/MinIO/Spark 4.0 environment, an FE
from this branch, and a branch-4.1 BE (`858fc0ea504`; the change is FE-only):
- The new suite fails at its first `EXPLAIN` with the reported error
before the fix and passes after it.
- `test_iceberg_variant_read` fails at line 925 (`info.payload['kind']`)
before the fix. After it, the suite gets past all Variant subpath queries, then
stops at a TopN profile-counter assertion at line 1233. That local BE predates
#67906, which changed TopN row-id fetching in FileScannerV2, so CI with a
current BE should confirm that part.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
