hubgeter opened a new pull request, #68962:
URL: https://github.com/apache/doris/pull/68962
### What problem does this PR solve?
Issue Number: N/A (tracked as DORIS-29685)
Related PR: N/A
Problem Summary:
A data file that carries no Iceberg field ids (an `add_files` import or a
migrated Parquet file written by a non-Iceberg writer) read as an all-NULL row
in Doris, while Iceberg's own reader returned the real values. Iceberg binds
such a file by position when the table has no usable
`schema.name-mapping.default` (`ParquetSchemaUtil.addFallbackIds` +
`pruneColumnsFallback`, gated by
`iceberg.netflix.unsafe-parquet-id-fallback.enabled`, whose default stays true
through 1.12 and is scheduled for removal in 2.0 - see apache/iceberg#17444).
Doris's FileScannerV2 kept `BY_FIELD_ID` in that case, so no field matched and
the row silently materialized as NULL. The legacy scanner
(`enable_file_scanner_v2=false`) read the same file correctly, which showed the
gap was in the V2 read path.
Solution: mirror Iceberg's decision for Parquet. When the split's file has
no field id at all, the plan carries version-2 scan semantics and the table
exposes no authoritative name mapping,
`IcebergTableReader::annotate_file_schema` stamps ordinal ids (1..N) on the
file's top-level fields, so the shared `BY_FIELD_ID` path resolves them instead
of returning NULL. Ids are stamped on top-level fields only, exactly like
`addFallbackIds`: nested children keep no id and stay NULL unless a name
mapping (or a rewrite that persists field ids) resolves them. ORC is
intentionally untouched (Iceberg's positional fallback lives in the Parquet
reader).
Behavior matrix:
- file has any real field id -> unchanged (Iceberg's `hasIds` is existential)
- no id + table has an authoritative name mapping -> unchanged (BY_NAME
aliases)
- no id + no name mapping -> top-level fields are now read by position (was
all-NULL); nested children stay NULL
- no id + old FE plan (scan semantics version 1) -> unchanged (legacy name
fallback)
Test:
- 6 new unit tests in `be/test/format_v2/table/iceberg_reader_test.cpp`:
positional binding when physical names differ, `file ordinal == field id`
semantics (dropped field), mapping precedence (an uncovered field stays NULL),
legacy plan unchanged, nested struct child stays NULL, partially-ided file
untouched.
- `test_iceberg_migrated_nested_without_name_mapping` regression expectation
updated from `\N \N \N` to the positionally-read `id` plus a predicate
assertion; the registered fixture Parquet has `field_id=-1` on every field.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
