hubgeter opened a new pull request, #68433:
URL: https://github.com/apache/doris/pull/68433
### What problem does this PR solve?
Issue Number: None
Related PR: #67479, #67687
Problem Summary:
An equality delete can reference a field that a later schema change dropped:
```java
// equality delete on k (field id 1), then:
table.updateSchema().deleteColumn("k").commit();
```
Iceberg 1.11.0 resolves such field IDs against every historical schema, and
#67479 synced Doris's `DeleteFileIndex` with it, so the default planner handles
this. The manifest-cache planner (`meta.cache.iceberg.manifest.enable=true`)
still builds the index from `specsById` alone. Those specs are bound to the
current schema, so planning fails with
```
IllegalArgumentException: Cannot find field for ID 1
```
and falls back to the SDK scan. Query results stay correct, but every query
on such a table logs a WARN with a stack trace, and `EXPLAIN VERBOSE` reports
`manifest cache: failures=1`, so the manifest cache is bypassed for that table.
This PR passes the table's schemas to the index, as Iceberg's
`DataTableScan` does and as master's connector already does (#67687).
`DeleteFileIndex.Builder.schemasById` becomes public so `IcebergScanNode` can
call it.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
`IcebergScanNodeTest#testManifestCachePlanningResolvesDroppedEqualityDeleteField`
fails with `Cannot find field for ID 1` without the fix. Manual test on a REST
catalog, with a v2 table where `k` was dropped after an equality delete on it:
with the manifest cache enabled, `SELECT`/`COUNT(*)`/`EXPLAIN ... FOR VERSION
AS OF` return the same rows as before (`B, M` / `2`), `EXPLAIN VERBOSE` now
shows `failures=0`, and the fallback WARN is gone.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
