lilei1128 opened a new pull request, #16523:
URL: https://github.com/apache/iceberg/pull/16523
When performing time-travel reads with a filter on a column that was
subsequently renamed, a ValidationException was thrown:
"Cannot find field 'col' in struct: struct<..., 2: value: ...>"
Root cause: BaseDistributedDataScan.specCache() called table().specs()
directly, which returns partition specs bound to the current table schema.
When the filter expression was projected via Projections.inclusive() in
newManifestEvaluator(), it tried to resolve column names against the
current schema instead of the snapshot schema, causing the failure.
Fix:
- Override useSnapshotSchema() to return true in BaseDistributedDataScan,
consistent with DataTableScan
- Change specCache() to use specs() instead of table().specs(), so
partition specs are re-bound to the snapshot schema during time-travel
- Also fix SparkTable.newScanBuilder() to resolve the scan schema against
the requested snapshot when snapshot-id is passed via options, as a
defensive fix for cases where SparkTable is not constructed with a
snapshotId field
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
