adityaksolves opened a new pull request, #18517: URL: https://github.com/apache/iceberg/pull/18517
Closes #18499 ### Problem `TableMetadataCache.schema()` iterates all historical table schemas in ascending schema id order and returns the first one that `CompareSchemasVisitor` marks as `SAME`. The visitor matches by field name, so after `DROP COLUMN c` followed by `ADD COLUMN c` (same name and type), a historical schema still holding the retired field id for `c` is matched before the current schema. The writer then writes Parquet files with the retired field id, and readers using the current schema return `null` for `c`. ### Fix Try the current schema first when resolving the input schema, so its field ids win over historical schemas that only match by name. The current schema id is now carried in the cache entry (`CacheItem.currentSchemaId`). The remaining historical schemas keep their ascending order. This is Option 1 from the issue. Applied identically to Flink 1.20, 2.2 and 2.3. ### Testing - Added `TestTableMetadataCache#testSchemaPrefersCurrentSchemaAfterDropAndReAddColumn` (add column, drop it, re-add it with the same name, then assert the resolved schema uses the current field id) in all three Flink versions. - Without the fix, the new test fails with `expected: 4 but was: 3`; with the fix it passes. - Full `org.apache.iceberg.flink.sink.dynamic.*` test package passes on Flink 1.20, 2.2 and 2.3. - `spotlessCheck` and `checkstyleMain`/`checkstyleTest` pass on all three versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
