hedger9487 opened a new pull request, #3865: URL: https://github.com/apache/iceberg-python/pull/3865
Closes #3105 # Rationale for this change When an Iceberg table undergoes schema evolution (e.g. `update_schema().add_column(...)` or `update_schema().union_by_name(...)`), calling `table.upsert()` previously failed with: ``` ValueError: Target schema's field names are not matching the table's field names: ['id', 'name', 'age', 'city', 'ping'], ['id', 'name', 'age', 'city'] ``` ### Root Cause In `pyiceberg/table/upsert_util.py` (`get_rows_to_update`), Step 1 previously cast the entire `source_table` (which has the new schema including evolved columns) to `target_table.schema` (which only contains columns present in older data files). PyArrow requires column names to match exactly when casting an entire table, raising `ValueError`. ### Fix 1. In Step 1, cast only the `join_cols` to the target table's join schema (`join_schema = pa.schema([target_table.schema.field(col) for col in join_cols])`). 2. In Step 4, safely retrieve non-key values from `target_row` as `None` if the column was added during schema evolution and absent in `target_table`. 3. Add regression tests covering upsert after `add_column`, `union_by_name`, multiple schema evolutions with composite keys, and no-op null comparisons, while explicitly verifying that underlying Parquet data files are properly replaced (Copy-on-Write) and snapshot operations include `OVERWRITE` and `APPEND`. ## Are these changes tested? Yes: - Added `test_upsert_after_schema_evolution` (verifies file replacement and snapshot operations) - Added `test_upsert_after_schema_evolution_union_by_name` - Added `test_upsert_after_multiple_schema_evolutions_with_composite_keys` - Added `test_upsert_after_schema_evolution_noop_and_nulls` - Verified mutation test fails without the fix and passes with the fix. - All 12 pre-commit linters and 471 table tests pass cleanly. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
