rahulsmahadev opened a new pull request, #4410:
URL: https://github.com/apache/iceberg-python/pull/4410

   # Rationale for this change
   
   `update_column` checks a type change with `promote()`, the read-side rule 
that the Avro and Parquet readers use. That rule is more lenient than the table 
spec: it also accepts `string -> binary`, `binary -> string` and `fixed(16) -> 
uuid`. So a populated `string` column can be changed to `binary` without any 
warning, and the commit succeeds:
   
   ```python
   tbl.update_schema().update_column("name", field_type=BinaryType()).commit()  
# name: string
   ```
   
   The spec's "Valid primitive type promotions" table does not list these 
pairs. Java's `SchemaUpdate.updateColumn` rejects them through 
`TypeUtil.isPromotionAllowed`, which allows equal types, `int -> long`, `float 
-> double` and decimal widening with the same scale.
   
   This PR validates primitive type changes using Java's v1/v2 promotion rules, 
while keeping PyIceberg's existing `unknown -> primitive` behaviour. It does 
not add v3 `date -> timestamp` or `date -> timestamp_ns` support, and it does 
not broaden unknown-type conversions. Validation runs when `field_type` is 
supplied and `allow_incompatible_changes` is false. `promote()` is unchanged, 
because the Avro resolver, the Arrow projection, the `union_by_name` no-op 
check and append compatibility (`_check_schema_compatible`) still rely on its 
leniency.
   
   Behaviour change: without `allow_incompatible_changes=True`, `update_column` 
warns before applying `string -> binary`, `binary -> string` or `fixed[16] -> 
uuid`. The warning is a `DeprecationWarning` that says the change is deprecated 
in 0.13.0 and will be removed in 0.14.0. A subsequent release will require the 
explicit opt-in. Every other invalid change still raises `ValidationError`, as 
on `main`. `allow_incompatible_changes=True` applies these three changes 
without a warning.
   
   `union_by_name` warns when an incoming UUID field replaces an existing 
`fixed[16]` field, including nested struct fields, list elements and map 
values. String/binary unions remain unchanged, because `union_by_name` checks 
them with `promote()` before it calls `update_column`. An incoming `fixed[16]` 
field over an existing UUID field also remains a no-op.
   
   Precedent: Fokko added the string <-> binary allowed-update tests in 
apache/iceberg#8374 (September 2023), and apache/iceberg-python#207 only moved 
that test file. #986 added `test_table_scan_default_to_large_types`, which 
changed a string column to binary with `update_column`. #1669 later changed 
that test to create the column as binary, so the update became a no-op.
   
   Alternative for maintainers: reject these three changes immediately instead 
of warning first. apache/iceberg-python#3613 rejected decimal promotions that 
change the scale without a deprecation cycle, and Fokko approved it. This PR 
chooses a warning cycle, because these changes were tested as supported 
(apache/iceberg#8374), and AGENTS.md asks for a deprecation cycle before 
breaking public API. Choosing immediate rejection means replacing the warning 
branch in `update_column` with a `ValidationError`.
   
   Other engines: in a local DuckDB 2.0.0 `iceberg_scan` reproduction, 
promoting a populated string column to binary made a non-ASCII value fail to 
read, and changed the four-character string `\x41` to the single byte `A`. 
PyIceberg continued to read the original UTF-8 bytes.
   
   ## Are these changes tested?
   
   Yes. `tests/test_schema.py` covers the following.
   
   - `test_update_column_rejects_invalid_promotions`: the exact error message 
for each invalid change (for example `int -> float`, a decimal scale change and 
`fixed[15] -> uuid`), unchanged table metadata after the rejections, and the 
valid promotions committed and read back after a reload.
   - `test_update_column_warns_for_non_spec_promotions`: the three deprecated 
changes warn with the exact message, commit, and read back after a reload.
   - `test_update_column_no_warning_with_allow_incompatible_changes`: the 
explicit opt-in applies the same three changes without a warning.
   - `test_union_by_name_fixed_uuid_and_string_binary`: `fixed(16)` and UUID in 
both directions, string/binary unchanged, and a list element.
   - `test_update_column_promotes_unknown_to_primitive_only`: the existing 
`unknown` behaviour, on a v3 table.
   
   On `main`, the three warning tests fail with `DID NOT WARN` and the union 
case fails because no warning is emitted. With this change they all pass. In 
`tests/integration/test_rest_schema.py`, `test_allowed_updates` and 
`test_disallowed_updates` expect the deprecation warning for string <-> binary, 
and `test_alter_map_key` still opts in with `allow_incompatible_changes=True` 
to reach the map key check.
   
   ## Are there any user-facing changes?
   
   Yes. Without `allow_incompatible_changes=True`, `update_column` emits a 
`DeprecationWarning` before applying `string -> binary`, `binary -> string` or 
`fixed[16] -> uuid`. Without this opt-in, `union_by_name` warns when incoming 
UUID fields replace existing fixed[16] fields. String/binary unions and 
incoming fixed[16] fields over existing UUID fields remain type no-ops. The 
deprecated implicit changes are scheduled to require the explicit opt-in in 
0.14.0. Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to