rahulsmahadev opened a new pull request, #4359:
URL: https://github.com/apache/iceberg-python/pull/4359

   # Rationale for this change
   
   `add_files` looks up each Parquet column path in a mapping built from the 
table's current column names, so a file that is valid for the table fails when 
its paths differ from those names:
   
   ```python
   catalog.create_table("db.t", schema=Schema(NestedField(1, "a", LongType()), 
NestedField(2, "b", StringType())))
   with catalog.load_table("db.t").update_schema() as update:
       update.rename_column("a", "c")
   # The file has field IDs 1 and 2, and column 1 is still named "a" in it.
   catalog.load_table("db.t").add_files(["a.parquet"])  # KeyError: 'a'
   ```
   
   The same happens for a nested field renamed after the file was written 
(`KeyError: 'address.zip'`) and for a list whose element is named `item`, the 
layout pyarrow writes with `use_compliant_nested_type=False` (`KeyError: 
'tags.list.item'`). #716 reported the list case; the stale bot closed it 
without a fix.
   
   The name-based mapping can also record statistics under the wrong column. If 
the table drops `a` (field ID 1) and adds a new `a` (field ID 3), an old file's 
`a` is recorded under field ID 3, so a scan for `a IS NULL` prunes the file and 
misses its rows.
   
   This change builds the mapping from the file's own column paths and field 
IDs (with name mapping for files without IDs), so statistics follow field IDs 
as they do for files PyIceberg writes.
   
   ## Are these changes tested?
   
   Yes, in `tests/io/test_pyarrow.py`. 
`test_add_files_matches_columns_by_field_id` covers the nested rename and the 
`item` list element with and without field IDs. 
`test_add_files_rejects_columns_missing_from_table` covers a file column whose 
field ID the table lacks and a column whose name matches but whose field ID 
differs. Both fail without this change.
   
   ## Are there any user-facing changes?
   
   Yes:
   - `add_files` accepts files whose columns were renamed in the table and 
files whose list element is not named `element`.
   - A file column whose field ID is not in the table now gets the existing 
"PyArrow table contains more columns" `ValueError` instead of a `KeyError`.
   - A file column whose name matches a table column but whose field ID differs 
is rejected with the same error. Before, such a file was added with its 
statistics under the wrong column.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to