kdn36 opened a new issue, #4318:
URL: https://github.com/apache/iceberg-python/issues/4318
### Apache Iceberg version
0.12.0 (latest release); the same logic is on main
### Please describe the bug 🐞
Adding a column changes what an existing column reads. Files added with
`add_files` have no field IDs and are read through the name mapping. After a
rename, the renamed column reads correctly. Adding a column that reuses the old
name then moves that name to the new field in the mapping
(`_remove_reassigned_names` in `pyiceberg/table/name_mapping.py`). From then
on, the renamed column reads as null for those files, and the new column gets
its values.
The manifest statistics of those files still describe the original field, so
planning also prunes with statistics that no longer match what is read: below,
`IsNull("b")` skips the file although both rows read `b` as null.
```python
import tempfile
import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import IsNull
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
d = tempfile.mkdtemp()
catalog = SqlCatalog("c", uri=f"sqlite:///{d}/c.db", warehouse=f"file://{d}")
catalog.create_namespace("ns")
table = catalog.create_table("ns.t", Schema(NestedField(1, "a", LongType())))
# a Parquet file without field IDs, read through the name mapping
pq.write_table(pa.table({"a": [1, 3]}), f"{d}/imported.parquet")
table.add_files([f"file://{d}/imported.parquet"])
with table.update_schema() as update:
update.rename_column("a", "b")
print(table.scan().to_arrow().to_pydict())
# {'b': [1, 3]}
with table.update_schema() as update:
update.add_column("a", LongType())
print(table.scan().to_arrow().to_pydict())
# {'b': [None, None], 'a': [1, 3]}: expected {'b': [1, 3], 'a': [None, None]}
print(table.scan(row_filter=IsNull("b")).to_arrow().num_rows)
# 0: the file is pruned by the statistics of `b`, which hold its original
values
```
Expected: adding a column leaves the values of existing columns unchanged,
as it does for files with field IDs. The name was written for the original
field in those files, so either the mapping keeps it there, or adding a column
whose name files without field IDs still use is refused.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
