Ankitkj1999 opened a new issue, #4355:
URL: https://github.com/apache/iceberg-python/issues/4355

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   ### Describe the bug
   If a table stops partitioning by a column 
(`update_spec().remove_field(...)`) and later drops that source column 
(`update_schema().delete_column(...)`):
   
   1. Calling `table.inspect.partitions()`, `entries()`, `files()`, 
`data_files()`, `delete_files()`, `all_files()`, `all_data_files()`, or 
`all_delete_files()` raises `ValueError: Could not find field with id: ...` 
inside `TableMetadata.specs_struct()`.
   2. Calling `table.inspect.manifests()` / `all_manifests()` succeeds, but 
renders `lower_bound` and `upper_bound` in `partition_summaries` as `'null'` 
because `_partition_summaries_to_rows()` evaluates 
`spec.partition_type(self.tbl.schema())`, which falls back to `UnknownType()`.
   
   *(Note: open PR #4115 fixes `delete()` manifest rewrites in 
`pyiceberg/avro/file.py` and `pyiceberg/manifest.py`, whereas this failure 
occurs in `TableMetadata.specs_struct()` and `InspectTable` even without 
calling `delete()`.)*
   
   ### To Reproduce
   ```python
   import tempfile
   import pyarrow as pa
   from pyiceberg.catalog.memory import InMemoryCatalog
   from pyiceberg.partitioning import PartitionField, PartitionSpec
   from pyiceberg.schema import Schema
   from pyiceberg.transforms import IdentityTransform
   from pyiceberg.types import LongType, NestedField, StringType
   
   with tempfile.TemporaryDirectory() as tmp:
       cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
       cat.create_namespace("ns")
       tbl = cat.create_table(
           "ns.t_v2",
           schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", 
StringType())),
           partition_spec=PartitionSpec(PartitionField(2, 1000, 
IdentityTransform(), "p")),
           properties={"format-version": "2"},
       )
       tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
       with tbl.update_spec() as spec_update:
           spec_update.remove_field("p")
       with tbl.update_schema() as schema_update:
           schema_update.delete_column("p")
   
       print(tbl.inspect.manifests().to_pydict()["partition_summaries"])
       # [[{'contains_null': False, 'contains_nan': False, 'lower_bound': 
'null', 'upper_bound': 'null'}]]
   
       tbl.inspect.partitions()
       # ValueError: Could not find field with id: 2
   ```
   
   ### Expected behavior
   `TableMetadata.specs_struct()` and `table.inspect.*` should resolve 
partition source field types from historical table schemas when the source 
column was dropped from the current schema, keeping metadata inspection tables 
and manifest partition summary bounds readable.
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to