Ankitkj1999 opened a new issue, #4355:
URL: https://github.com/apache/iceberg-python/issues/4355
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
### Describe the bug
If a table stops partitioning by a column
(`update_spec().remove_field(...)`) and later drops that source column
(`update_schema().delete_column(...)`):
1. Calling `table.inspect.partitions()`, `entries()`, `files()`,
`data_files()`, `delete_files()`, `all_files()`, `all_data_files()`, or
`all_delete_files()` raises `ValueError: Could not find field with id: ...`
inside `TableMetadata.specs_struct()`.
2. Calling `table.inspect.manifests()` / `all_manifests()` succeeds, but
renders `lower_bound` and `upper_bound` in `partition_summaries` as `'null'`
because `_partition_summaries_to_rows()` evaluates
`spec.partition_type(self.tbl.schema())`, which falls back to `UnknownType()`.
*(Note: open PR #4115 fixes `delete()` manifest rewrites in
`pyiceberg/avro/file.py` and `pyiceberg/manifest.py`, whereas this failure
occurs in `TableMetadata.specs_struct()` and `InspectTable` even without
calling `delete()`.)*
### To Reproduce
```python
import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType
with tempfile.TemporaryDirectory() as tmp:
cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
cat.create_namespace("ns")
tbl = cat.create_table(
"ns.t_v2",
schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p",
StringType())),
partition_spec=PartitionSpec(PartitionField(2, 1000,
IdentityTransform(), "p")),
properties={"format-version": "2"},
)
tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
with tbl.update_spec() as spec_update:
spec_update.remove_field("p")
with tbl.update_schema() as schema_update:
schema_update.delete_column("p")
print(tbl.inspect.manifests().to_pydict()["partition_summaries"])
# [[{'contains_null': False, 'contains_nan': False, 'lower_bound':
'null', 'upper_bound': 'null'}]]
tbl.inspect.partitions()
# ValueError: Could not find field with id: 2
```
### Expected behavior
`TableMetadata.specs_struct()` and `table.inspect.*` should resolve
partition source field types from historical table schemas when the source
column was dropped from the current schema, keeping metadata inspection tables
and manifest partition summary bounds readable.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]