HaoXuAI opened a new issue, #3840:
URL: https://github.com/apache/iceberg-python/issues/3840
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
`equality_ids` is declared as a list of **long** in the manifest-entry Avro
schema, but the Iceberg spec, the Java reference implementation, and
iceberg-cpp all specify **int**. pyiceberg is the only implementation that
disagrees, so manifests it writes are rejected outright by spec-conformant
readers.
## Evidence
| source | element type of field 136 |
|---|---|
| Iceberg spec, `format/spec.md` ("Manifests" table) | `list<136: int>` |
| Java, `api/src/main/java/org/apache/iceberg/DataFile.java` |
`ListType.ofRequired(136, IntegerType.get())` |
| iceberg-cpp, `src/iceberg/manifest_entry.h` | `iceberg::int32()` |
| **pyiceberg, `pyiceberg/manifest.py:298` and `:393`** | **`LongType()`** |
Both the v1 and v2 manifest-entry schemas are affected. Note that
pyiceberg's own type hint already says `int`:
```python
# pyiceberg/manifest.py:528
def equality_ids(self) -> list[int] | None:
```
So only the Avro schema disagrees; the Python side already treats these as
ints.
## Impact
A spec-conformant reader cannot read a pyiceberg-written manifest at all.
iceberg-cpp fails with:
```
Cannot read Iceberg type: int from Avro type: "long"
```
because its `kInt` case accepts only `AVRO_INT` (Avro permits `int` → `long`
promotion, not the reverse). Any Java or C++ consumer of pyiceberg-written
metadata hits the same wall.
This is not limited to equality-delete tables: the schema is written into
every manifest, so an ordinary append-only table with no delete files is
equally unreadable.
## Reproducer
```python
import fastavro
# any table written by pyiceberg
with open("<some>-m0.avro", "rb") as f:
schema = fastavro.reader(f).writer_schema
# -> equality_ids: ["null", {"element-id": 136, "type": "array", "items":
"long"}]
# spec says items should be "int"
```
## The obvious fix breaks backwards compatibility
Changing `LongType()` → `IntegerType()` in those two places makes pyiceberg
unable to read **every manifest it has previously written**:
```
ResolveError: Cannot promote long to int
```
Two reasons compound here:
1. `pyiceberg/schema.py` registers `promote` for `IntegerType` (int → long
is legal) but has no `LongType` registration, so long → int falls through to
the generic case and raises (`schema.py:1660`).
2. `MANIFEST_ENTRY_SCHEMAS` is used for **both** directions — the writer
takes `file_schema` from it and the reader takes `record_schema` from it — so
changing the constant changes the read path too.
Verified empirically: patching field 135's element type to `IntegerType` and
then reading a manifest written by the current release raises the above.
## Possible directions
A correct fix has to write `int` while continuing to read `long`. Options,
roughly:
- **Split the write and read schemas** so the writer emits the spec type
while the reader keeps accepting the historical one. Probably the cleanest, and
it changes no promotion rules.
- **Allow long → int narrowing** for this field specifically. Smaller diff,
but it widens the resolver in a direction the Avro and Iceberg promotion rules
deliberately exclude.
Deliberately not proposing a patch — which trade-off is acceptable seems
like a maintainer call, and there may be a preferred pattern for
schema-compatibility shims that I am not aware of.
## Environment
- pyiceberg 0.11.1 (current release) and `main`, both affected
- Cross-checked against the Iceberg spec, the Java reference implementation,
and iceberg-cpp
- Found while making iceberg-cpp read a pyiceberg-written table
## Status checks
**Still present on `main`.** Verified against `main` on 2026-08-24:
`manifest.py:298` and `:393` both still declare `LongType()`. Separately, no
commit in the repository's history mentions `equality_ids` at all, and the
recent commits touching `manifest.py` are unrelated (manifest cache
configuration, min sequence number, `ManifestEntry.snapshot_id` setter, lint
rules).
**No prior report found**, though this is the weaker of the two claims.
Searching this repository for `equality_ids` returns 13 issues and PRs, all of
which mention it incidentally (equality-delete tests, manifest entry status,
the files metadata table) rather than discussing its declared type. Searches
for `"Cannot promote long to int"` and `"Cannot read Iceberg type"` return
nothing. It is entirely possible this has been raised somewhere I did not look
— the dev mailing list or Slack, or phrased in a way my queries missed.
**Related but distinct:** #3744 (*Metadata inspection APIs fail with
`struct.error` after int→long / float→double type promotion*) also concerns
int/long handling, but it is about promotion of **user table columns** breaking
`inspect.*` on pre-promotion data files. It does not involve the manifest-entry
schema or `equality_ids`, so the two should not be conflated.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
