lawofcycles opened a new issue, #3236: URL: https://github.com/apache/iceberg-python/issues/3236
### Apache Iceberg version 0.11.0 (latest release) ### Please describe the bug 🐞 ### Description When PyIceberg performs a Copy-on-Write delete (`table.delete()`) or overwrite, the DELETED manifest entries written to the new manifest retain the `snapshot_id` of the snapshot that originally added the file, instead of the snapshot that is deleting it. According to [Iceberg spec (Manifest Entry Fields)](https://iceberg.apache.org/spec/#manifest-entry-fields): > **snapshot_id**: Snapshot id where the file was added, **or deleted if status is 2 (deleted)**. For entries with status=2 (DELETED), the `snapshot_id` should be set to the current (deleting) snapshot's ID. ### Impact The `snapshot_id` field on manifest entries is part of the Iceberg spec and any implementation that relies on it for DELETED entries will produce incorrect results. For example, Iceberg Java's `IncrementalChangelogScan` filters manifest entries by `snapshot_id` membership in the changelog snapshot set. ```java // BaseIncrementalChangelogScan.java .filterManifestEntries(entry -> changelogSnapshotIds.contains(entry.snapshotId())) ``` ### Reproduction ```python schema = Schema( NestedField(1, "id", StringType(), required=True), NestedField(2, "value", LongType()), ) table = catalog.create_table("default.bug_repro", schema=schema) # Step 1: INSERT df = pa.table({"id": [f"row-{i}" for i in range(10)], "value": list(range(10))}, schema=table.schema().as_arrow()) table.append(df) insert_snap = table.current_snapshot() # Step 2: DELETE (CoW overwrite) table.delete(delete_filter="id >= 'row-5'") delete_snap = table.current_snapshot() ``` ### Manifest comparison: PyIceberg vs Spark To confirm this is a PyIceberg-specific issue, I compared the raw manifest Avro files produced by PyIceberg and Spark for equivalent overwrite operations using `fastavro`. **PyIceberg** (DELETE snapshot_id=1582763037918157070, deleting a file originally added by snapshot 2824688255802688948): ``` # manifest <uuid>-m0.avro (new file) [0] status=ADDED snapshot_id=1582763037918157070 records=5 file=<new>.parquet # manifest <uuid>-m1.avro (deleted file) [0] status=DELETED snapshot_id=2824688255802688948 records=10 file=<old>.parquet ^^^^^^^^^^^^^^^^^^ Points to the original INSERT snapshot. Should be 1582763037918157070 (the deleting snapshot). ``` **Spark** (overwrite snapshot_id=3532553658031297701): ``` # manifest <uuid>-m0.avro (deleted files) [0] status=DELETED snapshot_id=3532553658031297701 records=70495 file=...-00001.parquet [1] status=DELETED snapshot_id=3532553658031297701 records=73361 file=...-00002.parquet [2] status=DELETED snapshot_id=3532553658031297701 records=80466 file=...-00003.parquet ... # manifest <uuid>-m2.avro (new files) [0] status=ADDED snapshot_id=3532553658031297701 records=81013 file=...-00001.parquet ... ``` ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
