saths008 opened a new issue, #18211: URL: https://github.com/apache/iceberg/issues/18211
### Apache Iceberg version 1.11.0 (latest release) ### Query engine Spark ### Please describe the bug 🐞 Spark’s `create_changelog_view` can return incorrect net changes when rows contain non-null binary values. With `net_changes => true`, an `INSERT` followed by a `DELETE` of the same row should cancel. However, `ChangelogIterator#isDifferentValue` compares individual column values using Objects.equals. Spark represents [binary values as a Java `byte[]`](https://github.com/apache/spark/blob/eaa4ade9a7a39ab59c8b2fc7015ce99d3b0b5a29/sql/api/src/main/scala/org/apache/spark/sql/Row.scala#L179), so this is comparing the reference rather than byte contents. So `INSERT` and `DELETE` rows containing identical bytes are treated as different rows. Intermediate changes remain in the output: Operations: `INSERT(A), DELETE(A), INSERT(B)` Expected net changes: `INSERT(B)` Actual net changes: `INSERT(A), DELETE(A), INSERT(B)` The final table snapshot remains correct. However, filtering the changelog to INSERTs can return multiple versions of the same logical row. The shared comparison also affects carryover removal and binary values nested inside arrays or maps. ## Repro ```sql CREATE TABLE catalog.default.binary_cdc_repro (id INT, data BINARY) USING iceberg; INSERT INTO catalog.default.binary_cdc_repro VALUES (1, X'01'), (2, X'01'); INSERT OVERWRITE catalog.default.binary_cdc_repro VALUES (1, X'02'), (2, X'01'); CALL catalog.system.create_changelog_view( table => 'default.binary_cdc_repro', changelog_view => 'binary_cdc_changes', net_changes => true ); SELECT id, hex(data) AS data, _change_type FROM binary_cdc_changes ORDER BY id, _change_ordinal, _change_type; ``` Before the fix: ```text id data _change_type 1 01 INSERT 1 01 DELETE 1 02 INSERT 2 01 INSERT 2 01 DELETE 2 01 INSERT ``` Expected, and returned after the fix: ```text id data _change_type 1 02 INSERT 2 01 INSERT ``` The final table has one row per ID. The net changelog fails to cancel equal INSERT/DELETE images because their binary values are separate `byte[]` instances. Filtering this changelog to INSERTs therefore returns duplicate IDs. ### Willingness to contribute - [x] I can contribute a fix for this bug independently - [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community - [ ] I cannot contribute a fix for this bug at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
