ranflarion opened a new pull request, #18557: URL: https://github.com/apache/iceberg/pull/18557
Closes #18556 `SparkBatch.equals` compares the hash code of the scan that created the batch, so Spark treats the batches of scans whose hash codes collide as equal and reuses one scan's exchange for the other, returning wrong rows. The issue has a reproduction with `'Aa'` and `'BB'` filters in a `UNION ALL`. Each scan now lists the values that define its equality once, in `equalityKey()`. `SparkScan` implements `equals` and `hashCode` over that list (`SparkChangelogScan`, which does not extend it, does the same), and `toBatch()` passes the list to `SparkBatch` in place of the hash code. The batch compares the values as they were when it was created, as it did with the hash code, so a runtime filter applied to the scan afterwards does not change the batch's equality. The scans' own equality and hash codes are unchanged. `TestSparkScan` covers batch equality for scans whose hash codes collide and the `UNION ALL` result from the issue; both fail without the change. This changes the latest Spark version only; the backports to 3.5, 4.0 and 4.1 can follow once the approach is agreed. #13437 proposed a stronger digest for the same problem, which makes collisions rarer but keeps them possible. AI disclosure: an AI coding assistant helped draft this change and its tests. I reviewed the design and the code, and ran `TestSparkScan`, `TestFilteredScan`, `TestRuntimeFiltering`, `TestSparkStagedScan`, `TestChangelogTable`, `TestCopyOnWriteUpdate` and `TestRemoteScanPlanning` locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
