arthurli-dotcom opened a new pull request, #16071:
URL: https://github.com/apache/iceberg/pull/16071
…runing
When Spark pushes predicates through UNION ALL views to Iceberg,
contradictory filters like `col = 'a' AND col != 'a'` can end up combined in a
single scan. Neither Spark's Catalyst optimizer nor Iceberg's expression system
detected these contradictions, causing Iceberg to plan tens of thousands of
file tasks that would return zero rows.
This adds contradiction detection to Expressions.and() for:
- EQ(col, a) AND NOT_EQ(col, a) with same value
- EQ(col, a) AND EQ(col, b) with different values
- EQ(col, val) AND NOT_IN(col, {val, ...})
- EQ(col, val) AND IN(col, set) where val not in set
- Contradictions through nested AND expressions
Handles both bound predicates (post-binding) and unbound predicates
(pre-binding, as used by SparkScanBuilder).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
