mbutrovich commented on code in PR #3302:
URL: https://github.com/apache/iceberg-rust/pull/3302#discussion_r4298011461
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -603,6 +604,14 @@ impl FileScanTaskReader {
&predicate,
use_position_fallback,
)?;
+ let predicate = residual_for_missing_fields(
Review Comment:
> The residual keeps the `OR` node and doesn't fold it into one constant.
This has changed. The residual now [folds constants out of `AND` and
`OR`](https://github.com/apache/iceberg-rust/blob/006d4b9a648b49e7ffbed59a2d2da9c8f425bd70/crates/iceberg/src/arrow/reader/predicate_visitor.rs#L296-L316),
as Java's `ResidualEvaluator` does through
[`Expressions.and`](https://github.com/apache/iceberg/blob/5e7169168db3d34e29354c6f59ec4d6e420b8d2d/api/src/main/java/org/apache/iceberg/expressions/Expressions.java#L33-L44)
and
[`Expressions.or`](https://github.com/apache/iceberg/blob/5e7169168db3d34e29354c6f59ec4d6e420b8d2d/api/src/main/java/org/apache/iceberg/expressions/Expressions.java#L50-L61).
Without folding, the keep predicate of a delete file with N rows left N `OR`
nodes of constants. The row filter then built a constant array for each leaf
and ran a kernel for each node on every batch. `b IS NULL OR b != 7` now
becomes `AlwaysFalse`, and `b IS NULL OR b != 5` becomes `AlwaysTrue`. The unit
test also covers a constant on either side of `AND` and `OR`, and
an `AND` with no constant child that is kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
