JeonDaehong commented on code in PR #3540:
URL: https://github.com/apache/iceberg-rust/pull/3540#discussion_r4443033043
##########
crates/iceberg/src/arrow/reader/row_filter.rs:
##########
@@ -62,8 +70,27 @@ impl ArrowReader {
// After collecting required leaf column indices used in the predicate,
// creates the projection mask for the Arrow predicates.
let projection_mask = ProjectionMask::leaves(parquet_schema,
column_indices.clone());
- let predicate_func = visit(&mut converter, predicates)?;
- let arrow_predicate = ArrowPredicateFn::new(projection_mask,
predicate_func);
+ let mut predicate_func = visit(&mut converter, predicates)?;
+ // Predicates are bound to the table schema, so evaluate them on the
filter columns
+ // converted to it, as the reader's output is (e.g. a file written
before an `int` ->
+ // `long` promotion). The transformer's schema holds only the
top-level filter fields, so
+ // its per-file setup does not grow with the table's width.
+ let filter_schema = Schema::builder()
+ .with_schema_id(snapshot_schema.schema_id())
+ .with_fields(
+ field_ids
+ .iter()
+ .filter_map(|id|
snapshot_schema.as_struct().field_by_id(*id).cloned()),
Review Comment:
`as_struct().field_by_id` only looks at top-level fields, so a nested leaf
never makes it into `filter_schema`. Not an issue on main yet, but I merged
this with #2906 locally and 5 of its nested predicate tests fail with `field
not found` (they pass on #2906 alone). Maybe build the filter schema from the
struct parents of those leaves?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
