comphead commented on code in PR #3540:
URL: https://github.com/apache/iceberg-rust/pull/3540#discussion_r4592849575
##########
crates/iceberg/src/arrow/reader/row_filter.rs:
##########
@@ -62,8 +70,27 @@ impl ArrowReader {
// After collecting required leaf column indices used in the predicate,
// creates the projection mask for the Arrow predicates.
let projection_mask = ProjectionMask::leaves(parquet_schema,
column_indices.clone());
- let predicate_func = visit(&mut converter, predicates)?;
- let arrow_predicate = ArrowPredicateFn::new(projection_mask,
predicate_func);
+ let mut predicate_func = visit(&mut converter, predicates)?;
+ // Predicates are bound to the table schema, so evaluate them on the
filter columns
+ // converted to it, as the reader's output is (e.g. a file written
before an `int` ->
+ // `long` promotion). The transformer's schema holds only the
top-level filter fields, so
+ // its per-file setup does not grow with the table's width.
+ let filter_schema = Schema::builder()
+ .with_schema_id(snapshot_schema.schema_id())
+ .with_fields(
+ field_ids
+ .iter()
+ .filter_map(|id|
snapshot_schema.as_struct().field_by_id(*id).cloned()),
Review Comment:
Thanks for merging it with #2906. Fixed in 08d7afe70: each leaf now maps to
its top-level field through the schema accessor, and that field is pruned to
the filter leaves with `prune_columns`. On a local merge with today's #2906, 6
of its tests fail on 03639f1bc and 5 of those pass on 08d7afe70. The one still
failing is `test_predicate_on_nested_struct_leaf_reads_real_parquet`, with
"Predicate column root `person` not found". The transformer relabels the filter
batch with table names (`who`), and #2906's `project_column` walks the file's
names. Resolving the path through the table schema in #2906 should fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]