mbutrovich commented on code in PR #3302:
URL: https://github.com/apache/iceberg-rust/pull/3302#discussion_r4606315356
##########
crates/iceberg/src/arrow/reader/predicate_visitor.rs:
##########
@@ -195,6 +197,251 @@ impl BoundPredicateVisitor for CollectFieldIdVisitor {
}
}
+/// Returns the residual of `predicate` for one data file: each leaf on a
top-level field that is
+/// missing from the file becomes `AlwaysTrue` or `AlwaysFalse`, based on the
value projection
+/// returns for that field. That value is the identity partition value,
otherwise the field's
+/// `initial-default`, and every row of the file holds it. Leaves on missing
fields with neither
+/// keep the null handling of the row filter and the page index evaluator. The
resulting constants
+/// are folded out of `AND` and `OR`.
+///
+/// Leaves on nested fields are kept, because a nested field also reads as
null in any row where
+/// an ancestor struct is null.
+pub(super) fn residual_for_missing_fields(
+ predicate: BoundPredicate,
+ predicate_field_ids: &HashSet<i32>,
+ field_id_map: &HashMap<i32, usize>,
+ schema: &Schema,
+ partition_spec: Option<&PartitionSpec>,
+ partition: Option<&Struct>,
+) -> Result<BoundPredicate> {
+ if predicate_field_ids
+ .iter()
+ .all(|id| field_id_map.contains_key(id))
+ {
+ return Ok(predicate);
+ }
+
+ let partition_constants = match (partition_spec, partition) {
+ (Some(spec), Some(data)) => constants_map(spec, data, schema)?,
Review Comment:
> The manifest is decoded with its own schema, so after `b` is promoted from
int to long, `task.partition()` holds `Literal::int(7)` and `constants_map`
builds `Datum::new(Long, Int(7))`.
Thanks @JeonDaehong. #3483 fixed this at planning time after your review.
The scan now [widens the partition tuple to the current
schema](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/scan/context.rs#L148)
when it reads a manifest, so `FileScanTask::partition()` holds
`Literal::long(7)` for a file written before the promotion.
[`test_scan_widens_old_partition_so_pruning_and_deletes_match`](https://github.com/apache/iceberg-rust/blob/7cf825fcfb64a8943ddb627ef7f0b9f09f0c8046/crates/iceberg/src/scan/mod.rs#L1902-L1906)
checks that. I merged `main` in d3a3f7c1bbed1797d3543846c9d1e766464c8cae, so
`constants_map` gets the widened value on this branch, and the residual and
projection both see `long(7)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]