a-agmon commented on issue #585: URL: https://github.com/apache/iceberg-rust/issues/585#issuecomment-2310334221
Hi @sdd , @FANNG1 , @liurenjie1024. I just want to make sure im in the right direction with this and would be happy to hear your feedback on this. The main functionality will be implemented in `IcebergTableScan` [here](https://github.com/apache/iceberg-rust/blob/905ebd2f332cd37fc86a3ddf845c335229d0abcd/crates/integrations/datafusion/src/physical_plan/scan.rs#L49). we can pass the filters `&[Expr]` via the `IcebergTableScan::new` function, and then convert them into Iceberg Predicates. Something like this: ```rust pub(crate) fn new(table: Table, schema: ArrowSchemaRef, filters: &[Expr]) -> Self { let plan_properties = Self::compute_properties(schema.clone()); let predicates = Self::convert_to_predicates(filters); println!("==> filters: {:?}", filters); Self { table, schema, plan_properties, predicates, } } ``` Then the predicates needs to be applied as followes: The impl of the `ExecutionPlan::execute()` function, calls `get_batch_stream()`, which in turn calls `table.scan()` [link](https://github.com/apache/iceberg-rust/blob/905ebd2f332cd37fc86a3ddf845c335229d0abcd/crates/integrations/datafusion/src/physical_plan/scan.rs#L128) In this function, the predicate can be applied to the `TableScanBuilder` Another important issue is that it seems that according to DataFusion docs, in order to get the filters passed to the `scan()` function, we need to impl the `TableProvider::supports_filters_pushdown` to return the filters we can push down [docs] (https://docs.rs/datafusion/41.0.0/datafusion/catalog/trait.TableProvider.html#method.supports_filters_pushdown) what do you think? does this make sense? (as a start I think I will try to implement just BinaryExpressions and see from there) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
