mohitgurav20 commented on issue #25775: URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5845850170
Hi @ion-elgreco @alamb, Since `DataSourceExec` delegates execution details to `Arc<dyn DataSource>`, we can support runtime file streams and child nodes by extending `DataSource` with child plan awareness: 1. **`DataSource` trait extension**: Add default `children(&self) -> Vec<&Arc<dyn ExecutionPlan>>` and `with_new_children(...)` methods to `DataSource`. 2. **`DataSourceExec` tree delegation**: Update `DataSourceExec::children()` and `with_new_children()` to delegate directly to `self.data_source`. 3. **Runtime execution context**: In `DataSourceExec::execute(partition, context)`, execute any child execution plans for that partition and pass their `SendableRecordBatchStream` outputs into `OpenArgs::inputs` (or `open_with_args`). This allows custom sources like `delta-rs` or `iceberg-rs` (or specialized `FileScanConfig` extensions) to take child metadata plans, consume runtime file skipping information during `open_with_args`, and stream only the kept files seamlessly. Would this design fit the requirements of `delta-io/delta-rs#4790`? I can draft the PR in `datafusion/datasource` if this approach looks good to everyone! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
