mohitgurav20 commented on issue #25775: URL: https://github.com/apache/datafusion/issues/25775#issuecomment-5857489348
Hi @alamb, thanks for jumping in. That's a fair point—my previous comment was definitely over-rotating towards a heavy "Option 2" implementation (treating metadata logs as actual executable child plans of `DataSourceExec`). I agree that modifying the core trait and execution tree structure like that is unnecessarily invasive if we don't strictly need it. Looking closely at what @ion-elgreco is describing for the Delta MERGE use case, it heavily leans toward your **Option 1**. They already have the baseline file list at planning time (since `FileScanConfig.file_groups` is populated via the initial log replay), but they need a mechanism to drop/skip specific target files at runtime once the source-side statistics become available. If we tackle this purely as a dynamic pruning problem, the existing `FilePruner` machinery you linked is exactly the right foundation. The missing piece for the MERGE case seems to be how to cleanly pipe that *runtime-discovered* state (which is evaluated on a completely separate execution branch) into the target `DataSourceExec`'s file opener. Since the source statistics come from a separate plan, do you envision doing something akin to the existing join `DynamicFilter` mechanism? For example, passing a shared future or state channel into the target's `FileOpener` (or a specialized `DataSource`), which waits for the source statistics at runtime and then applies the `FilePruner` to skip files before allocating the parquet readers? If you feel this direction (leveraging shared runtime state + `FilePruner` rather than child nodes) is more idiomatic for DataFusion, I'm completely on board. I can pivot my approach to draft a PoC showing how we might pipe these dynamic statistics into the file opener. @ion-elgreco, does this sound like it would satisfy the copy-on-write MERGE requirement without needing you to dynamically rewrite the execution plan at runtime? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
