zhuqi-lucas opened a new issue, #21780: URL: https://github.com/apache/datafusion/issues/21780
## Background In #21580, `DynamicFilterPhysicalExpr` was given a `fetch: Option<usize>` field that stores the TopK K value. This value is copied from `SortExec.fetch` at filter creation time in `SortExec::create_filter()`. ## Problem If a future optimizer calls `SortExec::with_fetch()` independently (without recreating the filter), the `DynamicFilterPhysicalExpr.fetch` would go stale. Currently this doesn't happen because `create_filter` and `fetch` are set in the same method, but the coupling is fragile. Raised by @xudong963 in https://github.com/apache/datafusion/pull/21580#discussion_r3123894244 ## Proposed fix Replace `fetch: Option<usize>` with a shared reference (e.g., `Arc<AtomicUsize>`) that reads directly from `SortExec.fetch`. This way any update to fetch is automatically visible to the parquet reader's stats init and cumulative pruning logic. ## Related - #21580 — PR that introduced `fetch` on `DynamicFilterPhysicalExpr` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
