zhuqi-lucas opened a new pull request, #24368: URL: https://github.com/apache/datafusion/pull/24368
Backport of #24354 to `branch-55` for the 55.0.0 release, per @timsaucer's request in #22393. ## Which issue does this PR close? - Backports the fix for #24352 (wrong TopK results from re-reading already-delivered row groups). ## Rationale #24352 is a **silent wrong-results** bug: with `pushdown_filters=true` + TopK dynamic filter pushdown (both on by default), a row group whose post-predicate selection is empty is finished by arrow-rs without handing back a reader, so DataFusion's `rg_plan` trails the decoder frontier by one and a later runtime prune rebuilds the decoder from a stale plan — re-reading an already-delivered row group, duplicating rows and dropping the true top-k tail. No error is raised. This is a clean cherry-pick of the squashed #24354 commit (`574fe67`); it applies to `branch-55` without conflicts. ## What changes are included? `push_decoder.rs`: sync `rg_plan` to the decoder frontier via `peek_next_row_group()` before each runtime prune/rebuild (gated on `row_group_pruner.is_some()` so ordinary scans pay nothing), with a defensive `internal_err!` if the frontier diverges from the plan. Plus the slt + rust regression tests from #24354. cc @timsaucer @alamb @adriangb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
