zhuqi-lucas commented on issue #24352: URL: https://github.com/apache/datafusion/issues/24352#issuecomment-5289348150
Thanks @hhhizzz for the exceptionally detailed report — the fixture, the instrumented trace, and the causal check that gates the step-2 prune/rebuild block made this immediate to confirm. This is already fixed by #23696, which reworks exactly this bookkeeping. It adds `sync_rg_plan_to_decoder_frontier`: at every row-group boundary (before any prune/rebuild) it asks the decoder via `peek_next_row_group()` which RG it will actually emit next and advances `rg_plan` to match. So a silently-finished empty-selection row group no longer leaves `rg_plan` trailing the decoder, and a rebuild can never re-include an already-delivered group — the "ask the decoder for the outstanding row groups" direction you suggested. (It also adds an internal-error guard for the case where a rebuild frontier names an RG not in the plan.) Verified against the #23696 branch on your fixture: it returns the correct `p0 p4096 p4097 … p4104` (vs the buggy `p0 p4096 p4096 …` on main). I added your scenario as a regression test there (`dynamic_row_group_pruning.slt`) and marked the PR `Closes #24352`. cc @alamb @adriangb — flagging that #23696 fixes this correctness bug in addition to the perf optimization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
