zhuqi-lucas commented on issue #24358:
URL: https://github.com/apache/datafusion/issues/24358#issuecomment-5290732704
Upstream dependency: **apache/arrow-rs#10624** already tracks the
arrow-rs-side API for this — @haohuaijin proposes
`with_row_group_selections(vec![RowGroupSelection { row_group_index, selection
}])`, i.e. the push decoder accepts **row-group-local** selections directly,
and explicitly *"rebuilding a push decoder preserves the remaining local
selections."*
That's a cleaner shape than the `retain_row_groups(FnMut)` I sketched above,
and it root-causes the whole drift family: with per-RG selections, dropping a
row group on rebuild keeps every survivor's selection aligned by construction,
so there's no global-selection slicing to get wrong (#24355) and no parallel
`rg_plan` to drift (#24352).
So the plan for this issue is:
1. arrow-rs#10624 lands `with_row_group_selections` (+ rebuild preserving
local selections).
2. DataFusion bumps parquet, then replaces `into_overall_row_selection` →
global `with_row_selection` + `rg_plan` / `sync_rg_plan_to_decoder_frontier` /
the `into_builder().with_row_groups()` dance with the row-group-local API.
3. Remove the stop-the-bleeding guards (#24354, #24359).
Have linked this from arrow-rs#10624. Deferring the design here to that
issue rather than adding a competing `retain_row_groups` proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
