adriangb commented on PR #24354: URL: https://github.com/apache/datafusion/pull/24354#issuecomment-5290007275
Is there any way we could structurally eliminate the possibility of drift? E.g. if we added: ```rust pub fn remaining_row_groups(&self) -> impl ExactSizeIterator<Item = usize> + '_ ``` Then the prune becomes decoder.remaining_row_groups().filter(|rg| !pruner.should_prune(&[*rg])), and rg_plan, RgPlanEntry, sync_rg_plan_to_decoder_frontier, advance_rg_plan_to, the pop in the Data arm, and the "we MUST build our rg_plan from this reordered list" comment in opener/mod.rs all delete. This is suggestion (2) in https://github.com/apache/datafusion/issues/24352#issue-5147368303. Or we give arrow-rs `decoder.retain_row_groups(impl FnMut(usize) -> bool)` that filters the frontier in place. Then there is no into_builder, no with_row_groups, no is_at_row_group_boundary gate, no rebuild. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
