haohuaijin opened a new issue, #23883: URL: https://github.com/apache/datafusion/issues/23883
### Is your feature request related to a problem or challenge? [arrow-rs#10141](https://github.com/apache/arrow-rs/pull/10141) allows `RowSelection` to be backed by a `BooleanBuffer`. However, `ParquetAccessPlan::into_overall_row_selection` currently calls `RowSelection::iter()` and converts each selection into a `Vec<RowSelector>`. This materializes mask-backed selections as RLE and loses the bitmap backing before it reaches the Parquet reader. This issue is blocked until DataFusion upgrades to an Arrow/Parquet release containing `arrow-rs#10141`. ### Describe the solution you'd like - Validate selection length using `row_count() + skipped_row_count()` instead of `iter()`. - When all row-group selections are mask-backed, concatenate their `BooleanBuffer`s directly. - Represent scanned row groups as all-set bitmap ranges and omit skipped row groups. - Keep the existing selector path as the fallback for selector-backed or mixed inputs. ### Describe alternatives you've considered Converting selectors back to a bitmap in the Parquet reader does not help because the large intermediate `Vec<RowSelector>` has already been allocated. ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
