adriangb commented on code in PR #24086:
URL: https://github.com/apache/datafusion/pull/24086#discussion_r4108649979
##########
datafusion/datasource-parquet/src/opener/mod.rs:
##########
@@ -651,7 +652,31 @@ impl ParquetOpenState {
}
ParquetOpenState::PruneWithStatistics(prepared) => {
let mut prepared_row_groups = (*prepared).prune_row_groups()?;
- if prepared_row_groups.should_load_page_index()? {
+ // EXPERIMENT: with an offset index the streaming policy
+ // fetches and releases pages; without one it fetches whole
+ // column chunks. Page-index loading is otherwise driven purely
+ // by whether *pruning* can use it, so load it here too when a
+ // row group to read is larger than the read-ahead window: a
+ // row group that fits gains little from page granularity.
Review Comment:
These comments are too verbose. WE can't have comments starting with
`EXPERIMENT:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
