YinZheng-Sun commented on code in PR #956:
URL: https://github.com/apache/iceberg-cpp/pull/956#discussion_r4237286260
##########
src/iceberg/parquet/parquet_reader.cc:
##########
@@ -387,39 +406,58 @@ class ParquetReader::Impl {
}
private:
+ Status OpenNextRowGroupRange() {
+ const size_t begin = context_->next_row_group_;
+ size_t end = begin + 1;
+ std::vector<int> row_groups{context_->row_groups_[begin].index};
+ // Share a reader across adjacent row groups so Arrow can coalesce
pre-buffered
+ // reads. Stop at gaps to keep physical row positions contiguous within
batches.
+ while (end < context_->row_groups_.size() &&
+ context_->row_groups_[end].index == context_->row_groups_[end -
1].index + 1) {
+ row_groups.push_back(context_->row_groups_[end].index);
+ ++end;
+ }
Review Comment:
This change is needed when integrating row-group pruning with physical
row-position tracking. Previously, the reader passed all selected row groups to
one GetRecordBatchReader() call. Pruning can introduce gaps, so the new
implementation splits the reader at those gaps to prevent a batch from spanning
discontinuous file positions.
Keeping adjacent groups together preserves the existing multi-row-group
batching and pre-buffering behavior, instead of creating a reader for every
group. I’ll clarify this motivation in the comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
