advancedxy opened a new pull request, #1006: URL: https://github.com/apache/iceberg-cpp/pull/1006
## What Add a `RowNumberReader`, modeled after parquet-rs, that emits absolute file row positions from the selected Parquet row groups. Use the resulting position array for `_pos` and inherited `_row_id` in record-batch projection. Preserve physical lineage values and cross-row-group batching. Restore consumed ranges if array construction fails so the batch can be retried. ## Why Follow-up to the [row-position tracking discussion in #956](https://github.com/apache/iceberg-cpp/pull/956#discussion_r4237138947). Updating `next_file_pos` by the emitted batch size assumes contiguous file rows. Row-group pruning can introduce gaps, so each selected row must retain its original position in the file. ## Testing - `parquet_test`, `arrow_test`, and `avro_test` passed. - Coverage includes skipped/reordered row groups, batches spanning row groups, physical and inherited lineage, and allocation-failure recovery. - Pre-commit and `git diff --check` passed. ## Limitations and follow-up The Iceberg-side `RowNumberReader` can account for row-group-level skipping only: it knows the selected row groups and their row counts. It cannot account for page-level skipping inside Arrow C++ because the skipped row ranges are not exposed alongside the output batches; batch lengths alone are insufficient to recover the original positions. Supporting page-level skipping requires integrating row-number generation into Arrow C++'s Parquet reader, where it can stay synchronized with page skipping and the actual rows emitted. This is the planned follow-up. Iceberg C++ can then consume that position array for `_pos` and inherited `_row_id`, replacing its local `RowNumberReader`. Keeping position generation separate from Iceberg metadata semantics prepares for that integration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
