laskoviymishka commented on code in PR #3277:
URL: https://github.com/apache/iceberg-rust/pull/3277#discussion_r4411024099
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -157,16 +160,30 @@ impl FileScanTaskReader {
.delete_file_loader
.load_deletes(task.deletes(), task.schema_ref());
- // Open the Parquet file once, loading its metadata
- let (parquet_file_reader, arrow_metadata) =
ArrowReader::open_parquet_file(
- task.data_file_path(),
- &self.file_io,
- task.file_size_in_bytes(),
- parquet_read_options,
- self.scan_metrics.bytes_read_counter(),
- task.key_metadata(),
- )
- .await?;
+ // `try_flatten_unordered` stops polling the buffered opens once it
holds
+ // `concurrency_limit_data_files` streams, so spawn the open (#3276).
+ // Aborts on drop.
Review Comment:
Could the comment also say that decode runs in this task? `get_metadata`
decodes the footer, plus the page index when `preload_page_index` is set,
inside the spawned open
([file_reader.rs](https://github.com/apache/iceberg-rust/blob/89d425cc6432ea2ef9da3ce2cdb9b10730059396/crates/iceberg/src/arrow/reader/file_reader.rs#L109-L139)).
With `Runtime::new_with_split` that work now runs on the IO runtime, where
`main` ran it on whatever polls the scan.
```suggestion
// `try_flatten_unordered` stops polling the buffered opens once it
holds
// `concurrency_limit_data_files` streams, so spawn the open (#3276).
// Footer and page-index decode also run on the IO runtime.
// Aborts on drop.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
