swanandx commented on code in PR #3277:
URL: https://github.com/apache/iceberg-rust/pull/3277#discussion_r4367641330
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -3795,4 +3811,220 @@ mod tests {
let total_rows: usize = batches.iter().map(|b| b.num_rows()).sum();
assert_eq!(total_rows, 3);
}
+
+ /// A data file must be opened even when nothing polls the scan. The stream
+ /// stops polling the opens once it holds `concurrency_limit_data_files` of
+ /// them, and on a shared HTTP/2 connection an open stopped mid-request
holds
+ /// part of the connection window, which stalls every other read (#3276).
+ mod open_progress {
+ use std::ops::Range;
+
+ use async_trait::async_trait;
+ use bytes::Bytes;
+ use futures::StreamExt;
+ use futures::stream::BoxStream;
+ use serde::{Deserialize, Serialize};
+
+ use super::*;
+ use crate::Result;
+ use crate::arrow::scan_metrics::ScanMetrics;
+ use crate::io::{
+ FileIOBuilder, FileMetadata, FileRead, FileWrite, InputFile,
MemoryStorage, OutputFile,
+ Storage, StorageConfig, StorageFactory,
+ };
+ use crate::scan::ArrowRecordBatchStream;
+
+ const CONCURRENCY: usize = 2;
+ /// Smaller than a footer, so one open takes more than one request.
+ const METADATA_SIZE_HINT: usize = 64;
+ /// Scheduler turns, far more than a request needs to finish.
+ const YIELDS: usize = 100;
+
+ /// In-memory files whose reads take a scheduler turn to answer.
+ #[derive(Debug, Clone, Default, Serialize, Deserialize)]
+ struct CountedStorage {
Review Comment:
rename done.
for rest, basically something like this?:
```rust
struct SpawnedFileRead(Arc<dyn FileRead>);
impl FileRead for SpawnedFileRead {
async fn read(&self, range: Range<u64>) -> Result<Bytes> {
let inner = Arc::clone(&self.0);
spawn(async move { inner.read(range).await
}).abort_on_drop().await?
}
}
```
i couldn't repro stall for bloom filters, so kept the fix targeted.
but if this is still preferred [ cc: @comphead ], i can either do it in this
PR or this can be follow up. tradeoff can be cost becomes a task per read
request, data pages included?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
