swanandx opened a new issue, #3576:
URL: https://github.com/apache/iceberg-rust/issues/3576
#3277 fixed the two reads that `process` awaits, the file open and the bloom
filter fetch, by spawning each one.
Both fixes are at the call site, and only a comment there says why the spawn
is needed. If someone adds another read to `process` later, the stall comes
back.
A wrapper at the read layer would cover every read, the ones that exist now
and the ones added later:
```rust
struct SpawnedFileRead {
inner: Arc<dyn FileRead>,
runtime: Runtime,
}
#[async_trait::async_trait]
impl FileRead for SpawnedFileRead {
async fn read(&self, range: Range<u64>) -> Result<Bytes> {
let inner = Arc::clone(&self.inner);
self.runtime
.io()
.spawn(async move { inner.read(range).await })
.abort_on_drop()
.await?
}
}
```
The cost is one task per read request, data page reads included. It needs to
be measured properly with this.
A test can target the wrapper on its own, without a scan:
1. Wrap a `FileRead` that yields once and then sets a flag.
2. Poll the `read` future once and never again. Yield a few times, then
check that the flag is set.
3. Do the same, but drop the future after the first poll, and check that the
flag stays unset.
[ ps: this was raised in review comments of #3277 . so refer to that for
more context ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
