Kaileshwar16 opened a new issue, #3310:
URL: https://github.com/apache/iceberg-rust/issues/3310
The OpenDAL FileIO adapter applies `TimeoutLayer::new()` to every storage
operator without
exposing a property to configure its per-I/O timeout.
Confirmed on commit `cf365628`, using the repository-pinned OpenDAL 0.58.1.
In `crates/storage/opendal/src/lib.rs`, `OpenDalStorage::create_operator`
installs:
operator.layer(TimeoutLayer::new()).layer(RetryLayer::new())
OpenDAL defaults to a 10-second timeout for individual I/O calls, including
writer
`write()` and `close()`. Slow object-store requests or sufficiently large
multipart
uploads can exceed this limit even while making progress.
RetryLayer retries these temporary errors, but each attempt receives the
same timeout. If
every attempt requires more than 10 seconds, retries repeatedly fail and
eventually return
a persistent error.
There is currently no supported FileIO property to increase or disable this
timeout. The
problem affects the shared OpenDAL adapter, not only S3.
Deterministic reproduction:
- Inject an artificial writer requiring 11 seconds into the unchanged
OpenDalStorage
adapter.
- Use Tokio’s paused clock to exercise the actual default timeout without
real AWS
credentials.
- Observe four attempts, zero successful completions, and failure after 47
virtual
seconds.
- Configure TimeoutLayer directly with a 20-second I/O timeout: the same
operation
succeeds on its first attempt.
Observed error:
Unexpected (persistent) at write,
context: { timeout: 10 } => io operation timeout reached
The timeout is per individual I/O call, not per complete file upload. A
sequence of
shorter writes can successfully take more than 10 seconds overall.
Expected behavior:
Expose a common FileIO property such as `client.io-timeout-ms`, preserve the
existing
default when absent, and reject invalid values clearly. Apply the setting
through both
explicit and resolving OpenDAL storage factories while preserving the
existing timeout-
before-retry layer order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
