zakariya-s commented on code in PR #2976:
URL: https://github.com/apache/iceberg-rust/pull/2976#discussion_r4376494625
##########
crates/iceberg/src/io/storage/mod.rs:
##########
@@ -139,4 +140,201 @@ pub trait StorageFactory: Debug + Send + Sync {
/// A `Result` containing an `Arc<dyn Storage>` on success, or an error
/// if the storage could not be created.
fn build(&self, config: &StorageConfig) -> Result<Arc<dyn Storage>>;
+
+ /// Build a new Storage instance, optionally supplying a credential
provider
+ /// that the backend can call to obtain and refresh short-lived
credentials.
+ ///
+ /// Backends that cannot use the provider ignore it and use the credentials
+ /// in `config`, as they would without one. The default does so and logs a
+ /// warning once per process, since the storage's credentials are then not
+ /// refreshed. Factories that wrap another factory should forward the
+ /// provider to it, and factories whose storage needs no credentials can
+ /// ignore it silently.
+ fn build_with_credential_provider(
+ &self,
+ config: &StorageConfig,
+ credential_provider: Option<Arc<dyn StorageCredentialProvider>>,
+ ) -> Result<Arc<dyn Storage>> {
+ if credential_provider.is_some() {
+ static WARNED: Once = Once::new();
+ WARNED.call_once(|| {
+ tracing::warn!(
+ "{} ignores storage credential providers; its storage uses
the credentials \
+ in its configuration, which are not refreshed",
+ std::any::type_name::<Self>()
+ )
+ });
+ }
+ self.build(config)
+ }
+}
+
+/// Supplies fresh, backend-specific storage credentials on demand.
+///
+/// A catalog that vends temporary credentials implements this trait so that
+/// storage backends can re-fetch credentials as they approach expiry instead
+/// of failing once the initial token's TTL runs out.
+///
+/// # Debug output
+///
+/// [`FileIO`](crate::io::FileIO) and storage implementations print the
+/// provider in their `Debug` output, so the provider's `Debug` implementation
+/// must not expose credentials or other secrets.
+///
+/// # Caching
+///
+/// [`load_credential`](Self::load_credential) may be called very frequently.
+/// Implementations must cache internally and only re-fetch when the current
+/// credential is at or near expiry; otherwise every object-store request could
+/// trigger a call back to the catalog.
+///
+/// Backends may treat a credential as stale before it expires, and load
+/// another on every request until they get a fresher one. The OpenDAL S3 and
+/// GCS backends do so within two minutes of expiry, and reject a newly loaded
+/// credential that expires within ten seconds. Implementations should
+/// therefore refresh early enough that the credential they return has more
+/// than two minutes left.
+#[async_trait]
+pub trait StorageCredentialProvider: Debug + Send + Sync {
+ /// Return whether this provider supplies credentials for `path`.
+ ///
+ /// Backends replace their own credential chain with the provider only for
+ /// paths it supports, so a provider must return `false` for the storage it
+ /// does not serve, such as other schemes behind a resolving storage.
+ fn supports_path(&self, path: &str) -> bool;
+
+ /// Load a fresh credential for the storage location identified by `path`.
+ ///
+ /// `path` is an absolute location: usually the file being accessed, such
+ /// as `s3://bucket/warehouse/db/table/data/file.parquet`, but for a bulk
+ /// delete the location shared by a batch, which may be a storage root
+ /// (`s3://bucket/`) or the prefix of a previously returned credential
+ /// (`s3://bucket/warehouse/db/table`). Providers that vend distinct
+ /// credentials per location prefix use it to select the most specific
+ /// match, which must [cover](StorageCredential::covers) `path`.
+ ///
+ /// Backends read the credential's expiry from its config, such as
+ /// `s3.session-token-expires-at-ms`, and load a new one before it expires.
+ /// A credential without an expiry is used for as long as the backend
lives.
+ ///
+ /// An error, or a credential that does not cover `path`, fails the
request:
+ /// backends do not fall back to other credentials.
+ async fn load_credential(&self, path: &str) -> Result<StorageCredential>;
Review Comment:
added `StorageCredential::select` and pointed the `load_credential` docs at
it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
