tbirch-cyber commented on PR #2332: URL: https://github.com/apache/iceberg-python/pull/2332#issuecomment-3198306711
> hey @tbirch-cyber thanks for the PR. could you explain whats the reasoning behind this change? Hi @kevinjqliu thanks for taking the time to review! Sorry I should have put more in the PR message. Basically, it is very common to have a catalog split across multiple "storage accounts" (similar to an S3 bucket). Because of this, I needs to be a way to provide a credential with an audience of "https://storage.azure.com"; once and have the library determine which storage accounts it needs to access on-the-fly. The fsspec library can only handle one "storage account" per filesystem instantiation, so a separate filesystem needs to be cached for each storage account. `RestCatalog( "my-catalog", **{ "uri": "http://my-catalog";, "header.Authorization": f"Bearer {my_token}", "adls.credential": my_cred, }, )` I think a similar approach could probably also be used with S3: https://github.com/fsspec/s3fs/blob/main/s3fs/core.py#L203 Let me know if there’s anything else you’d like changed or if you think a different approach is needed. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
