kevinjqliu commented on PR #3507: URL: https://github.com/apache/iceberg-python/pull/3507#issuecomment-4760415412
We can reuse the `LoadCredentialsResponse` components from #3499 now that its merged 😄 . I think it'll be a good idea to keep separate concerns between the Rest Catalog, FileIO, and object storage credentials. Here's how I'm thinking about it: - RestCatalog: owns REST/auth/session concerns. It should parse inline credentials, know how to call `/credentials`, and create a credential provider. It should not make PyArrow/Fsspec-specific decisions. - CredentialsProvider: owns credential refresh and path lookup. A good Python contract is something like `properties_for(location) -> Properties`. It can keep the full StorageCredential list, refresh when needed, and do longest-prefix matching internally. This avoids duplicating prefix logic in every FileIO. - FileIO: owns file access and backend cache behavior. PyArrow/Fsspec should ask the provider for credential properties for the actual file location, merge those into local backend construction properties, and invalidate/rebuild cached filesystems when returned credentials change. - Cloud-specific helpers: own property validation/mapping. S3 expiry fields and required key/token checks belong in an S3 helper/provider path, not in generic catalog logic. What do you think about this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
