sunyuhan1998 opened a new issue, #10790:
URL: https://github.com/apache/gravitino/issues/10790
### What would you like to be improved?
When credential vending (`enable_credential_vending=True`) is enabled in
the Python SDK's GVFS, every file operation (ls, open, exists, info, etc.)
makes an HTTP request to the Gravitino server's /credentials endpoint to fetch
credentials, even if the previously fetched credentials have not yet expired.
In contrast, the Java SDK implements client-side lazy credential caching
in each CredentialsProvider (e.g., `OSSCredentialsProvider`,
`S3CredentialsProvider`). Credentials are cached locally and only refreshed
when they approach expiration (at 50% of their lifetime), avoiding unnecessary
network round-trips.
### How should we improve?
Add a credential-level cache in `BaseGVFSOperations` that mirrors the Java
SDK's lazy refresh pattern:
- Cache credentials locally with an expiration time calculated using the
existing `credential_expiration_ratio config` (default 0.5).
- On each file operation, return cached credentials if they have not
expired (zero network overhead).
- Only fetch new credentials from the server when the cache misses or
expires.
- Use read-write locks with double-checked locking for thread safety,
consistent with the `existing _get_filesystem()` pattern.
- Use `LRUCache` to bound memory usage, consistent with the `existing
_filesystem_cache`.
This change requires no modifications to the public API or configuration —
all changes are internal to `BaseGVFSOperations`.
I'll open a PR later to implement this feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
