mengxianwen1 opened a new issue, #10976:
URL: https://github.com/apache/gravitino/issues/10976
### What would you like to be improved?
When creating a Lance table, object storage credentials (_lance.storage.*_
properties such as AK/SK, endpoint, region) must be explicitly provided at the
table level every time. There is no inheritance mechanism for these properties
from the
catalog or schema level.
Currently, only the _location_ property has a fallback chain: table
properties → schema _location_ → catalog _location_ (implemented in
`GenericCatalogOperations.calculateTableLocation()`). However,
_lance.storage.*_ properties are defined only at the
table level via `LanceTableDelegator.tablePropertyEntries()` and
`LancePropertiesUtils.getLanceStorageOptions()` only extracts them from table
properties.
**Key observations:**
1. `GenericCatalogPropertiesMetadata` only defines
`Catalog.PROPERTY_LOCATION `— no _lance.storage.*_ entries exist at the catalog
level.
2. `GenericSchemaPropertiesMetadata` only defines
`Schema.PROPERTY_LOCATION` — no storage credential entries at the schema level
either.
3. `GenericCatalogOperations.createTable() `merges _location_ and
_table-format_ into table properties, but never merges any storage credentials
from catalog or schema.
4. `LanceTableOperations.createTableInternal()` calls
`LancePropertiesUtils.getLanceStorageOptions`(properties), which only filters
the _lance.storage._ prefix from the passed-in table properties — if no storage
credentials were merged upstream,
they are simply absent.
This means users must redundantly specify the same AK/SK and endpoint on
every createTable call, even though these values are typically shared across
all tables under the same catalog or schema.
### How should we improve?
Introduce a storage property inheritance chain similar to the existing
location fallback:
1. **Define** _lance.storage.*_ **as a prefix property at catalog and
schema levels** — add
`stringOptionalPropertyPrefixEntry(LANCE_STORAGE_OPTIONS_PREFIX, ...)` to
`GenericCatalogPropertiesMetadata` and `GenericSchemaPropertiesMetadata`.
2. **Implement a** `calculateStorageOptions()` **method in**
`GenericCatalogOperations` (or similar location) that merges storage properties
with a fallback chain: table properties → schema _lance.storage.*_ → catalog
_lance.storage.*_. Table-level
properties should take precedence (override) over schema and catalog
level.
3. **Merge the resolved storage options into table properties before
delegating to** `LanceTableOperations.createTable()`, so that
`LancePropertiesUtils.getLanceStorageOptions()` can pick them up as it does
today without changes.
4. **Consider security implications** — catalog-level storage credentials
are stored in Gravitino metadata. Evaluate whether these should be marked as
hidden to avoid exposing them in API responses, similar to how other catalogs
handle sensitive
properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
