roryqi commented on code in PR #10732:
URL: https://github.com/apache/gravitino/pull/10732#discussion_r3077626222
##########
design-docs/external_secret_management_design.md:
##########
@@ -0,0 +1,1402 @@
+# External Secret Management Integration for Apache Gravitino
+
+---
+
+
+## Background
+
+Apache Gravitino currently supports credential vending for cloud storage
systems (S3, GCS, ADLS, OSS) through various credential providers. However,
sensitive credentials (access keys, secret keys, passwords, API tokens) are
currently configured directly in configuration files or catalog properties as
plain text, which poses significant security risks in production environments.
+
+Modern cloud-native applications follow the principle of **never storing
secrets in database as plain-text or in configuration files** and instead rely
on external secret management systems that provide:
+- Centralized secret storage with encryption at rest
+- Fine-grained access control and audit logging
+- Automatic secret rotation
+- Secret versioning and rollback capabilities
+- Integration with cloud IAM systems
+
+### Security Risks Today
+
+1. **Plain Text Catalog Credentials in Database:**
+ When users create catalogs via UI/API, passwords are stored as plain text:
+ ```properties
+ # PostgreSQL Catalog
+ jdbc-password = MyDatabasePassword123
+
+ # MySQL Catalog
+ jdbc-password = SuperSecret456
+
+ # Hive Catalog
+ authentication.kerberos.keytab-uri = /path/to/keytab
+ ```
+ These credentials are stored in Gravitino's backend database in plain text
and accessible to anyone with database access.
+
+2. **Lack of Secret Rotation:** When catalog passwords are rotated, users must
manually update the catalog, which exposes the new password again as plain text.
+
+3. **Compliance Requirements:** Many industries (finance, healthcare,
government) mandate external secret management for SOC2, HIPAA, PCI-DSS
compliance.
+
+
+### Industry Best Practices
+
+Many organizations running applications in production already use an external
secret management solution, such as:
+- AWS Secrets Manager
+- HashiCorp Vault
+- Azure Key Vault
+- Google Secret Manager
+- Kubernetes Secrets with external secret operators
+
+Gravitino should integrate seamlessly with these existing secret management
infrastructures and provide an extensible framework to support any other vault
providers.
+
+## Current State
+
+### Understanding CredentialProvider vs SecretProvider
+
+It's important to clarify the relationship between two similar-sounding
concepts:
+
+| Component | Purpose | Layer | Changes Needed |
+|-----------|---------|-------|----------------|
+| **CredentialProvider** (existing) | Vends credentials to clients (Iceberg,
Spark, etc.) for accessing cloud storage | Application layer | None - works
transparently |
+| **SecretProvider** (proposed) | Fetches secrets from external systems (AWS
Secrets Manager, Vault) | Infrastructure layer | New abstraction |
+
+**Key Point:** `SecretProvider` is a **prerequisite layer** that runs
**before** `CredentialProvider`. It resolves secret references
(`${secret:...}`) to actual values, then passes those resolved values to
`CredentialProvider` implementations.
+
+**Example Flow[Postgresql catalog]:**
+```
+User creates PostgreSQL catalog via UI with:
+ jdbc-password = ${secret:prod-postgres-catalog/password}
+ ↓
+Gravitino stores the reference "${secret:prod-postgres-catalog/password}" in
backend DB
+ ↓
+When catalog needs to connect to PostgreSQL:
+ SecretProvider: Calls AWS Secrets Manager API → returns "mySecureP@ssw0rd"
+ ↓
+ Catalog/CredentialProvider: Receives resolved password "mySecureP@ssw0rd"
and connects to PostgreSQL
+ ↓ (no code changes needed!)
+Client: Uses the PostgreSQL catalog to query tables
+```
+
+They work together at different layers, and existing `CredentialProvider`
implementations require **zero code changes**.
+
+### Catalog Credential Flow Today
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ User Creates Catalog via UI/API │
+│ ┌───────────────────────────────────────────────────────┐ │
+│ │ Catalog: "prod-postgres" │ │
+│ │ Provider: "jdbc-postgresql" │ │
+│ │ Properties: │ │
+│ │ jdbc-url = jdbc:postgresql://db.example.com/prod │ │
+│ │ jdbc-user = app_user │ │
+│ │ jdbc-password = PlainTextPassword123 │ │
+│ └───────────────────────────────────────────────────────┘ │
+└─────────────────────────────────────────────────────────────┘
+ │
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ Gravitino Backend Database │
+│ - Stores catalog properties including plain text password │
+│ - Password visible in database and logs │
+└─────────────────────────────────────────────────────────────┘
+ │
+ ▼
+┌─────────────────────────────────────────────────────────────┐
+│ Catalog.initialize(properties) │
+│ - Reads password from properties Map │
+│ - Connects to external database with plain text password │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Proposed Solution
+
+### High-Level Architecture
+
+Introduce a **Secret Provider** abstraction layer that resolves secret
references :
+
+```
+┌──────────────────────────────────────────────────────────────────┐
+│ Gravitino Server Configuration (gravitino.conf) │
+│ ┌────────────────────────────────────────────────────────────┐ │
+│ │ # Global secret provider configuration │ │
+│ │ secret-provider = aws-secrets-manager │ │
+│ │ secret-provider.region = us-east-1 │ │
+│ │ secret-provider.secret-path-prefix = gravitino/prod/ │ │
+│ └────────────────────────────────────────────────────────────┘ │
+└──────────────────────────────────────────────────────────────────┘
+ │
+ ▼
+┌──────────────────────────────────────────────────────────────────┐
+│ SecretProviderFactory │
+│ - Creates appropriate SecretProvider based on configuration │
+└──────────────────────────────────────────────────────────────────┘
+ │
+ ▼
+┌──────────────────────────────────────────────────────────────────┐
+│ SecretProvider Interface │
+│ ┌────────────────────────────────────────────────────────────┐ │
+│ │ + resolveSecret(secretReference): String │ │
+│ │ + resolveSecrets(secretReferences): Map<String, String> │ │
+│ │ + refreshSecrets(): void │ │
+│ └────────────────────────────────────────────────────────────┘ │
+└──────────────────────────────────────────────────────────────────┘
+ │ │ │
+ ▼ ▼ ▼
+┌──────────────┐ ┌──────────────────┐ ┌─────────────────┐
+│ AWS Secrets │ │ HashiCorp │ │ Plain Text │
+│ Manager │ │ Vault │ │ (Backward │
+│ Provider │ │ Provider │ │ Compatibility) │
+└──────────────┘ └──────────────────┘ └─────────────────┘
+ │
+ ▼
+┌──────────────────────────────────────────────────────────────────┐
+│ SecretCache │
+│ - TTL-based caching │
+│ - Thread-safe access │
+└──────────────────────────────────────────────────────────────────┘
+ │
+ ▼
+┌──────────────────────────────────────────────────────────────────┐
+│ CredentialProviders │
+│ - Receives resolved catalog credentials transparently │
+│ - No code changes required │
+└──────────────────────────────────────────────────────────────────┘
+```
+
+### Hybrid Approach: Global vs Explicit Provider (Catalog Properties)
+
+The design supports **two complementary syntax patterns for catalog
properties**:
+
+1. **Global Provider (Recommended for single-tenant):**
+ - Configure one secret provider in `gravitino.conf`
+ - Use simple syntax in catalog properties: `${secret:path/to/secret}`
+ - Simpler to manage, easier to migrate providers
+
+2. **Explicit Provider (Multi-tenant scenarios):**
+ - Configure multiple secret providers in `gravitino.conf`
+ - Use provider-specific syntax in catalog properties:
`${secret:provider-type:path/to/secret}`
+ - Each catalog can use different secret management systems
+ - Enables team-specific or compliance-driven provider selection
+
+**Example (Catalog Property Values):**
+```properties
+# Catalog property using global provider
+jdbc-password = ${secret:postgres/password}
+
+# Catalog property with explicit provider override
+jdbc-password = ${secret:vault:postgres/password}
+```
+
+This hybrid approach provides **simplicity by default** while enabling
**flexibility when needed**.
+
+## Goals (V1 Scope: Catalog Properties Only)
+
+**Primary Focus:** Secret resolution for **catalog create/update/read
operations only**.
+
+1. **Provide pluggable external secret management** through a well-defined
abstraction
+2. **Implement AWS Secrets Manager integration** as the first reference
implementation
+3. **Support secret references in catalog properties** (`jdbc-password`,
`jdbc-user`, Kerberos keytabs, etc.)
+4. **Maintain backward compatibility** with existing plain-text catalog
properties
+5. **Implement API redaction rules** to prevent secret exposure in responses
+6. **Minimize performance overhead** through intelligent caching
+
+## Non-Goals (Deferred to Future Work)
+
+1. **Server configuration secret interpolation** (e.g., `gravitino.conf`,
`gravitino-iceberg-rest-server.conf`) — **V2 feature**
+2. Encrypting Gravitino's internal metadata store (separate concern)
+
+## Design Details
+
+### Core Components
+
+#### 1. SecretProvider Interface
Review Comment:
Maybe we can refer to the Iceberg similar design, you can see
```
/** A minimum client interface to connect to a key management service (KMS).
*/
public interface KeyManagementClient extends Serializable, Closeable {
/**
* Wrap a secret key, using a wrapping/master key which is stored in KMS
and referenced by an ID.
* Wrapping means encryption of the secret key with the master key, and
adding optional
* KMS-specific metadata that allows the KMS to decrypt the secret key in
an unwrapping call.
*
* @param key a secret key being wrapped
* @param wrappingKeyId a key ID that represents a wrapping key stored in
KMS
* @return wrapped key material
*/
ByteBuffer wrapKey(ByteBuffer key, String wrappingKeyId);
/**
* Some KMS systems support generation of secret keys inside the KMS
server.
*
* @return true if KMS server supports key generation and
KeyManagementClient implementation is
* interested to leverage this capability. Otherwise, return false -
Iceberg will then
* generate secret keys locally (using the SecureRandom mechanism) and
call {@link
* #wrapKey(ByteBuffer, String)} to wrap them in KMS.
*/
default boolean supportsKeyGeneration() {
return false;
}
/**
* Generate a new secret key in the KMS server, and wrap it using a
wrapping/master key which is
* stored in KMS and referenced by an ID. This method will be called only
if supportsKeyGeneration
* returns true.
*
* @param wrappingKeyId a key ID that represents a wrapping key stored in
KMS
* @return key in two forms: raw, and wrapped with the given wrappingKeyId
*/
default KeyGenerationResult generateKey(String wrappingKeyId) {
throw new UnsupportedOperationException("Key generation is not supported
in this KmsClient");
}
/**
* Unwrap a secret key, using a wrapping/master key which is stored in KMS
and referenced by an
* ID.
*
* @param wrappedKey wrapped key material (encrypted key and optional KMS
metadata, returned by
* the wrapKey method)
* @param wrappingKeyId a key ID that represents a wrapping key stored in
KMS
* @return raw key bytes
*/
ByteBuffer unwrapKey(ByteBuffer wrappedKey, String wrappingKeyId);
/**
* Initialize the KMS client with given properties.
*
* @param properties kms client properties
*/
void initialize(Map<String, String> properties);
/**
* Close KMS Client to release underlying resources, this could be
triggered in different threads
* when KmsClient is shared by multiple encryption managers.
*/
@Override
default void close() {}
/**
* For KMS systems that support key generation, this class keeps the key
generation result - the
* raw secret key, and its wrap.
*/
class KeyGenerationResult {
private final ByteBuffer key;
private final ByteBuffer wrappedKey;
public KeyGenerationResult(ByteBuffer key, ByteBuffer wrappedKey) {
this.key = key;
this.wrappedKey = wrappedKey;
}
public ByteBuffer key() {
return key;
}
public ByteBuffer wrappedKey() {
return wrappedKey;
}
}
}
```
You can survey about this and realize its design, background and
consideration by finding the discussion and design documents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
