cvr-aditya opened a new pull request, #18220: URL: https://github.com/apache/iceberg/pull/18220
Fix for [#18205](https://github.com/apache/iceberg/issues/18205) ### Summary Adds Iceberg table encryption support to `GlueCatalog`. Previously encryption was only implemented for the Hive and REST catalogs — creating a table with `encryption.key-id` against Glue silently produced unencrypted files. Glue now follows the same model as `HiveCatalog`: a `KeyManagementClient` is built from the catalog properties and the per-table encryption parameters are persisted as Glue table parameters. ### Changes **AWS** - `GlueCatalog` creates a `KeyManagementClient` via `EncryptionUtil.createKmsClient()` when `encryption.kms-type` or `encryption.kms-impl` is set, registers it with the catalog's `CloseableGroup`, and passes it to each `GlueTableOperations`. - `GlueTableOperations`: - `io()` returns an `EncryptingFileIO` once a table key is known; `encryption()` builds a `StandardEncryptionManager`, or returns the plaintext manager for unencrypted tables. - `doRefresh()` reads `encryption.key-id`, `encryption.data-key-length` and `metadata_hash` from the **Glue table parameters**, then verifies the metadata file against the recorded hash. - `doCommit()` / `prepareProperties()` write those three parameters back on every commit, with a hash of the metadata being committed. - Rejects removing or changing the key ID of an existing table. - Fails fast with an actionable message when a table key is set but no KMS client is configured, instead of failing partway through a commit. - Overrides `temp()` so writes against uncommitted metadata (CTAS) pick up the table key. **Core** - Moved the table-metadata hashing helpers out of `HMSTablePropertyHelper` into `EncryptionUtil` as `metadataHash(TableMetadata)` and `verifyMetadataHash(TableMetadata, String)`, so Glue and Hive share one implementation rather than duplicating a security checksum across modules. **Hive** - `HMSTablePropertyHelper.setMetadataHash` / `verifyMetadataHash` now delegate to `EncryptionUtil`. No behavior change; one error message no longer says "HMS". ### Why the key ID is read from Glue and not from metadata.json This satisfies the catalog security requirements documented in `encryption.md`: 1. The key ID must not be modifiable or removable for the lifetime of the table — enforced in `doCommit()`. 2. The catalog must not trust encryption parameters taken from storage. An attacker with write access to the object store could strip `encryption.key-id` from metadata.json (along with existing snapshots) and cause writers to emit plaintext. The key ID therefore comes from the Glue catalog, and the metadata file is checksummed against a hash held in Glue. ### Testing New `TestGlueTableEncryption` (mocked Glue client backed by a stateful fake table, `InMemoryFileIO`, `UnitestKMS`): - encryption parameters and metadata hash are persisted to Glue on create - unencrypted tables get no encryption parameters and a `PlaintextEncryptionManager` - a reloaded encrypted table gets a `StandardEncryptionManager` - rewriting the metadata JSON in storage is detected on load - key ID cannot be removed or modified - a missing KMS client produces a clear configuration error Existing `TestHiveCatalog#testMetadataHashing` covers the Hive delegation. ### Docs `docs/docs/encryption.md` now lists Glue alongside Hive and REST. ### AI Disclosure Model: Opus 4.8 Platform/Tool: Claude Code Human Oversight: Partially reviewed Prompt Summary: Update Glue Catalog and TableOperations classes to support table encryption -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
