mbutrovich commented on code in PR #3968: URL: https://github.com/apache/iceberg-python/pull/3968#discussion_r4076170708
########## pyiceberg/encryption/kms.py: ########## @@ -0,0 +1,114 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Key management client interface for table encryption, and an in-memory implementation.""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from dataclasses import dataclass, field + +from pyiceberg.encryption.ciphers import AesGcmCipher, AesKeySize, SecureKey + + +@dataclass(frozen=True) +class GeneratedKey: + """A newly generated key, both in the clear and wrapped by the key management service.""" + + key: bytes = field(repr=False) + wrapped_key: bytes + + +class KeyManagementClient(ABC): + """A base class for key management service implementations. + + Wraps and unwraps table encryption keys using master keys that the service holds. + """ Review Comment: How will a client be constructed once the catalog wiring lands? Java's `KeyManagementClient` has [`initialize(Map<String, String> properties)`](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/core/src/main/java/org/apache/iceberg/encryption/KeyManagementClient.java#L80). [`EncryptionUtil.createKmsClient`](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java#L71-L96) builds the class named by `encryption.kms-impl` with a no-arg constructor and then calls `initialize` with the catalog properties. This ABC has no equivalent yet, so a custom client has no construction contract to code against. PyIceberg already has a pattern for classes loaded by name. [`FileIO.__init__`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/io/__init__.py#L277) takes `properties`, and [`_import_file_io`](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/io/__init__.py#L335-L343) calls `class_(properties)`. Could `KeyManagementClient` define `__init__(self, properties: Properties = EMPTY_DICT)` the same way? If the loader starts calling `cls(properties)` in a later PR, any subclass written against this version with a different constructor will fail to load. `MemoryKeyManagementClient` would need `master_key_size` to come from a property in that model. If you'd rather mirror iceberg-rust, its [`KmsClientFactory`](https://github.com/apache/iceberg-rust/blob/bb1e4a4861f02377489eff818b75138f414c4cb0/crates/iceberg/src/encryption/kms/factory.rs#L41-L50) solves the same problem, but either way I'd like t he contract in this PR. ########## pyiceberg/encryption/kms.py: ########## @@ -0,0 +1,114 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Key management client interface for table encryption, and an in-memory implementation.""" + +from __future__ import annotations + +from abc import ABC, abstractmethod +from dataclasses import dataclass, field + +from pyiceberg.encryption.ciphers import AesGcmCipher, AesKeySize, SecureKey + + +@dataclass(frozen=True) +class GeneratedKey: + """A newly generated key, both in the clear and wrapped by the key management service.""" + + key: bytes = field(repr=False) + wrapped_key: bytes + + +class KeyManagementClient(ABC): + """A base class for key management service implementations. + + Wraps and unwraps table encryption keys using master keys that the service holds. + """ + + @abstractmethod + def wrap_key(self, key: bytes, wrapping_key_id: str) -> bytes: + """Wrap a key using the master key identified by `wrapping_key_id`. + + Args: + key (bytes): The key to wrap. + wrapping_key_id (str): Identifies the master key held by the service. + """ + + @abstractmethod + def unwrap_key(self, wrapped_key: bytes, wrapping_key_id: str) -> bytes: + """Unwrap a key using the master key identified by `wrapping_key_id`. + + Args: + wrapped_key (bytes): The wrapped key, as returned by `wrap_key`. + wrapping_key_id (str): Identifies the master key held by the service. + """ + + def supports_key_generation(self) -> bool: + """Whether the service generates keys itself, rather than only wrapping them.""" + return False + + def generate_key(self, wrapping_key_id: str) -> GeneratedKey: + """Generate a new key, wrapped by the master key identified by `wrapping_key_id`. + + Args: + wrapping_key_id (str): Identifies the master key held by the service. + """ + raise NotImplementedError(f"{type(self).__name__} does not support key generation") + + +class MemoryKeyManagementClient(KeyManagementClient): + """A key management service that holds its master keys in memory, for testing and demonstration. + + Master keys live only in this process, with no durability or access control, so this is + not for production use. Mirrors Java's `MemoryMockKMS` and iceberg-rust's + `MemoryKeyManagementClient`. Review Comment: Does `MemoryKeyManagementClient` need to ship in the `pyiceberg` package? Java keeps [`MemoryMockKMS`](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/core/src/test/java/org/apache/iceberg/encryption/MemoryMockKMS.java#L25) under `core/src/test`, so it's not public API there. iceberg-rust does export its version. Once this is released, removing it needs a [`@deprecated` cycle](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/AGENTS.md#L108). If PyIceberg's own tests are the only consumers, moving it under `tests/` keeps that option open. A `UnitestKMS`-style subclass with Java's fixed keys could live next to it for the cross-client tests. If you expect users to run it for demos, could you say so in the PR description? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
