mbutrovich commented on code in PR #3969: URL: https://github.com/apache/iceberg-python/pull/3969#discussion_r4076238656
########## pyiceberg/encryption/stream.py: ########## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Format primitives for the AGS1 stream, used to encrypt manifests and manifest lists. + +An AGS1 stream is an 8 byte header followed by a sequence of AES-GCM blocks:: + + "AGS1" || plain_block_size (4 bytes, little endian) + nonce || ciphertext || tag (block 0, up to PLAIN_BLOCK_SIZE of plaintext) + nonce || ciphertext || tag (block 1..n, the last of which may be shorter) + +Each block authenticates `aad_prefix || block_index` as additional data, so blocks cannot +be reordered or moved between files. Byte-compatible with Java's `AesGcmInputStream` and +`AesGcmOutputStream`, and with iceberg-rust. Review Comment: Only the header is checked against Java's bytes so far. Could the cross-client fixture issue I asked for on #3968 include AGS1 files written by Java's `AesGcmOutputStream`: an empty file, a single partial block, and a block-aligned multi-block file? The issue should also list the close-path tests from apache/iceberg-rust#2286 for the output stream PR, so a block-aligned write doesn't add a trailing empty block. ########## pyiceberg/encryption/stream.py: ########## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Format primitives for the AGS1 stream, used to encrypt manifests and manifest lists. + +An AGS1 stream is an 8 byte header followed by a sequence of AES-GCM blocks:: + + "AGS1" || plain_block_size (4 bytes, little endian) + nonce || ciphertext || tag (block 0, up to PLAIN_BLOCK_SIZE of plaintext) + nonce || ciphertext || tag (block 1..n, the last of which may be shorter) + +Each block authenticates `aad_prefix || block_index` as additional data, so blocks cannot +be reordered or moved between files. Byte-compatible with Java's `AesGcmInputStream` and +`AesGcmOutputStream`, and with iceberg-rust. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from pyiceberg.encryption.ciphers import AesGcmCipher + +GCM_STREAM_MAGIC = b"AGS1" +PLAIN_BLOCK_SIZE = 1024 * 1024 +GCM_STREAM_HEADER_LENGTH = len(GCM_STREAM_MAGIC) + 4 +BLOCK_OVERHEAD = AesGcmCipher.NONCE_LENGTH + AesGcmCipher.TAG_LENGTH +CIPHER_BLOCK_SIZE = PLAIN_BLOCK_SIZE + BLOCK_OVERHEAD +BLOCK_INDEX_LENGTH = 4 +MAX_BLOCKS = 2 ** (8 * BLOCK_INDEX_LENGTH) - 1 + + +def stream_block_aad(aad_prefix: bytes | None, block_index: int) -> bytes: + """Return the additional authenticated data for the block at `block_index`. + + Args: + aad_prefix (bytes | None): The file's AAD prefix, from its key metadata. + block_index (int): The zero-based index of the block within the stream. + """ + return (aad_prefix or b"") + block_index.to_bytes(BLOCK_INDEX_LENGTH, "little") + + +def encode_stream_header() -> bytes: + """Encode the AGS1 header that precedes the first block.""" + return GCM_STREAM_MAGIC + PLAIN_BLOCK_SIZE.to_bytes(4, "little") + + +def decode_stream_header(header: bytes) -> int: + """Decode an AGS1 header, returning the plaintext block size it declares. + + Args: + header (bytes): At least `GCM_STREAM_HEADER_LENGTH` bytes from the start of the stream. + """ + if len(header) < GCM_STREAM_HEADER_LENGTH: + raise ValueError(f"Invalid AGS1 header: expected {GCM_STREAM_HEADER_LENGTH} bytes, got {len(header)}") + + if (magic := header[: len(GCM_STREAM_MAGIC)]) != GCM_STREAM_MAGIC: + raise ValueError(f"Invalid AGS1 header: magic {magic!r} does not match {GCM_STREAM_MAGIC!r}") + + plain_block_size = int.from_bytes(header[len(GCM_STREAM_MAGIC) : GCM_STREAM_HEADER_LENGTH], "little") + if plain_block_size != PLAIN_BLOCK_SIZE: + raise ValueError(f"Unsupported AGS1 block size: {plain_block_size} (expected {PLAIN_BLOCK_SIZE})") + + return plain_block_size + + +def calculate_plaintext_length(encrypted_length: int) -> int: + """Return the plaintext length of an AGS1 stream that occupies `encrypted_length` bytes.""" + if encrypted_length < GCM_STREAM_HEADER_LENGTH: + raise ValueError(f"Invalid AGS1 stream: expected at least {GCM_STREAM_HEADER_LENGTH} bytes, got {encrypted_length}") + + stream_length = encrypted_length - GCM_STREAM_HEADER_LENGTH + if stream_length == 0: + return 0 + + full_blocks, cipher_bytes_in_last_block = divmod(stream_length, CIPHER_BLOCK_SIZE) + if cipher_bytes_in_last_block == 0: + return full_blocks * PLAIN_BLOCK_SIZE + + if cipher_bytes_in_last_block < BLOCK_OVERHEAD: + raise ValueError( + f"Truncated AGS1 stream: last block is {cipher_bytes_in_last_block} bytes, expected at least {BLOCK_OVERHEAD}" + ) + + return full_blocks * PLAIN_BLOCK_SIZE + cipher_bytes_in_last_block - BLOCK_OVERHEAD + + +@dataclass(frozen=True) +class Ags1Layout: + """Where each block of an AGS1 stream sits, derived from the encrypted file length. + + Only the final block may hold less than `PLAIN_BLOCK_SIZE` of plaintext, so the layout + follows from the encrypted length alone, without reading the stream. + """ + Review Comment: Could the docstring say that `encrypted_length` must be the trusted length from `StandardKeyMetadata.file_length` ([key_metadata.py#L52](https://github.com/apache/iceberg-python/blob/0d584073f67948c46f34d4c2a1672455bc10b5c9/pyiceberg/encryption/key_metadata.py#L52)), never a file system stat? The spec's [File length](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/format/gcm-stream-spec.md#file-length) section requires this. Otherwise an attacker can drop whole trailing blocks, and every remaining block still authenticates. Java deprecated the [`AesGcmInputFile` constructor without a length](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java#L32-L38) because it's "not safe". apache/iceberg-rust#3236 now makes a missing `file_length` a hard error on read, because Java can't read files written without it. Please also file an issue under #3222 so the read er PR fails when `file_length` is `None`, and link it here. ########## pyiceberg/encryption/stream.py: ########## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Format primitives for the AGS1 stream, used to encrypt manifests and manifest lists. + +An AGS1 stream is an 8 byte header followed by a sequence of AES-GCM blocks:: + + "AGS1" || plain_block_size (4 bytes, little endian) + nonce || ciphertext || tag (block 0, up to PLAIN_BLOCK_SIZE of plaintext) + nonce || ciphertext || tag (block 1..n, the last of which may be shorter) + +Each block authenticates `aad_prefix || block_index` as additional data, so blocks cannot +be reordered or moved between files. Byte-compatible with Java's `AesGcmInputStream` and +`AesGcmOutputStream`, and with iceberg-rust. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from pyiceberg.encryption.ciphers import AesGcmCipher + +GCM_STREAM_MAGIC = b"AGS1" +PLAIN_BLOCK_SIZE = 1024 * 1024 +GCM_STREAM_HEADER_LENGTH = len(GCM_STREAM_MAGIC) + 4 +BLOCK_OVERHEAD = AesGcmCipher.NONCE_LENGTH + AesGcmCipher.TAG_LENGTH +CIPHER_BLOCK_SIZE = PLAIN_BLOCK_SIZE + BLOCK_OVERHEAD +BLOCK_INDEX_LENGTH = 4 +MAX_BLOCKS = 2 ** (8 * BLOCK_INDEX_LENGTH) - 1 + + +def stream_block_aad(aad_prefix: bytes | None, block_index: int) -> bytes: + """Return the additional authenticated data for the block at `block_index`. + + Args: + aad_prefix (bytes | None): The file's AAD prefix, from its key metadata. + block_index (int): The zero-based index of the block within the stream. + """ + return (aad_prefix or b"") + block_index.to_bytes(BLOCK_INDEX_LENGTH, "little") + + +def encode_stream_header() -> bytes: + """Encode the AGS1 header that precedes the first block.""" + return GCM_STREAM_MAGIC + PLAIN_BLOCK_SIZE.to_bytes(4, "little") + + +def decode_stream_header(header: bytes) -> int: + """Decode an AGS1 header, returning the plaintext block size it declares. + + Args: + header (bytes): At least `GCM_STREAM_HEADER_LENGTH` bytes from the start of the stream. + """ + if len(header) < GCM_STREAM_HEADER_LENGTH: + raise ValueError(f"Invalid AGS1 header: expected {GCM_STREAM_HEADER_LENGTH} bytes, got {len(header)}") + + if (magic := header[: len(GCM_STREAM_MAGIC)]) != GCM_STREAM_MAGIC: + raise ValueError(f"Invalid AGS1 header: magic {magic!r} does not match {GCM_STREAM_MAGIC!r}") + + plain_block_size = int.from_bytes(header[len(GCM_STREAM_MAGIC) : GCM_STREAM_HEADER_LENGTH], "little") + if plain_block_size != PLAIN_BLOCK_SIZE: + raise ValueError(f"Unsupported AGS1 block size: {plain_block_size} (expected {PLAIN_BLOCK_SIZE})") + + return plain_block_size + + +def calculate_plaintext_length(encrypted_length: int) -> int: + """Return the plaintext length of an AGS1 stream that occupies `encrypted_length` bytes.""" + if encrypted_length < GCM_STREAM_HEADER_LENGTH: + raise ValueError(f"Invalid AGS1 stream: expected at least {GCM_STREAM_HEADER_LENGTH} bytes, got {encrypted_length}") + + stream_length = encrypted_length - GCM_STREAM_HEADER_LENGTH + if stream_length == 0: + return 0 + + full_blocks, cipher_bytes_in_last_block = divmod(stream_length, CIPHER_BLOCK_SIZE) + if cipher_bytes_in_last_block == 0: + return full_blocks * PLAIN_BLOCK_SIZE + + if cipher_bytes_in_last_block < BLOCK_OVERHEAD: + raise ValueError( + f"Truncated AGS1 stream: last block is {cipher_bytes_in_last_block} bytes, expected at least {BLOCK_OVERHEAD}" + ) + + return full_blocks * PLAIN_BLOCK_SIZE + cipher_bytes_in_last_block - BLOCK_OVERHEAD + + +@dataclass(frozen=True) +class Ags1Layout: + """Where each block of an AGS1 stream sits, derived from the encrypted file length. + + Only the final block may hold less than `PLAIN_BLOCK_SIZE` of plaintext, so the layout + follows from the encrypted length alone, without reading the stream. + """ + + plaintext_length: int + num_blocks: int + last_cipher_block_size: int + + @classmethod + def from_encrypted_length(cls, encrypted_length: int) -> Ags1Layout: + """Derive the layout of an AGS1 stream that occupies `encrypted_length` bytes.""" + plaintext_length = calculate_plaintext_length(encrypted_length) + stream_length = encrypted_length - GCM_STREAM_HEADER_LENGTH + if stream_length == 0: + return cls(plaintext_length=0, num_blocks=0, last_cipher_block_size=0) Review Comment: Accepting a header-only stream here follows the spec, which says [the last block has a non-zero length](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/format/gcm-stream-spec.md#cipher-block-structure), so an empty plaintext has no blocks. Java disagrees in both directions. `AesGcmOutputStream` encrypts one empty block on close for an empty file (the [`currentBlockIndex != 0` guard](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/core/src/main/java/org/apache/iceberg/encryption/AesGcmOutputStream.java#L150-L152) only skips the trailing block once a block exists). `AesGcmInputFile` rejects anything shorter than [`MIN_STREAM_LENGTH`](https://github.com/apache/iceberg/blob/1504ddd5d5119e934e7cbcaed82b9384caf35b82/core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java#L68-L74), the header plus one empty block. Accepting both forms on read seems right to me. On write, a PyIceberg writer that follows the spec for an empty file would produce 8 bytes that Java refuses to open. Could you open an issue on apache/iceberg about the discrepancy and link it here, so the output stream PR has a settled answer on which form to write? ########## tests/encryption/test_stream.py: ########## @@ -0,0 +1,232 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +import pytest + +from pyiceberg.encryption.ciphers import AesGcmCipher, SecureKey +from pyiceberg.encryption.stream import ( + BLOCK_OVERHEAD, + CIPHER_BLOCK_SIZE, + GCM_STREAM_HEADER_LENGTH, + GCM_STREAM_MAGIC, + MAX_BLOCKS, + PLAIN_BLOCK_SIZE, + Ags1Layout, + calculate_plaintext_length, + decode_stream_header, + encode_stream_header, + stream_block_aad, +) + +KEY = SecureKey(b"0123456789012345") +AAD_PREFIX = b"0123456789abcdef" + +# The header a Java `AesGcmOutputStream` writes: "AGS1" then 1 MiB as a little-endian int32. +JAVA_HEADER = b"AGS1\x00\x00\x10\x00" + + +def build_stream(plaintext: bytes, aad_prefix: bytes | None = AAD_PREFIX) -> bytes: + """Encrypt `plaintext` into an AGS1 stream, as an output stream implementation would.""" + blocks = [ + AesGcmCipher(KEY).encrypt(plaintext[start : start + PLAIN_BLOCK_SIZE], stream_block_aad(aad_prefix, index)) + for index, start in enumerate(range(0, len(plaintext), PLAIN_BLOCK_SIZE)) + ] + return encode_stream_header() + b"".join(blocks) + + +def test_format_constants() -> None: + assert GCM_STREAM_MAGIC == b"AGS1" + assert PLAIN_BLOCK_SIZE == 1024 * 1024 + assert GCM_STREAM_HEADER_LENGTH == 8 + assert BLOCK_OVERHEAD == 28 + assert CIPHER_BLOCK_SIZE == PLAIN_BLOCK_SIZE + BLOCK_OVERHEAD + assert MAX_BLOCKS == 2**32 - 1 + + +def test_encode_stream_header_matches_java() -> None: + assert encode_stream_header() == JAVA_HEADER + + +def test_decode_stream_header() -> None: + assert decode_stream_header(JAVA_HEADER) == PLAIN_BLOCK_SIZE + assert decode_stream_header(encode_stream_header()) == PLAIN_BLOCK_SIZE + + +def test_decode_stream_header_ignores_trailing_block_bytes() -> None: + assert decode_stream_header(JAVA_HEADER + b"block bytes") == PLAIN_BLOCK_SIZE + + [email protected]("length", [0, 4, 7]) +def test_decode_stream_header_rejects_a_short_header(length: int) -> None: + with pytest.raises(ValueError, match=f"Invalid AGS1 header: expected 8 bytes, got {length}"): + decode_stream_header(bytes(length)) + + +def test_decode_stream_header_rejects_the_wrong_magic() -> None: + with pytest.raises(ValueError, match="magic b'AGS2' does not match b'AGS1'"): + decode_stream_header(b"AGS2\x00\x00\x10\x00") + + +def test_decode_stream_header_rejects_an_unsupported_block_size() -> None: + with pytest.raises(ValueError, match=f"Unsupported AGS1 block size: 512 \\(expected {PLAIN_BLOCK_SIZE}\\)"): + decode_stream_header(GCM_STREAM_MAGIC + (512).to_bytes(4, "little")) + + [email protected]( + "block_index, expected", + [(0, b"\x00\x00\x00\x00"), (1, b"\x01\x00\x00\x00"), (258, b"\x02\x01\x00\x00"), (MAX_BLOCKS, b"\xff\xff\xff\xff")], +) +def test_stream_block_aad_encodes_the_index_little_endian(block_index: int, expected: bytes) -> None: + assert stream_block_aad(None, block_index) == expected + assert stream_block_aad(b"", block_index) == expected + assert stream_block_aad(AAD_PREFIX, block_index) == AAD_PREFIX + expected + + [email protected]( + "encrypted_length, expected", + [ + (GCM_STREAM_HEADER_LENGTH, 0), + (GCM_STREAM_HEADER_LENGTH + BLOCK_OVERHEAD, 0), + (GCM_STREAM_HEADER_LENGTH + BLOCK_OVERHEAD + 100, 100), + (GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE, PLAIN_BLOCK_SIZE), + (GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE + BLOCK_OVERHEAD + 5, PLAIN_BLOCK_SIZE + 5), + (GCM_STREAM_HEADER_LENGTH + 2 * CIPHER_BLOCK_SIZE, 2 * PLAIN_BLOCK_SIZE), + ], +) +def test_calculate_plaintext_length(encrypted_length: int, expected: int) -> None: + assert calculate_plaintext_length(encrypted_length) == expected + + [email protected]("encrypted_length", [0, 1, 7]) +def test_calculate_plaintext_length_rejects_a_stream_shorter_than_the_header(encrypted_length: int) -> None: + with pytest.raises(ValueError, match=f"expected at least 8 bytes, got {encrypted_length}"): + calculate_plaintext_length(encrypted_length) + + [email protected]("last_block_size", [1, 27]) +def test_calculate_plaintext_length_rejects_a_truncated_last_block(last_block_size: int) -> None: + with pytest.raises(ValueError, match=f"last block is {last_block_size} bytes, expected at least 28"): + calculate_plaintext_length(GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE + last_block_size) + + [email protected]( + "encrypted_length, plaintext_length, num_blocks, last_cipher_block_size", + [ + (GCM_STREAM_HEADER_LENGTH, 0, 0, 0), + (GCM_STREAM_HEADER_LENGTH + BLOCK_OVERHEAD, 0, 1, BLOCK_OVERHEAD), + (GCM_STREAM_HEADER_LENGTH + BLOCK_OVERHEAD + 100, 100, 1, BLOCK_OVERHEAD + 100), + (GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE, PLAIN_BLOCK_SIZE, 1, CIPHER_BLOCK_SIZE), + (GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE + BLOCK_OVERHEAD + 5, PLAIN_BLOCK_SIZE + 5, 2, BLOCK_OVERHEAD + 5), + (GCM_STREAM_HEADER_LENGTH + 2 * CIPHER_BLOCK_SIZE, 2 * PLAIN_BLOCK_SIZE, 2, CIPHER_BLOCK_SIZE), + ], +) +def test_layout_from_encrypted_length( + encrypted_length: int, plaintext_length: int, num_blocks: int, last_cipher_block_size: int +) -> None: + layout = Ags1Layout.from_encrypted_length(encrypted_length) + + assert layout == Ags1Layout( + plaintext_length=plaintext_length, num_blocks=num_blocks, last_cipher_block_size=last_cipher_block_size + ) + + +def test_layout_rejects_more_blocks_than_the_index_can_address() -> None: + encrypted_length = GCM_STREAM_HEADER_LENGTH + (MAX_BLOCKS + 1) * CIPHER_BLOCK_SIZE + + with pytest.raises(ValueError, match=f"AGS1 streams hold at most {MAX_BLOCKS} blocks"): + Ags1Layout.from_encrypted_length(encrypted_length) + + +def test_layout_block_sizes_and_offsets() -> None: + layout = Ags1Layout.from_encrypted_length(GCM_STREAM_HEADER_LENGTH + 2 * CIPHER_BLOCK_SIZE + BLOCK_OVERHEAD + 7) + + assert layout.num_blocks == 3 + assert layout.cipher_block_size(0) == layout.cipher_block_size(1) == CIPHER_BLOCK_SIZE + assert layout.plain_block_size(0) == layout.plain_block_size(1) == PLAIN_BLOCK_SIZE + assert layout.cipher_block_size(2) == BLOCK_OVERHEAD + 7 + assert layout.plain_block_size(2) == 7 + assert layout.encrypted_block_offset(0) == GCM_STREAM_HEADER_LENGTH + assert layout.encrypted_block_offset(1) == GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE + assert layout.encrypted_block_offset(2) == GCM_STREAM_HEADER_LENGTH + 2 * CIPHER_BLOCK_SIZE + + [email protected]("block_index", [-1, 1, 2]) +def test_layout_rejects_an_out_of_range_block_index(block_index: int) -> None: + layout = Ags1Layout.from_encrypted_length(GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE) + + with pytest.raises(ValueError, match=f"Block index out of range: {block_index} \\(stream holds 1 blocks\\)"): + layout.cipher_block_size(block_index) + + with pytest.raises(ValueError, match=f"Block index out of range: {block_index}"): + layout.encrypted_block_offset(block_index) + + [email protected]( + "plaintext_offset, expected", + [(0, 0), (1, 0), (PLAIN_BLOCK_SIZE - 1, 0), (PLAIN_BLOCK_SIZE, 1), (PLAIN_BLOCK_SIZE + 6, 1)], +) +def test_layout_block_index_for_plaintext_offset(plaintext_offset: int, expected: int) -> None: + layout = Ags1Layout.from_encrypted_length(GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE + BLOCK_OVERHEAD + 7) + + assert layout.block_index_for(plaintext_offset) == expected + + [email protected]("plaintext_offset", [-1, PLAIN_BLOCK_SIZE]) +def test_layout_rejects_an_out_of_range_plaintext_offset(plaintext_offset: int) -> None: + layout = Ags1Layout.from_encrypted_length(GCM_STREAM_HEADER_LENGTH + CIPHER_BLOCK_SIZE) + + with pytest.raises(ValueError, match=f"Plaintext offset out of range: {plaintext_offset}"): + layout.block_index_for(plaintext_offset) + + [email protected]("plaintext_length", [1, 100, PLAIN_BLOCK_SIZE, PLAIN_BLOCK_SIZE + 7, 2 * PLAIN_BLOCK_SIZE]) Review Comment: Could `0` be added to these cases? `build_stream(b"")` produces the header-only stream the spec describes, and nothing round-trips an empty plaintext yet. Java's empty-file form (header plus one empty block) is already covered as a layout case, but not decrypted, so a round-trip test that builds that form with one empty block would pin it too. ########## pyiceberg/encryption/stream.py: ########## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Format primitives for the AGS1 stream, used to encrypt manifests and manifest lists. + +An AGS1 stream is an 8 byte header followed by a sequence of AES-GCM blocks:: + + "AGS1" || plain_block_size (4 bytes, little endian) + nonce || ciphertext || tag (block 0, up to PLAIN_BLOCK_SIZE of plaintext) + nonce || ciphertext || tag (block 1..n, the last of which may be shorter) + +Each block authenticates `aad_prefix || block_index` as additional data, so blocks cannot +be reordered or moved between files. Byte-compatible with Java's `AesGcmInputStream` and +`AesGcmOutputStream`, and with iceberg-rust. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from pyiceberg.encryption.ciphers import AesGcmCipher + +GCM_STREAM_MAGIC = b"AGS1" +PLAIN_BLOCK_SIZE = 1024 * 1024 +GCM_STREAM_HEADER_LENGTH = len(GCM_STREAM_MAGIC) + 4 +BLOCK_OVERHEAD = AesGcmCipher.NONCE_LENGTH + AesGcmCipher.TAG_LENGTH +CIPHER_BLOCK_SIZE = PLAIN_BLOCK_SIZE + BLOCK_OVERHEAD +BLOCK_INDEX_LENGTH = 4 +MAX_BLOCKS = 2 ** (8 * BLOCK_INDEX_LENGTH) - 1 + + +def stream_block_aad(aad_prefix: bytes | None, block_index: int) -> bytes: Review Comment: This PR has the format primitives but no encrypting or decrypting stream yet, so `stream_block_aad`, `calculate_plaintext_length`, and `Ags1Layout` become public API with no consumer. iceberg-rust keeps `stream_block_aad` `pub(crate)`. This is the same question as `MemoryKeyManagementClient` on #3968. Could you prefix these with `_` until the reader and writer land, or say in the PR description which upcoming PR consumes them and why they need to be public? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
