mbutrovich commented on code in PR #3969: URL: https://github.com/apache/iceberg-python/pull/3969#discussion_r4085176867
########## pyiceberg/encryption/stream.py: ########## @@ -0,0 +1,153 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +"""Format primitives for the AGS1 stream, used to encrypt manifests and manifest lists. + +An AGS1 stream is an 8 byte header followed by a sequence of AES-GCM blocks:: + + "AGS1" || plain_block_size (4 bytes, little endian) + nonce || ciphertext || tag (block 0, up to PLAIN_BLOCK_SIZE of plaintext) + nonce || ciphertext || tag (block 1..n, the last of which may be shorter) + +Each block authenticates `aad_prefix || block_index` as additional data, so blocks cannot +be reordered or moved between files. Byte-compatible with Java's `AesGcmInputStream` and +`AesGcmOutputStream`, and with iceberg-rust. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +from pyiceberg.encryption.ciphers import AesGcmCipher + +GCM_STREAM_MAGIC = b"AGS1" +PLAIN_BLOCK_SIZE = 1024 * 1024 +GCM_STREAM_HEADER_LENGTH = len(GCM_STREAM_MAGIC) + 4 +BLOCK_OVERHEAD = AesGcmCipher.NONCE_LENGTH + AesGcmCipher.TAG_LENGTH +CIPHER_BLOCK_SIZE = PLAIN_BLOCK_SIZE + BLOCK_OVERHEAD +BLOCK_INDEX_LENGTH = 4 +MAX_BLOCKS = 2 ** (8 * BLOCK_INDEX_LENGTH) - 1 + + +def stream_block_aad(aad_prefix: bytes | None, block_index: int) -> bytes: + """Return the additional authenticated data for the block at `block_index`. + + Args: + aad_prefix (bytes | None): The file's AAD prefix, from its key metadata. + block_index (int): The zero-based index of the block within the stream. + """ + return (aad_prefix or b"") + block_index.to_bytes(BLOCK_INDEX_LENGTH, "little") + + +def encode_stream_header() -> bytes: + """Encode the AGS1 header that precedes the first block.""" + return GCM_STREAM_MAGIC + PLAIN_BLOCK_SIZE.to_bytes(4, "little") + + +def decode_stream_header(header: bytes) -> int: + """Decode an AGS1 header, returning the plaintext block size it declares. + + Args: + header (bytes): At least `GCM_STREAM_HEADER_LENGTH` bytes from the start of the stream. + """ + if len(header) < GCM_STREAM_HEADER_LENGTH: + raise ValueError(f"Invalid AGS1 header: expected {GCM_STREAM_HEADER_LENGTH} bytes, got {len(header)}") + + if (magic := header[: len(GCM_STREAM_MAGIC)]) != GCM_STREAM_MAGIC: + raise ValueError(f"Invalid AGS1 header: magic {magic!r} does not match {GCM_STREAM_MAGIC!r}") + + plain_block_size = int.from_bytes(header[len(GCM_STREAM_MAGIC) : GCM_STREAM_HEADER_LENGTH], "little") + if plain_block_size != PLAIN_BLOCK_SIZE: + raise ValueError(f"Unsupported AGS1 block size: {plain_block_size} (expected {PLAIN_BLOCK_SIZE})") + + return plain_block_size + + +def calculate_plaintext_length(encrypted_length: int) -> int: + """Return the plaintext length of an AGS1 stream that occupies `encrypted_length` bytes.""" + if encrypted_length < GCM_STREAM_HEADER_LENGTH: + raise ValueError(f"Invalid AGS1 stream: expected at least {GCM_STREAM_HEADER_LENGTH} bytes, got {encrypted_length}") + + stream_length = encrypted_length - GCM_STREAM_HEADER_LENGTH + if stream_length == 0: + return 0 + + full_blocks, cipher_bytes_in_last_block = divmod(stream_length, CIPHER_BLOCK_SIZE) + if cipher_bytes_in_last_block == 0: + return full_blocks * PLAIN_BLOCK_SIZE + + if cipher_bytes_in_last_block < BLOCK_OVERHEAD: + raise ValueError( + f"Truncated AGS1 stream: last block is {cipher_bytes_in_last_block} bytes, expected at least {BLOCK_OVERHEAD}" + ) + + return full_blocks * PLAIN_BLOCK_SIZE + cipher_bytes_in_last_block - BLOCK_OVERHEAD + + +@dataclass(frozen=True) +class Ags1Layout: + """Where each block of an AGS1 stream sits, derived from the encrypted file length. + + Only the final block may hold less than `PLAIN_BLOCK_SIZE` of plaintext, so the layout + follows from the encrypted length alone, without reading the stream. + """ + + plaintext_length: int + num_blocks: int + last_cipher_block_size: int + + @classmethod + def from_encrypted_length(cls, encrypted_length: int) -> Ags1Layout: + """Derive the layout of an AGS1 stream that occupies `encrypted_length` bytes.""" + plaintext_length = calculate_plaintext_length(encrypted_length) + stream_length = encrypted_length - GCM_STREAM_HEADER_LENGTH + if stream_length == 0: + return cls(plaintext_length=0, num_blocks=0, last_cipher_block_size=0) Review Comment: Thanks for filing apache/iceberg#18219. iceberg-rust `main` still accepts a header-only stream: [`calculate_plaintext_length`](https://github.com/apache/iceberg-rust/blob/830bb886542527f4e8956f03040e7c407b621401/crates/iceberg/src/encryption/stream.rs#L206-L217) returns 0 for a stream length of 0. The `MIN_STREAM_LENGTH` check is in apache/iceberg-rust#3236, which is still open. So the README's "as does iceberg-rust" isn't true of any released or merged Rust code yet. The bigger question is that the spec's form of an empty file is the header alone, since the last block must have a non-zero length. With this change, PyIceberg refuses the one encoding the spec defines. The trusted `file_length` already rules out truncation, so accepting 8 bytes on read doesn't open an attack. Could the reader accept both forms (header only, and header plus one empty block) until #18219 settles which one writers produce? That keeps the writer question open for the output stream PR without making PyIceberg reject spec-compliant files. The module docstring (line 26) and the README (lines 36-40) would then describe the discrepancy and link #18219, instead of stating Java's behavior as the format. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
