xanderbailey opened a new pull request, #4326: URL: https://github.com/apache/iceberg-python/pull/4326
<!-- Thanks for opening a pull request! --> Working towards https://github.com/apache/iceberg-python/issues/3222 <!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. --> <!-- Closes #${GITHUB_ISSUE_ID} --> # Rationale for this change We need to be able to read deletion vectors stored in encrypted Puffin files. - An encrypted Puffin file is one AGS1 stream, like a manifest. Its delete file's `key_metadata` holds the plaintext key metadata for that stream, so this doesn't need the `EncryptionManager` or the KMS. - `read_deletion_vectors` decrypts the file when `key_metadata` is set, for both whole-file reads and `content_offset` / `content_size_in_bytes` range reads. Those offsets refer to the decrypted file, as in iceberg-rust's deletion vector loader (apache/iceberg-rust#3035, on top of Puffin encryption in [14/N] apache/iceberg-rust#2822). - The check from #4309 that rejects encrypted non-Parquet files now allows Puffin. Encrypted ORC files still raise. Range reads decrypt the whole file for now, and Rust only decrypts the blocks it needs. A deletion vector file is usually small, so I've left that for a follow-up. ## Are these changes tested? Yes. A Puffin file with one deletion vector, encrypted by iceberg-rust, is checked in under `tests/table/encryption`. Reading it both as a whole file and by content range returns its deleted positions. There's also a round trip of an encrypted Puffin file through both read paths. Reads fail with a wrong key, and without key metadata the encrypted bytes don't parse. ## Are there any user-facing changes? No. Nothing changes for unencrypted tables. <!-- In the case of user-facing changes, please add the changelog label. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
