xanderbailey commented on code in PR #2584:
URL: https://github.com/apache/iceberg-rust/pull/2584#discussion_r3414948337
##########
crates/iceberg/src/arrow/reader/pipeline.rs:
##########
@@ -431,14 +437,44 @@ impl ArrowReader {
)
.with_parquet_read_options(parquet_read_options);
- let arrow_metadata = ArrowReaderMetadata::load_async(&mut reader,
Default::default())
+ let arrow_reader_options =
Self::build_arrow_reader_options(key_metadata)?;
+
+ let arrow_metadata = ArrowReaderMetadata::load_async(&mut reader,
arrow_reader_options)
.await
.map_err(|e| {
Error::new(ErrorKind::Unexpected, "Failed to load Parquet
metadata").with_source(e)
})?;
Ok((reader, arrow_metadata))
}
+
+ /// Builds `ArrowReaderOptions`, adding `FileDecryptionProperties` when
+ /// key metadata is present for Parquet Modular Encryption.
+ fn build_arrow_reader_options(key_metadata: Option<&[u8]>) ->
Result<ArrowReaderOptions> {
Review Comment:
Ah this is very interesting, I opened a thread
https://github.com/apache/iceberg/pull/16527/changes and
https://lists.apache.org/[email protected]:lte=1M:[DISCUSS]%20Specifying%20the%20encryption%20key%20metadata%20formats%20for%20cross-implementation%20interop
to actually close this gap in the spec. It would appear to me to be the case
that adding specific implementations here would beak a table's ability to
interop with different engines / clients and hence I have suggested closing
that gap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
