mbutrovich commented on code in PR #2868:
URL: https://github.com/apache/iceberg-rust/pull/2868#discussion_r3866887325
##########
crates/iceberg/src/scan/task.rs:
##########
@@ -239,6 +242,27 @@ pub struct FileScanTaskDeleteFile {
#[builder(default)]
pub equality_ids: Option<Vec<i32>>,
+ /// For a deletion vector, the location of the data file whose rows it
deletes. Required for
+ /// deletion vectors, and may also be set on a position delete file scoped
to one data file.
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ #[builder(default)]
+ pub referenced_data_file: Option<String>,
+
+ /// For a deletion vector, the offset of the blob within its Puffin file.
Set only for
+ /// deletion vectors, where it locates the blob for direct access.
+ #[serde(default)]
+ #[serde(skip_serializing_if = "Option::is_none")]
+ #[builder(default)]
+ pub content_offset: Option<i64>,
+
+ /// For a deletion vector, the length in bytes of the blob within its
Puffin file. Set
+ /// whenever `content_offset` is.
Review Comment:
Took your wording verbatim.
Agreed on the DV check: the rule should be `content == POSITION_DELETES &&
format == PUFFIN`, and a DV missing `content_offset` or `content_size_in_bytes`
should error instead of falling through to the position-delete path.
`FileScanTaskDeleteFile` has no format field today, only `file_type`, which
is why #3035 keys on `content_offset.is_some()` in the loader.
`DeleteFileIndex` works on `DataFile`, which has `file_format()`, so it can use
the rule as written.
Rather than land an unused field here, I'll do this in #3035: add
`file_format` to `FileScanTaskDeleteFile`, switch both the index and the loader
to the type + format check, and error when a DV is missing either coordinate.
Let me know if you'd rather the field land in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
