JohnEarle commented on PR #17834:
URL: https://github.com/apache/iceberg/pull/17834#issuecomment-5431620743
> Isn't the Server supposed to do this logic already? It should have sent
back information with which delete files are associated with which content
files so the client doesn't actually have to determine this.
>
> The response should look like
>
> ```
> {
> "data-file": { "... ContentFile with content: data ..." },
> "delete-file-references": [0, 2], // All of these MUST be applied
> "residual-filter": { "... optional predicate ..." }
> }
> ```
Hi Russell,
during testing my POC did send the resolved delete-file references, but this
failed on Trino because its delete filter reads the sequence-number fields.
The shared ContentFileParser has no handling for those fields, so they come
back null on the client side.
I've since tested pyspark and pyiceberg and they both handled the same
merge-on-read case correctly under server planning.
It looks like it’s an implementation problem with Trino, maybe others.
This could be merged as a compatibility measure, that allows Trino and
others to continue this method of delete verification or I can look into
addressing it on the Trino side but it seems this safeguard is a design choice
Another item that popped up, the v3 lineage column
(_last_updated_sequence_number) appears to be populated from the data file's
sequence number.
Using server planning it comes back null even where deletes
work (pyspark and pyiceberg).
This part may still be worth adding still
With all this said, this is outside my area of expertise, so I may be
overlooking or misunderstanding concepts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
