tanmayrauth commented on PR #2444:
URL: https://github.com/apache/iceberg-go/pull/2444#issuecomment-6909970724

    Thanks @laskoviymishka, all six threads are addressed in the latest push. 
ComputeNDVs now returns NDVStats{Snapshot, Columns}, so the Puffin writer gets 
the snapshot ID and sequence number without resolving it again; Snapshot is  
nil only when the table has no snapshot, in which case the result is empty like 
Java's EMPTY_RESULT instead of an NDV=0 sketch per column. Column names are 
resolved against the snapshot schema first, and only the columns being  
sketched go to WithSelectedFields, so nested and geo columns are no longer 
read; deletes on unselected columns still apply. Geo columns are skipped by 
default and rejected with ErrInvalidArgument when named, because Java's    
Conversions serializes geometry and geography only as a GeospatialBound point, 
so there's no shared single-value encoding for Go-only WKB sketches to match. 
Since names are checked before the scan is built, every unknown name,  "*" 
included, now returns ErrInvalidArgument instead of some returning ErrInvalidS
 chema from Schema.Select, which matches Java's checkArgument. 
   
   The matrix is in this PR: TestComputeNDVsTypes covers 14 primitive types, 
each checked against a sketch built from Iceberg literals (pinned to the Java 
fixtures), and TestComputeNDVsSnapshotContents covers several data files, 
position deletes, deletion vectors, equality deletes, and an older snapshot 
after a schema change, with each delete case checking that real delete files 
are present. One thing it turned up: empty strings and empty binary values 
aren't counted, because DataSketches ignores empty byte arrays, and Java's 
string fixture behaves the same way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to