tanmayrauth opened a new issue, #2616:
URL: https://github.com/apache/iceberg-go/issues/2616

   ### Feature Request / Improvement
   
   Table.ComputeNDVs  currently feeds each value into its Theta sketch by 
converting it to an Iceberg Literal (getArrowValueAsIcebergLiteral) and then 
calling MarshalBinary. That costs an interface conversion and at least one 
allocation per non-null value, on every row of every analyzed column. It's fine 
for correctness, but slow on large tables.                                      
                                                           
   
   Proposal: add per-type paths that write each value's Iceberg single-value 
serialization straight from the Arrow array into a reused buffer (or pass 
existing bytes straight through for string, binary and fixed), and fall back to 
  the literal path for anything not covered. The tricky cases are:              
                                                                                
                                                                       
    - decimal (minimal-length big-endian two's complement, not the fixed 
16-byte Arrow storage)                                                          
                                                                                
    - uuid (16 bytes big-endian)                                                
                                                                                
                                                                         
    - the upcast cases the literal path handles today (int8/16 → int, unsigned 
ints, Decimal32/64)                                                             
                                                                          
    - timestamp unit handling                                                   
                                                                                
                                                                         
   
   The output must stay byte-identical. TestComputeNDVsTypes checks every 
primitive type against a sketch built from Iceberg literals, and 
TestComputeNDVsMatchesJavaFixtures checks the end-to-end path against 
Iceberg-Java's  sketches, so either test fails if the encoding drifts. A 
benchmark on a wide table should show the gain.   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to