tanmayrauth opened a new issue, #2616:
URL: https://github.com/apache/iceberg-go/issues/2616
### Feature Request / Improvement
Table.ComputeNDVs currently feeds each value into its Theta sketch by
converting it to an Iceberg Literal (getArrowValueAsIcebergLiteral) and then
calling MarshalBinary. That costs an interface conversion and at least one
allocation per non-null value, on every row of every analyzed column. It's fine
for correctness, but slow on large tables.
Proposal: add per-type paths that write each value's Iceberg single-value
serialization straight from the Arrow array into a reused buffer (or pass
existing bytes straight through for string, binary and fixed), and fall back to
the literal path for anything not covered. The tricky cases are:
- decimal (minimal-length big-endian two's complement, not the fixed
16-byte Arrow storage)
- uuid (16 bytes big-endian)
- the upcast cases the literal path handles today (int8/16 → int, unsigned
ints, Decimal32/64)
- timestamp unit handling
The output must stay byte-identical. TestComputeNDVsTypes checks every
primitive type against a sketch built from Iceberg literals, and
TestComputeNDVsMatchesJavaFixtures checks the end-to-end path against
Iceberg-Java's sketches, so either test fails if the encoding drifts. A
benchmark on a wide table should show the gain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]