ebyhr opened a new issue, #18516: URL: https://github.com/apache/iceberg/issues/18516
### Proposed Change ### Proposal Add a standard HyperLogLog (HLL) blob type to the Puffin spec for column NDV statistics, alongside `apache-datasketches-theta-v1`. ### Motivation Theta sketch computation for NDV is measurably slow. Trino reports a 10-20% insert slowdown in some workloads (trinodb/trino#31173). The PR's `BenchmarkNdvSketchUpdate` results (sketch update cost only, ms/op, lower is better): | Type | Theta (ms/op) | HLL (ms/op) | Speedup | |---|---|---|---| | bigint | 0.364 | 0.105 | 3.5x | | varchar(64) | 0.618 | 0.386 | 1.6x | The Puffin spec has no HLL blob type, so Trino's PR stores HLL as a Trino-specific type (`trino-datasketches-hll-v1`). Other engines can't read it, and Trino rejects switching algorithms on a table that already has statistics. A standard type would let engines share HLL-based NDV statistics. ### Proposed spec change Add `apache-datasketches-hll-v1`: a serialized Apache DataSketches HLL sketch, built from values in Iceberg's single-value serialization, with an `ndv` blob property like the Theta type. See the [DataSketches HLL docs](https://datasketches.apache.org/docs/HLL/HllSketches.html). ### Proposal document _No response_ ### Specifications - [ ] Table - [ ] View - [ ] REST - [x] Puffin - [ ] Encryption - [ ] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
