andygrove opened a new issue, #7395: URL: https://github.com/apache/datafusion-comet/issues/7395
## What is the problem the feature request solves? Comet pins the Rust `datasketches` crate at `=0.3.0` with only the `hll` feature. That release has two HLL decode defects, and `native/spark-expr/src/agg_funcs/hll_sketch.rs` carries a workaround for each: - `Array4`/`Array6`/`Array8::deserialize` skip the register block when the COMPACT flag is set, so a decoded sketch keeps its own HIP estimate but every union built from it is wrong. Comet clears the flag before decoding (`normalize_compact_hll_array`). - An updatable HLL_4 sketch with auxiliary-map exceptions is read with the compact aux layout, so the exceptions are dropped and the estimate is silently wrong. Comet rejects that shape with an error (`reject_undecodable_hll4`), which means a sketch column written by DataSketches-Java with its default target type (HLL_4) can fail in `hll_sketch_estimate`, `hll_union` and `hll_union_agg` once it holds enough distinct values. The crate's changelog lists fixes for both (compact HLL4 register restore in 0.5.0, the compact auxiliary-map flag in 0.4.0), plus HLL union estimate fixes that match Java and C++. 0.6.0 was released on 2026-10-29. ## Describe the potential solution Bump to `=0.6.0` and delete both workarounds. Before deleting them, check the 0.6.0 decoder against DataSketches-Java 6.2.0 (the version Spark 4.1 ships) rather than trusting the changelog: serialize HLL_4, HLL_6 and HLL_8 sketches in compact and updatable form at several cardinalities, including ones with aux entries, and compare estimates and union estimates with Java's. The API changes are small: `hash_value` moved to `hash::value`, and `HllSketch::new` / `HllUnion::new` now return `Result`. ## Additional context Follow-on work is tracked separately: an HLL audit against the new crate, and an epic for the sketch families the crate now provides (theta, KLL, frequent items, tuple). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
