andygrove opened a new issue, #7397: URL: https://github.com/apache/datafusion-comet/issues/7397
## What is the problem the feature request solves? Comet accelerates one DataSketches family, HLL, on Spark 4.0+. Spark ships four more families as built-in SQL functions, and today every one of them falls back to the JVM. The Rust `datasketches` crate now provides implementations of all of them (it was HLL-only in how we used it, and the other families were thinner or absent before 0.6.0). | Family | Spark functions | Spark versions | Crate feature | |---|---|---|---| | HLL | `hll_sketch_agg`, `hll_union_agg`, `hll_sketch_estimate`, `hll_union` | 4.0+ | `hll` (done) | | Theta | `theta_sketch_agg`, `theta_union_agg`, `theta_intersection_agg`, `theta_sketch_estimate`, `theta_union`, `theta_intersection`, `theta_difference` | 4.1+ | `theta` | | KLL | `kll_sketch_agg_*`, `kll_merge_agg_*`, `kll_sketch_get_quantile_*`, `kll_sketch_get_rank_*`, `kll_sketch_get_n_*`, `kll_sketch_merge_*`, `kll_sketch_to_string_*` (bigint, float, double) | 4.1+ | `kll` | | Frequent items | `approx_top_k`, `approx_top_k_accumulate`, `approx_top_k_combine`, `approx_top_k_estimate` | 4.1+ | `frequencies` | | Tuple | `tuple_sketch_agg`, `tuple_union_agg`, `tuple_intersection_agg`, `tuple_difference`, `tuple_sketch_estimate/summary/theta`, and variants for integer and double summaries | 4.2 | `tuple` | Spark 4.1 and 4.2 use DataSketches-Java 6.2.0 and Spark 4.0 uses 6.1.1. These are exactly the "exact-correctness traps" the expression docs warn about: the sketch bytes cross the JVM/native boundary and can be read by either side, so serialization has to be compatible with DataSketches-Java, input hashing has to match Spark's (including how each input type is widened, and how empty strings and nulls are treated), and the type rules differ per family. ## Describe the potential solution Take one family at a time, in the order below, each as its own issue and PR. For each, the same three steps that worked for HLL: 1. **Audit** the crate against Java: serialize sketches with DataSketches-Java 6.2.0 (and 6.1.1 for 4.0), commit the bytes as fixtures, and check that the crate decodes them and that unions, estimates and bounds match Java's numbers. Check byte-for-byte parity for what Comet writes. 2. **Implement** the serde and native aggregate/scalar functions, with the type and null handling taken from the Spark source for every version Comet supports, behind the usual opt-in until step 1 shows parity. 3. **Test** with SQL file tests, including a Spark-produced sketch column read by Comet and a Comet-produced one read by Spark. Order, by how much of the HLL work carries over: theta (hash-based, same hashing wrappers), then KLL, then frequent items, then tuple. Not scored yet for throughput gain over the JVM codegen dispatcher; each family issue should do that first via the `suggest-native-expression` skill before investing in the implementation. ## Additional context Follows #7395 (the 0.6.0 upgrade) and the HLL audit. Part of the broader native expression effort in #7300. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
