andygrove opened a new issue, #7398: URL: https://github.com/apache/datafusion-comet/issues/7398
## What is the problem the feature request solves? Part of #7397. Spark 4.1+ provides the Theta sketch functions `theta_sketch_agg`, `theta_union_agg`, `theta_intersection_agg`, `theta_sketch_estimate`, `theta_union`, `theta_intersection` and `theta_difference`. None of them run natively in Comet. Theta is the closest family to HLL, since it hashes the same way, so most of the HLL serde and testing approach carries over. ## Describe the potential solution 1. Score the family with `suggest-native-expression` first, since the sketch update is cheap next to the cost of crossing the boundary for small groups. 2. Audit the crate's `theta` feature against DataSketches-Java 6.2.0 using committed fixtures: update sketches and compact sketches, union, intersection and A-not-B results, estimates, and serialization of an empty sketch (0.6.0 changed it to match the canonical cross-language encoding). Also check the seed handling and what Spark passes for `lgNomEntries`. 3. Implement the serde and native functions once the audit shows byte parity, opt-in until then. 4. Add SQL file tests, including a Spark-produced sketch column read by Comet and the reverse. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
