xiangfu0 opened a new pull request, #19015: URL: https://github.com/apache/pinot/pull/19015
## Summary - upgrade `com.tdunning:t-digest` from 3.2 to 3.3 while retaining the K1 scale function - centralize mixed-version-compatible serialization and deserialization, including legacy buffer limits, boundary repair, low and fractional compression, double-precision weights, and non-finite values - route raw, Smart, standard, StarTree, and merge-and-rollup paths through a primitive accumulator that avoids repeated digest materialization and sorting - add matched 3.2/3.3 JMH coverage, benchmark results, rollout guidance, sample SQL, and a sample StarTree configuration ## Why upgrade instead of switching to DataSketches This keeps Pinot's existing t-digest SQL behavior and serialized intermediate-state format. Replacing it with the DataSketches implementation would be a separate wire-format, API, and mixed-version migration rather than a dependency upgrade. ## Compatibility and rollout - finite t-digest state is readable in both directions between 3.2 and 3.3; no schema, table-config, or segment rebuild is required - exact serialized bytes and centroid layouts can differ, as expected for an approximate aggregation - compression values below 10 have an effective compression of 10 in t-digest 3.3 - non-finite centroids remain structurally readable by 3.2, but generic 3.2 readers can produce incorrect quantiles for them; filter or sanitize non-finite inputs until all readers are upgraded The detailed operator guide and reproduce commands are in [`percentile-tdigest-aggregation-100m-results.md`](pinot-perf/benchmark-results/percentile-tdigest-aggregation-100m-results.md). ## Performance Matched JMH runs used JDK 25, JMH 1.37, 2 forks, 2 x 1-second warmups, 5 x 1-second measurements, one thread, and the GC profiler. | Path | 3.3 latency vs 3.2 | 3.3 allocation vs 3.2 | |---|---:|---:| | Raw P75 query, 100M rows | +5.2% | +0.4% | | StarTree query, no groups | +22.6% | +10.7% | | StarTree query, 1,000 groups | +21.7% | +15.1% | | Raw digest construction, 100M rows | -7.9% | +0.8% | | Construction from 10 serialized leaves per parent | -62.8% | -6.8% | | Reducer merge kernel | -84.3% | -53.1% | | `IndexedTable` combine | -83.9% | -50.7% | | Combine plus final extraction | -84.2% | -50.6% | The StarTree regression is called out explicitly; the largest production gain is the serialized reducer path. The maximum displayed candidate quantile error remained below `0.000147` absolute for benchmark values in `[0, 1]`. ## Validation - 439 focused `pinot-core` aggregation and StarTree tests - 28 `TDigestUtils` and `PercentileTDigestValueAggregator` tests - 35 `ObjectSerDeUtilsTest` tests under JDK 21 - 8 `MergeRollupTDigestTaskExecutorTest` tests - `pinot-perf` package build - `spotless:apply`, `checkstyle:check`, `license:format`, and `license:check` for all affected modules -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
