andygrove opened a new issue, #4936: URL: https://github.com/apache/datafusion-comet/issues/4936
## Background This is an umbrella issue for performance-tuning the native scalar expression implementations in `datafusion-comet-spark-expr` (`native/spark-expr/`), prioritizing the expressions that carry the most per-row cost in TPC-DS. The methodology is documented in the contributor guide (Optimizing Scalar Expressions): benchmark-first with criterion, keep output bit-identical to `main` (existing unit tests are the correctness gate), cover the input shapes where fast paths regress (no-null / sparse-null / dense-null, short / long), and do not submit a change that meaningfully regresses any shape. Tuned expressions are recorded as `Performance (tuned ...)` entries in the per-expression audit pages. ## Scope TPC-DS per-row scalar work is dominated by decimal arithmetic, casts, and a handful of string/conditional expressions. Aggregates (`sum`/`avg`/`count`/`stddev_samp`) are the biggest overall cost but are out of scope here (aggregate, not scalar). Candidate expressions (to be confirmed and benchmarked individually): - [ ] `CheckOverflow` (`math_funcs/internal/checkoverflow.rs`) - wraps the result of every decimal `+ - * /`, `sum`, and `avg`. The common non-ANSI path always allocates a new Decimal128 array via `null_if_overflow_precision`, even when nothing overflows. Opportunity: a no-overflow zero-copy fast path. No benchmark exists yet. - [ ] Decimal rescale (`math_funcs/internal/decimal_rescale_check.rs`) - used by decimal-to-decimal casts; audit for per-row allocation. - [ ] Further casts used heavily in TPC-DS (decimal casts), triaged against work already merged or in flight. ## Already optimized / in flight Many scalar expressions have already been tuned in a prior campaign (casts, `substring`, `date_trunc`, `floor`/`ceil`, `to_json`, `parse_url`, `size`, `unhex`, and others). New work under this epic should check the per-expression audit pages first to avoid re-treading tuned expressions. ## How to contribute Each expression is a small, self-contained PR: 1. Add a criterion benchmark under `native/spark-expr/benches/` covering the shapes above. 2. Capture a baseline on `main`, apply the optimization, confirm bit-identical output, re-measure. 3. Only submit a strict improvement (no meaningful regression on any shape). 4. Record a `Performance (tuned ...)` entry in the relevant audit page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
