mkleen commented on PR #26000:
URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6271195478
@mohitgurav20 I ran this through a conversion with claude and it came up
with the following decision-tree based on the benchmarks when to rewrite the
query and when to add a switch:
```
Does the query has a GROUP BY?
│
├─ No ──► Is there more than 1 partition, and is it not count(DISTINCT
tinyint/smallint)?
│ ├─ Yes ──► REWRITE (2.6–5.5x faster: the only way the
distinct runs in parallel)
│ └─ No ──► SKIP (faster and uses less memory)
│
└─ Yes ─► Does the distinct aggregate have its own fast grouped
accumulator?
├─ Yes (count(DISTINCT <integer>)) ──► SKIP (faster or a
tie, less memory)
└─ No (sum/avg DISTINCT, floats, decimals,
strings, user-defined) ──► REWRITE (1.4–20x
faster)
└─ config option to skip
it when memory
matters more (few
groups: 2.4–6x more memory)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
