somandal opened a new issue, #10745: URL: https://github.com/apache/pinot/issues/10745
The aggregation functions supported by v1 engine are not fully supported in v2 engine and we should be able to leverage them without rewriting them for v2 engine. Today the multistage engine expects that to add support for each new aggregation function, we modify the planner and runtime code. This approach is not scalable in the long term and requires detailed knowledge of the Pinot engine to add support for new functions. We should make the multistage engine aggregation support generic and extensible. Some of the high-level changes we need to make: - Generic way to register all existing and new aggregate functions dynamically to the SqlOperatorTable (using annotations / etc). (By overriding getOperatorList()) . Also have an additional check in Optimization rule matches() to early terminate if a agg function doesn’t exist. This way, when a new Aggregation Function is introduced in Pinot, the author need not make heavy planner and runtime changes. - Onmatch() The AggregationFunction should dictate the splitting logic. Planner Optimization should not get into the business of determining the logic for each and every aggregation function. - AggregateOperator (in intermediate stage) should be merged or at least call into AggregationOperator (in leaf). The accumulate logic should be common. The row vs column processing can be implemented as a wrapper to convert wherever necessary. - AggregateOperator should not work on RexExpression. Should be converted to a ExpressionContext . Also leaf AggregationFunction currently takes in the QueryContext. This should also be avoided. cc @siddharthteotia @vvivekiyer @walterddr @kishoreg -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
