yashmayya opened a new pull request, #13255:
URL: https://github.com/apache/pinot/pull/13255
- Currently, queries such as `SELECT add(null, 1) FROM mytable` fail with
the multi-stage query engine returning the following exception:
```
Caught exception planning request 346348145000000011: SELECT add(null, 1)
FROM mytable;, Error composing query plan for: SELECT add(null, 1) FROM mytable;
org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:138)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:145)
org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:125)
org.apache.pinot.broker.requesthandler.BrokerRequestHandlerDelegate.handleRequest(BrokerRequestHandlerDelegate.java:86)
Failed to generate a valid execution plan for query:
LogicalProject(EXPR$0=[add(null:JavaType(class java.lang.Double), 1)])
LogicalTableScan(table=[[default, mytable]])
org.apache.pinot.query.QueryEnvironment.optimize(QueryEnvironment.java:289)
org.apache.pinot.query.QueryEnvironment.compileQuery(QueryEnvironment.java:236)
org.apache.pinot.query.QueryEnvironment.planQuery(QueryEnvironment.java:129)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:145)
type mismatch:
type1:
DOUBLE
type2:
DOUBLE NOT NULL
org.apache.calcite.util.Litmus.lambda$static$0(Litmus.java:31)
org.apache.calcite.plan.RelOptUtil.eq(RelOptUtil.java:2204)
org.apache.calcite.rex.RexUtil.compatibleTypes(RexUtil.java:1220)
org.apache.calcite.rel.core.Project.isValid(Project.java:255)
```
- This issue arises from a type mismatch issue when Calcite's rule based
`HepPlanner` fires the
[PinotEvaluateLiteralRule](https://github.com/apache/pinot/blob/717220cf12f4a4b79735b3654b14ff5091af839f/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotEvaluateLiteralRule.java#L54).
- The scalar function [double plus(double a, double
b)](https://github.com/apache/pinot/blob/518fd180981ffa614042b8a5236650a2af41bfe5/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ArithmeticFunctions.java#L34)
is registered with Calcite
[here](https://github.com/apache/pinot/blob/fdfae5e57aaeb4cd8ca2921cee33c2a45534e40c/pinot-common/src/main/java/org/apache/pinot/common/function/FunctionRegistry.java#L135)
and the function's return type is determined as `DOUBLE NOT NULL` because of
the primitive return type of the method (see
[here](https://github.com/apache/calcite/blob/694b556a2ece4953d8e9145352eb2340e1fac908/core/src/main/java/org/apache/calcite/schema/impl/ScalarFunctionImpl.java#L158)
/
[here](https://github.com/apache/calcite/blob/694b556a2ece4953d8e9145352eb2340e1fac908/core/src/main/java/org/apache/calcite/rel/type/RelDataTypeFactoryImpl.java#L617-L634)).
This causes the type mismatch issue in the `PinotEvaluateLiteralRule` because
the new project crea
ted
[here](https://github.com/apache/pinot/blob/717220cf12f4a4b79735b3654b14ff5091af839f/pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotEvaluateLiteralRule.java#L77-L98)
has a `RexLiteral` with type `DOUBLE` and value `null`, whereas the old
project has a `RexCall` with type `DOUBLE NOT NULL`.
- The core issue is that even though these arithmetic functions return
primitive values, they can still return `null` because they are "null
intolerant" and `null` will be returned without even invoking the method if any
of the arguments is null
([here](https://github.com/apache/pinot/blob/518fd180981ffa614042b8a5236650a2af41bfe5/pinot-common/src/main/java/org/apache/pinot/common/function/FunctionInvoker.java#L131-L136)).
- Calcite's `Strict` (function returns null if and only if one of the
arguments are null) and `SemiStrict` (function returns null if one of the
arguments is null, and possibly other times) annotations can help us out here -
https://github.com/apache/calcite/blob/694b556a2ece4953d8e9145352eb2340e1fac908/core/src/main/java/org/apache/calcite/schema/impl/ScalarFunctionImpl.java#L185-L205.
With this, the function's return type in Calcite will be `DOUBLE` instead of
`DOUBLE NOT NULL` and the `PinotEvaluateLiteralRule` works out fine.
- This patch also switches from table based null handling to column based
null handling in `NullHandlingIntegrationTest` (to support null handling in the
multi-stage query engine) and enables two additional tests to run on both query
engines.
- Suggested label: `bugfix`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
