yashmayya commented on PR #14102:
URL: https://github.com/apache/pinot/pull/14102#issuecomment-2379383099
The failing integration test is an interesting case:
```
SELECT AVG(AirTime), SUM(OriginStateFips) FROM mytable WHERE DivTailNums
BETWEEN 'N8315C' AND 'N130DL' OR DivAirports IN ('PIH', 'DLH', 'MTJ', 'SEA',
'LIT') AND TaxiOut NOT IN (59, 112, 57, 72) LIMIT 5
```
In this query, the `BETWEEN` filter predicate is on an MV column
`DivTailNums`. The [dictionary based predicate
evaluator](https://github.com/apache/pinot/blob/4709954097703f88dbd419ab0436a4c1356e9c85/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/PredicateEvaluatorProvider.java#L64-L65)
created for the `BETWEEN` filter predicate is reduced to an always false
predicate here -
https://github.com/apache/pinot/blob/4709954097703f88dbd419ab0436a4c1356e9c85/pinot-core/src/main/java/org/apache/pinot/core/operator/filter/predicate/RangePredicateEvaluatorFactory.java#L163-L165
because `N8315C` is > `N130DL`.
However, when the `BETWEEN` filter predicate is re-written to `DivTailNums
>= 'N8315C' AND DivTailNums <= 'N130DL'`, there are actually matches found
since it is an MV column with multiple values (and the nature of filter
predicates on MV columns is such that true is returned if any single value
matches). Unfortunately, since the query rewrite is being done in the
`CalciteSqlParser` which doesn't have access to the schema, there is no way to
skip this rewrite for MV columns. There's also no straightforward way to remove
such filters altogether due to the various type combinations we'd need to
handle in order to check if such a filter is "always false".
We could do this in the filter optimizer phase instead, which does have
access to the schema, but then if we also want to support `col1 BETWEEN col2
AND 100` like use cases we'll need to duplicate the identifier / literal
re-ordering logic across the query rewriter and query optimizer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
