grorge123 commented on PR #6716:
URL:
https://github.com/apache/datafusion-comet/pull/6716#issuecomment-6125302660
Thanks, 3090d859a fixes it. My query and a range-partitioning one whose two
keys share the cast now raise as in Spark, and the updated fixtures and
`QueryPlanSerdeSuite` pass on Spark 4.1.
There's one gap left in the same class. `CometTakeOrderedAndProjectExec`
computes the shared set from `expressions`, so from the sort keys and the
projection together. Spark evaluates them separately: its ordering evaluates
each sort key on its own, with no subexpression elimination, and the projection
runs only on the rows it keeps. So Spark returns the rows for both of these,
while Comet raises `CAST_INVALID_INPUT`:
```sql
-- the cast is in a sort key and in the projection
SELECT array_contains(ai, CAST(s AS INT))
FROM (SELECT * FROM t ORDER BY array_position(ai, CAST(s AS INT)), k LIMIT
10)
-- the cast is in two sort keys
SELECT k FROM (SELECT * FROM t
ORDER BY array_contains(ai, CAST(s AS INT)), array_position(ai, CAST(s AS
INT)), k LIMIT 10)
```
Main raises for both too, so neither is a regression, but it's the failure
#6613 is about, and the fixture comment already describes the intended behavior
("its projection list"). Using an empty set for the sort keys and `projectList`
for the projection would match Spark. If you'd rather leave it, it could go in
#6850.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
