sunchao commented on PR #6035: URL: https://github.com/apache/datafusion-comet/pull/6035#issuecomment-6282991945
@viirya, I revised the PR to address your [latest review](https://github.com/apache/datafusion-comet/pull/6035#pullrequestreview-5576270147). You are right that only the cast is dispatched and that the native conditional masks its input. The whole-projection fallback is removed. `CometCast.scala`, `cast_literal_ansi.sql`, and the shared error helper now match main, preserving #6826's dispatch and native-execution guarantees. There is still a separate LIMIT case. A minimal reproducer uses ANSI mode and Parquet rows `(id, value) = (0, '0'), (1, '1')` in the same batch: ```sql SELECT CAST(IF(id = 1, 'bad', value) AS INT) AS parsed FROM cast_branch_rows LIMIT 1 ``` Spark returns `0`. On the PR's main base (`80b518173`), Comet raises `CAST_INVALID_INPUT`. Row 1 actually satisfies the IF condition, so branch masking correctly selects it. The problem is that the projection evaluates that row before the outer LIMIT can stop consumption. The PR now caps the input of an immediately adjacent native projection: ```text CollectLimit(n, offset) CometProject CometLocalLimit(n) child ``` The outer limit still combines partitions and drops OFFSET rows. Column selections and aliases skip the extra cap to preserve filter column pruning. Other plan shapes are outside this PR. At `1dc896732`, 116 local Spark 4.0 tests passed, including native projection/cast-dispatch assertions, demanded errors, OFFSET, multiple batches and partitions, AQE, and pruning coverage. Spotless and Scalastyle also passed. Hosted CI, including the requested Spark 4.1 SQL coverage, is still pending. I also replied to the three inline comments about assertions, folding coverage, and `checkNative`. Could you take another look at the revised scope? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
