sunchao commented on PR #5318: URL: https://github.com/apache/datafusion-comet/pull/5318#issuecomment-5684715528
Found **one P2 issue** at head `a4541f6`. ### Valid MERGE fails for nested non-null fields The new check in [planner.rs:2206](https://github.com/apache/datafusion-comet/blob/a4541f6dd9166cf7e46b657ec7ae280c2152c742/native/core/src/execution/planner.rs#L2206) rejects valid expressions when native metadata marks a nested field nullable. For example, with target type `ARRAY<STRUCT<n: INT NOT NULL>>`: ```sql WHEN MATCHED THEN UPDATE SET payload = array(named_struct('n', coalesce(s.id, 0))) ``` This succeeds on Spark **4.0.4**, but enabling native MERGE fails with `incompatible with Spark output type`. Comet widens the nested nullability during expression conversion; the new guard rejects it before runtime schema normalization can handle it. Allow safe normalization with runtime validation, or fall back during Spark planning. ### Validation - **23 native tests passed.** - **10 existing Spark 4.0 MERGE tests passed.** - **Two added SQL probes reproduced this same defect.** - The four previous findings are fixed. - Current CI has only a passing labeling check. Spark 3.5 and Iceberg-specific execution were not tested locally. Checkout restored; nothing posted to GitHub. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
