qzyu999 commented on issue #23692: URL: https://github.com/apache/datafusion/issues/23692#issuecomment-5012549225
## Root Cause Analysis Update After deeper investigation, the bug is **NOT** specific to Utf8View or any particular join type. Key findings: ### Reproduction Multi-condition IS NOT DISTINCT FROM in ANY join ON clause fails type_coercion: - Single condition works: ON l.a IS NOT DISTINCT FROM r.a - Multi-condition fails: ON l.a IS NOT DISTINCT FROM r.a AND l.b IS NOT DISTINCT FROM r.b - Fails for ALL join types (INNER, LEFT, LEFT ANTI, LEFT SEMI) - Fails for ALL data types (Int32, Int64, Utf8, Utf8View) - Fails with CREATE TABLE, register_parquet, and register_record_batches ### Exception Case **Works when the data contains NULLs in the join columns AND both tables have the same rows.** Example: \\\sql CREATE TABLE t1 (val1 INT, val2 INT) AS VALUES (10, 100), (NULL, 200), (30, NULL); CREATE TABLE t2 (val1 INT, val2 INT) AS VALUES (10, 100), (NULL, 200), (30, NULL); -- This works: SELECT t1.* FROM t1 LEFT ANTI JOIN t2 ON t1.val1 IS NOT DISTINCT FROM t2.val1 AND t1.val2 IS NOT DISTINCT FROM t2.val2; \\\ This suggests some optimizer rule (possibly constant folding or join elimination) short-circuits the plan before type_coercion hits the problematic path. ### Likely Root Cause Location The error occurs in **\TypeCoercionRewriter::f_up\** (analyzer/type_coercion.rs) when processing the \AND\ node in the join filter expression: 1. \map_expressions\ processes the join filter via \ _up\ (bottom-up) 2. When reaching the \AND\ node, it calls \coerce_binary_op(*left, schema, And, *right, schema)\ 3. \coerce_binary_op\ calls \left.get_type(schema)\ which should return Boolean (since left child is \IS NOT DISTINCT FROM\ already rewritten by the bottom-up pass) 4. But \left.get_type(schema)\ returns the **column type** (Int32, Utf8View) instead of Boolean The suspicion is that \get_type()\ on the rewritten \BinaryExpr(left_col, IsNotDistinctFrom, right_col)\ is somehow not recognizing it as a comparison operation that returns Boolean possibly because the schema lookup resolves the expression to a column reference instead of computing the binary operator's return type. This would happen if a \NamePreserver\ or alias wrapping is interfering with the expression structure. ### Environment - DataFusion 54.0.0 (pip: datafusion==54.0.0) - Python 3.12 on Windows - Confirmed NOT platform-specific (same behavior expected on Linux) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
