benbellick opened a new issue, #25720: URL: https://github.com/apache/datafusion/issues/25720
### Describe the bug DataFusion rejects a spec-valid Substrait `InPredicate` when it contains more than one needle expression. This represents row-valued SQL `IN` predicates like: ```sql SELECT 1 AS result WHERE (1, 2) IN (SELECT 1, 2) ``` The Substrait spec allows this shape. `InPredicate.needles` is repeated, and the proto comment explicitly includes `(x, y) IN (SELECT a, b FROM t)`: https://github.com/substrait-io/substrait/blob/b0341cea26fd0a83eb2dfd8e9d4f64df1b8e1759/proto/substrait/algebra.proto#L1818-L1820 ### To Reproduce Run DataFusion's Substrait consumer on this minimal plan, generated with `substrait-go`: ```json { "version": { "minorNumber": 29, "producer": "substrait-go v9.0.0-alpha.0 linux/arm64" }, "relations": [ { "root": { "input": { "project": { "common": { "direct": {} }, "input": { "filter": { "common": { "direct": {} }, "input": { "read": { "common": { "direct": {} }, "baseSchema": { "struct": { "nullability": "NULLABILITY_REQUIRED" } }, "virtualTable": { "expressions": [ {} ] } } }, "condition": { "subquery": { "inPredicate": { "needles": [ { "literal": { "i64": "1", "nullable": true } }, { "literal": { "i64": "2", "nullable": true } } ], "haystack": { "project": { "common": { "direct": {} }, "input": { "read": { "common": { "direct": {} }, "baseSchema": { "struct": { "nullability": "NULLABILITY_REQUIRED" } }, "virtualTable": { "expressions": [ {} ] } } }, "expressions": [ { "literal": { "i64": "1", "nullable": true } }, { "literal": { "i64": "2", "nullable": true } } ] } } } } } } }, "expressions": [ { "literal": { "i64": "1", "nullable": true } } ] } }, "names": [ "result" ] } } ] } ``` DataFusion returns: ```text Substrait error: InPredicate Subquery type must have exactly one Needle expression ``` The immediate rejection is in the Substrait consumer: https://github.com/apache/datafusion/blob/a98f30478427246ebf50674048167ab49e391672/datafusion/substrait/src/logical_plan/consumer/expr/subquery.rs#L53-L57 ### Expected behavior DataFusion should accept this Substrait plan and return one row: ```text result ------ 1 ``` ### Additional context This likely needs broader support beyond removing the Substrait guard. The logical expression model currently describes `InSubquery` as one expression compared against a single-column subquery: https://github.com/apache/datafusion/blob/a98f30478427246ebf50674048167ab49e391672/datafusion/expr/src/expr.rs#L1384-L1388 The SQL planner also validates `IN` subqueries as single-column: https://github.com/apache/datafusion/blob/a98f30478427246ebf50674048167ab49e391672/datafusion/sql/src/expr/subquery.rs#L73-L78 There is related SQL-side work in https://github.com/apache/datafusion/pull/19857, but I did not find an issue tracking Substrait multi-needle `InPredicate` support specifically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
