AdamGS opened a new issue, #23847: URL: https://github.com/apache/datafusion/issues/23847
### Describe the bug The optimizer rule `FilterNullJoinKeys` doesn't take null-awarness into account, which can cause it to transform the logical plan in an invalid way, returning wrong results. This behavior isn't enabled by default, but requires setting `datafusion.optimizer.filter_null_join_keys` to `true`. ### To Reproduce From `null_aware_anti_join.slt`, run ```sql set datafusion.optimizer.filter_null_join_keys = true; SELECT * FROM outer_table WHERE id NOT IN (SELECT id FROM inner_table_with_null); ``` which will return incorrect results, which are different than the default `datafusion.optimizer.filter_null_join_keys = false`: ``` 1 a 3 c 4 d ``` ### Expected behavior Config changes shouldn't impact correctness ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
