mizulun opened a new pull request, #6073: URL: https://github.com/apache/datafusion-comet/pull/6073
## Which issue does this PR close? Closes #6019. ## Rationale for this change Floating-point normalization for IN / InSet currently handles only top-level FLOAT and DOUBLE operands. Arrays and structs containing floating-point values remain unnormalized, so Comet can return false where Spark returns true when comparing nested -0.0 and +0.0 values. This change extends normalization to nested membership operands so that signed zero and different NaN representations follow Spark's equality semantics. ## What changes are included in this PR? - Add a NormalizeNestedFloats physical expression using the existing recursive floating-point normalization logic. - Normalize both the input and candidate values in native IN planning for arrays and structs containing floating-point values. - Preserve scalar constant candidates for static membership lookup. - Add regression tests for nested floating-point membership, including native IN and InSet execution. ## How are these changes tested? Native unit tests cover nested floating-point membership with signed zero, distinct NaN representations, nulls, and empty arrays, including constant and column-based candidates. SQL regression tests compare Comet results with Spark for arrays and structs. A CometExpressionSuite test verifies native execution of both IN and InSet. Local validation: ```./mvnw test -Dtest=none -Dsuites='org.apache.comet.CometSqlFileTestSuite @sql-file: expressions/conditional/in_nested_zero.sql [spark.sql.optimizer.inSetConversionThreshold=100]' ``` ```./mvnw test -Dtest=none -Dsuites='org.apache.comet.CometSqlFileTestSuite @sql-file: expressions/conditional/in_nested_zero.sql [spark.sql.optimizer.inSetConversionThreshold=0]' ``` ```./mvnw test -Dtest=none -Dsuites='org.apache.comet.CometExpressionSuite @nested floating point membership uses native In and InSet' ``` ```./mvnw test -Dtest=none -Dsuites='org.apache.comet.CometSqlFileTestSuite' ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
