andygrove commented on PR #6989: URL: https://github.com/apache/datafusion-comet/pull/6989#issuecomment-6234816416
Ran them against the staged candidates. There is no v4.1.4-rc3 yet: apache/spark has no `v4.1.4-rc3` tag, Nexus has no third 4.1.4 staging repository, and dev@ only has the RC2 vote. So I used the latest candidate of each line. The `revision` in each staged `spark-version-info.properties` matches the tag: | Candidate | Staging repository | Commit | | --- | --- | --- | | v4.0.5-rc2 | `orgapachespark-1535` | `51d701c09d` | | v4.1.4-rc2 | `orgapachespark-1533` | `07af17464b` | | v4.2.1-rc2 | `orgapachespark-1537` | `cbae39b356` | Each run used the matching profile with the version overridden (`-Pspark-4.1 -Dspark.version=4.1.4`, and so on), and the test JVM logged the candidate's version. None of the three rewrites `arrays_overlap` in `NormalizeFloatingNumbers` (only 4.2.0 does), so the kernel gets the raw `-0.0`. I ran `CometFloatSemanticsSuite`, `expressions/array/arrays_overlap.sql` and the `arrays_overlap` tests in `CometArrayExpressionSuite`. `query` checks the operators but not whether `arrays_overlap` itself ran natively, so I also ran an uncommitted copy of the fixture's `FLOAT` and `DOUBLE` queries under `expect_native(arrays_overlap)`. For comparison, I ran the same set on this PR's base, 7d8d9b1544, which has no `arrays_overlap native variant by Spark version` test and so runs 419: | | v4.0.5-rc2 | v4.1.4-rc2 | v4.2.1-rc2 | | --- | --- | --- | --- | | This PR, 0870b3d8b4 | 420 passed | 420 passed | 420 passed | | Base, 7d8d9b1544 | 4 of 419 failed | 4 of 419 failed | 4 of 419 failed | On the base, the four failures are the cases #6768 reports, on the same rows for every candidate. `expression: arrays_overlap [double]` and `[float]` differ only on the `(0.0, -0.0)` and `(-0.0, 0.0)` pairs. `arrays_overlap.sql` and its native copy fail on `test_overlap_dbl`, where Spark returns `[List(0.0),List(-0.0),true]` and Comet `false`. With this PR all four pass, and `arrays_overlap native variant by Spark version` confirms that `convert` asks for `spark_arrays_overlap_normalized` on each candidate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
