sunchao commented on PR #5472:
URL:
https://github.com/apache/datafusion-comet/pull/5472#issuecomment-5456485460
Fixed the remaining infinity-list pruning regression in
be281b7b770534397eda13d8fc6ad30f94f2a02b.
The fast path now accepts infinity literals by checking for non-NaN, nonzero
candidates. Lists containing NaN or either zero sign still normalize. The
change is limited to the two guard checks, their comment, and extensions to the
existing serialization and membership tests.
Fresh full-reactor JVM runs, including formatting checks, passed: 39 tests
on Spark 4.1.3 and 37 on Spark 3.4.3, with two expected version-related
cancellations. The updated serialization tests fail with the previous guard and
pass with this fix.
In a fresh Spark 4.1 native scan, `f IN (1, CAST('Infinity' AS DOUBLE))` now
prunes 15 of 16 row groups and reads 1,583 data bytes instead of 25,328, with
the same result. FLOAT, negative infinity, and InSet also pass; lists of both
infinities prune all 16 groups. Queries over actual infinity values also match
Spark with native row filtering enabled or disabled.
Native code is unchanged; these runs reused the matching native library and
verified the loaded classes/library. CI for this commit is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
