mrhhsg opened a new pull request, #67768:
URL: https://github.com/apache/doris/pull/67768
### What problem does this PR solve?
Issue Number: None
Problem Summary:
`-0.0` and `+0.0` are equal floating point values (and Doris also treats all
NaN payloads as
one value), but their bit patterns differ. The aggregation operators already
collapse such
group-by keys with `replace_float_special_values()` before hashing, while
the hash join and
the INTERSECT / EXCEPT operators hashed and compared the raw bits. As a
result a `-0.0` row
never matched a `+0.0` build key: `EXISTS` / `IN` subqueries and equivalent
joins silently
dropped the row although `x = 0.0` on the same table returned true.
Two more paths hashed raw float bits and would still hide the row even with
normalized keys:
- the shuffle / spill partitioners (`ColumnVector::update_crcs_with_value` /
`update_crc32c_batch`) sent `-0.0` and `+0.0` to different instances or
partitions;
- the bloom runtime filter (`HashCRC32<Float64>` through
`fixed_len_to_uint32_v2`) rejected
`-0.0` probe rows for a filter built from `+0.0`.
Fix:
- add `normalize_float_hash_key()`, which collapses `-0.0` / NaN payloads in
place for key
columns whose type contains FLOAT / DOUBLE (including nested array / map /
struct leaves) and
leaves every other key column untouched, and call it for the build and
probe keys of the
hash join and set operators. Like the aggregation keys, a `-0.0` join key
is emitted as
`+0.0`.
- normalize float values in `HashCRC32<Float32/Float64>` and in the crc32 /
crc32c hashes of
`ColumnVector`, so bloom filters and shuffle / spill partitions agree with
the equality
semantics. Ordinary values keep their previous hash.
- drop the repeated default argument of `clear_blocks()` in
`async_result_writer.cpp`: the
new source file changes the unity batches and the duplicate default
argument clashed with
the declaration in `local_exchanger.h`.
### Release note
None
### Check List (For Author)
- Test:
- Unit Test: `HashFloatTest.*`, `HashKeyNormalizeTest.*`,
`HashJoinProbeOperatorTest.InnerJoinFloatSpecialValueKeys`,
`IntersectOperatorTest/ExceptOperatorTest.test_float_signed_zero_key`,
`BloomFilterFuncTest.*`,
`ColumnVectorTest.float_special_values_hash_alike`
- Regression test: `query_p0/join/test_join_float_signed_zero`
- Behavior changed: Yes. `-0.0` rows now match `+0.0` keys in hash joins,
semi/anti joins,
IN / EXISTS subqueries, INTERSECT / EXCEPT and bloom runtime filters; a
`-0.0` join key is
output as `+0.0`, consistent with GROUP BY.
- Does this need documentation: No
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
https://claude.ai/code/session_016A7UJu7EA7j4NkGz3yjkt6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
