Gabriel39 opened a new pull request, #65493:
URL: https://github.com/apache/doris/pull/65493
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
FileScannerV2 treated Parquet FLOAT and DOUBLE min/max statistics containing
NaN as valid. Doris floating-point comparisons can then interpret NaN as equal
to predicate bounds, allowing range or inequality predicates to incorrectly
prune row groups or pages that still contain matching finite values. The same
invalid statistics could also affect min/max aggregate pushdown.
This change validates native Parquet min/max bounds before decoding them. If
either floating-point bound is NaN, the footer or ColumnIndex min/max is
ignored and pruning conservatively keeps the data. Null-count metadata remains
available, finite FLOAT/DOUBLE statistics retain their existing behavior, and
non-floating-point statistics are unchanged.
### Release note
Fix missing rows when Parquet floating-point min/max statistics contain NaN.
### Check List (For Author)
- Test: Unit Test
- `ParquetStatisticsTransformTest.IgnoresNaNFloatAndDoubleMinMax`
-
`ParquetStatisticsTransformTest.IgnoresNaNFloatAndDoubleColumnIndexMinMax`
- Existing finite min/max conversion regression test
- Behavior changed: Yes. NaN floating-point min/max statistics are ignored
for pruning.
- Does this need documentation: No
Validation on the designated Linux build host:
- `build-support/check-format.sh`
- Targeted BE unit tests: 3 tests passed
- `run-clang-tidy.sh` was attempted; analysis is blocked by pre-existing
diagnostics in the source file and `jni-util.h` toolchain static assertions
unrelated to this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
