rgbuilds opened a new issue, #26451:
URL: https://github.com/apache/datafusion/issues/26451
### Describe the bug
Parquet statistics pruning can incorrectly discard a row group when a
timestamp column is cast to a time type and the row group crosses midnight.
This changes query results: matching rows are returned when pruning is
disabled but disappear when pruning is enabled.
The issue affects both `Time32(Second)` and `Time64(Nanosecond)` and was
reproduced on current DataFusion `main`:
```text
Commit: 484183dc34ea565190d9cd4a66db6d24789f95d0
DataFusion: 55.1.0
Rust: 1.99.0
```
### To Reproduce
Create a Parquet file with one row group containing these timestamp values:
```text
1970-01-01 23:00:00
1970-01-02 00:00:00
1970-01-02 01:00:00
```
Run predicates equivalent to:
```sql
arrow_cast(ts, 'Time32(Second)')
= arrow_cast('00:00:00', 'Time32(Second)')
```
and:
```sql
arrow_cast(ts, 'Time32(Second)')
< arrow_cast('02:00:00', 'Time32(Second)')
```
Repeat with `Time64(Nanosecond)`.
File-statistics collection and page-index pruning were disabled to isolate
row-group statistics pruning.
### Results
With pruning disabled:
```text
Time32 equality: expected 1, returned 1
Time32 less-than: expected 2, returned 2
Time64 equality: expected 1, returned 1
Time64 less-than: expected 2, returned 2
```
With pruning enabled:
```text
Time32 equality: expected 1, returned 0
Time32 less-than: expected 2, returned 0
Time64 equality: expected 1, returned 0
Time64 less-than: expected 2, returned 0
```
As a control, the same predicates return the expected rows when executed
against an in-memory table containing identical data. The incorrect results
occur only for the Parquet-backed table when row-group pruning is enabled.
### Expected behavior
Enabling Parquet pruning must not change query results.
The equality predicate should return the midnight row. The less-than
predicate should return the midnight and `01:00` rows.
If a cast cannot safely transform min/max statistics, DataFusion should
retain the row group rather than prune it.
### Additional context
The likely cause is `verify_support_type_for_prune` in
`datafusion/pruning/src/pruning_predicate.rs`.
It currently permits a cast when the source and destination are both strings
or both non-strings. This allows timestamp-to-time casts even though they do
not preserve ordering across midnight.
For this row group, the timestamp bounds are:
```text
1970-01-01 23:00:00 → 1970-01-02 01:00:00
```
After casting to time-of-day values, they become:
```text
23:00:00 → 01:00:00
```
Those bounds are no longer ordered. The resulting pruning expression can
therefore reject the entire row group even though it contains matching rows.
I have a standalone reproducer and plan to contribute a conservative fix
that prevents timestamp-to-time casts from being used for statistics pruning,
along with regression tests for `Time32` and `Time64`. A broader allowlist of
order-preserving cast families could be considered as a separate follow-up.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
