jayzhan211 opened a new pull request, #26982:
URL: https://github.com/apache/datafusion/pull/26982
## Which issue does this PR close?
- Closes #26981.
- Related to #26451, the same cause for timestamp → time (fixed on its own
by #26452).
## Rationale for this change
Parquet statistics pruning can drop row groups that contain matching rows
when the predicate casts the column. With `i` in (-1, 0, 1, 255, 256) and
timestamps at 23:00, 00:00 and 01:00 across midnight, each in one row group:
| Query | In-memory table | Parquet |
|---|---|---|
| `SELECT count(*) FROM t WHERE CAST(i AS BOOLEAN) <= false` | 1 | 0 |
| `SELECT count(*) FROM t WHERE CAST(i AS BYTEA) < X'05'` | 3 | 0 |
| `SELECT count(*) FROM t WHERE CAST(ts AS TIME) < TIME '02:00:00'` | 2 | 0 |
Pruning casts the min/max statistics and compares them with the literal,
which is only correct when the cast keeps the order of the values. Pruning
decides that with its own check: any cast is allowed unless exactly one side is
a string. So int → boolean, int → binary (little-endian bytes), timestamp →
time of day and named-timezone timestamp → date all pass, although none of them
keep order.
Sort elimination already asks the same question through
`is_order_preserving_cast`, which rejects these casts. The two checks disagree
in both directions: pruning accepts the casts above, and
`is_order_preserving_cast` rejects casts that do keep order and that pruning
relies on today (integer ↔ date/timestamp/duration, string and binary
re-encodings, naive → named-timezone timestamps).
## What changes are included in this PR?
1. `is_order_preserving_cast` (now public in
`datafusion_physical_expr::expressions`) also accepts the casts that keep order
and that only pruning knew about:
- string ↔ string and binary ↔ binary re-encodings (including
`FixedSizeBinary`)
- numbers ↔ temporal types stored as a count (date, timestamp, time,
duration)
- date → timestamp with a timezone, and naive timestamp → named-timezone
timestamp. Arrow reads these as local times and rejects local times that a
transition skips or repeats; the local times that remain keep their order in
UTC.
2. Pruning (`verify_support_type_for_prune`) asks `is_order_preserving_cast`
instead of its own string check.
3. A test that checks `is_order_preserving_cast` against the Arrow cast
kernel: every pair it accepts must keep the order of sample values that convert
(including values around midnight, a DST gap and overlap, and a timezone that
moved the date backwards). A positive control asserts the samples expose the
casts the rule rejects.
Pruning keeps every other cast it used before (compared over all castable
pairs of the numeric, string, binary and temporal types) and stops using:
| Cast | Why |
|---|---|
| integer/float → boolean | -1 and 1 are true, 0 is false |
| integer → binary | little-endian bytes do not sort like the integers |
| timestamp → time | wraps at midnight |
| named-timezone timestamp → date | a transition can move the local date
backwards |
| `Time64` → `Time32`, `Time64(µs)` → `Time64(ns)` | Arrow does not check
these for overflow, so values can wrap (existing rule) |
| `Null` → any, any → list/struct, interval casts | the rule has no entry
for them |
## What is the testing strategy for this PR?
- `test_order_preserving_cast_matches_cast_kernel` (physical-expr): kernel
check of the rule, with a positive control. Making `Timestamp(tz) → Date32`
return `true` fails it with `562129259 -> 1987-10-25, but 562129260 ->
1987-10-24`.
- `row_group_predicate_cast_must_preserve_order` (pruning): casts that must
not prune and casts that must keep pruning, for both `CAST` and `TRY_CAST`.
- `push_down_filter_parquet.slt`: the three queries above against an
in-memory table and Parquet, plus an `EXPLAIN` showing a naive timestamp
compared with a `TIMESTAMPTZ` under a named session timezone still prunes.
- `cast.slt`: an `EXPLAIN` showing the sort on `arrow_cast(i,
'Timestamp(Second, None)')` is removed when the input is ordered by `i`.
- The new pruning unit test and the three Parquet queries fail with the old
pruning check; the `cast.slt` case fails without the widened rule.
## Are there any user-facing changes?
- Queries that filter Parquet through the casts in the table above return
the correct rows; those predicates no longer prune row groups.
- Sort elimination now treats the newly accepted casts as order-preserving
under `CAST` (not `TRY_CAST`), so a sort on, for example, `CAST(int_col AS
TIMESTAMP)` over input ordered by `int_col` can be removed.
- New public function
`datafusion_physical_expr::expressions::is_order_preserving_cast` (addition
only).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
