jayzhan211 opened a new issue, #26981: URL: https://github.com/apache/datafusion/issues/26981
### Describe the bug Parquet statistics pruning drops row groups that contain matching rows when the predicate casts an integer column to `BOOLEAN` or `BYTEA`. ### To Reproduce ```sql CREATE TABLE s(i INT) AS VALUES (-1), (0), (1), (255), (256); COPY s TO '/tmp/i.parquet' STORED AS PARQUET; CREATE EXTERNAL TABLE p STORED AS PARQUET LOCATION '/tmp/i.parquet'; ``` | Query | Actual (Parquet) | Expected (same as the in-memory table `s`) | |---|---|---| | `SELECT count(*) FROM p WHERE CAST(i AS BOOLEAN) <= false` | 0 | 1 | | `SELECT count(*) FROM p WHERE CAST(i AS BYTEA) < X'05'` | 0 | 3 | ### Cause Pruning casts the row group's min/max and compares them with the literal, which is only correct for casts that keep the order of the values. `verify_support_type_for_prune` in `datafusion/pruning/src/pruning_predicate.rs` allows any cast unless exactly one side is a string. Int → boolean (-1 and 1 are true, 0 is false) and int → binary (little-endian bytes) don't keep order, so the bounds -1 and 256 become `true`/`true` and `X'FFFFFFFF'`/`X'00010000'`, and the row group is pruned. #26451 is the same cause for timestamp → time. ### Expected behavior Parquet returns the same rows as the in-memory table. I have a fix ready that makes pruning use `is_order_preserving_cast`, the rule sort elimination already uses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
