jayzhan211 opened a new pull request, #26458: URL: https://github.com/apache/datafusion/pull/26458
## Which issue does this PR close? - Closes #26457. ## Rationale for this change A scan with a declared ordering keeps the count-based file groups from `split_files`, while an unordered scan is cut into size-balanced byte ranges. With uneven files, one partition of an ordered scan gets most of the data and every query over the table waits for it (see the issue: 233 ms ordered vs 93 ms unordered for the same full scan). ## What changes are included in this PR? `FileScanConfig::repartitioned`: when the scan has an output ordering, first ask the file source for the split it would use without an ordering (the size-balanced split, which keeps files in group order and only moves the group boundaries). Use it only if `validated_output_ordering()` is unchanged, that is, statistics prove every new group is still sorted. Otherwise fall back to the current order-preserving split. - Groups from `split_groups_by_statistics` interleave value ranges, so the re-cut fails validation and they are kept as they are. - Files without statistics cannot be proven sorted in a multi-file group, so they also keep today's behavior. - File sources that cannot split files (compressed files, Arrow IPC stream) return `None` and keep today's behavior. ## What is the testing strategy for this PR? - `repartition_balances_ordered_groups`: groups `[[8000 B, 8000 B], [1000 B, 1000 B]]` become four 4500 B groups and keep the ordering. - `repartition_keeps_ordered_groups_when_rebalancing_breaks_order`: overlapping files across the old boundary keep the old groups. Fails if the validation check is removed. - `repartition_scan.slt`: the ordered plan now has the same file groups as the unordered plan above it; results unchanged. - Extended test suite passes (12,730 tests). `parallelization_multiple_files` snapshot updated: at 8 partitions each group is still one range of one file, only the group order changes; the 3-partition case keeps its old groups because the balanced split would put two files without statistics in one group. Benchmarks (release-nonlto, median of 5 interleaved runs, ms). 16 sorted files (4 x 8M rows, 12 x 0.5M), `WITH ORDER (a ASC)`, `target_partitions = 8`: | Query | pushdown_filters | main | this PR | |---|---|---|---| | `SELECT sum(a + b) FROM t WHERE b > 0.5` | off / on | 228 / 304 | 78 / 105 | | `SELECT * FROM t WHERE b > 0.999999` | off / on | 31 / 29 | 12 / 11 | | `SELECT a, count(*) FROM t WHERE b > 0.99 GROUP BY a ORDER BY a LIMIT 5` | off / on | 103 / 102 | 74 / 74 | | `SELECT * FROM t WHERE b > 0.999999 LIMIT 10` | off / on | 11 / 3 | 8 / 4 | | `SELECT * FROM t WHERE b > 0.999 ORDER BY a LIMIT 10` | off / on | 4 / 4 | 4 / 5 | | `SELECT * FROM t ORDER BY a LIMIT 10` | off / on | 3 / 3 | 3 / 3 | TPC-H SF10 `lineitem` (12 even files) declared `WITH ORDER (l_orderkey)`, 5 queries x pushdown on/off, two interleaved rounds: all within noise (for example `ORDER BY l_orderkey LIMIT 10`: 19-20 vs 18 ms; `WHERE l_shipdate > '1998-11-25' ORDER BY l_orderkey LIMIT 10`: 35-37 vs 35-36 ms). Worst case: when the old groups were already balanced, the new split moves the boundaries and can leave a small byte range of a file in a group (the `repartition_scan.slt` plan has `2.parquet:0..6`), as unordered scans already do. ## Are there any user-facing changes? EXPLAIN output for ordered file scans shows byte-range file groups more often. No API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
