jayzhan211 opened a new issue, #26457:
URL: https://github.com/apache/datafusion/issues/26457
### Describe the bug
A file scan with an output ordering (`WITH ORDER`, or ordering inferred from
Parquet sorting metadata) keeps the file groups that `ListingTable` builds with
`split_files`. Those groups have the same number of files, not the same size.
Unordered scans are cut into size-balanced byte ranges by
`FileGroupPartitioner::repartition_evenly_by_size`, but ordered scans only
split single-file groups, and do nothing once there are `target_partitions`
groups.
With a few large files next to many small ones, one partition ends up with
most of the data and sets the wall time of every query over the table,
including queries that never use the ordering.
### To Reproduce
16 sorted Parquet files: `f00`-`f03` have 8M rows, `f04`-`f15` have 0.5M
rows. `target_partitions = 8`.
```sql
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'data/' WITH ORDER (a
ASC);
EXPLAIN SELECT sum(a + b) FROM t WHERE b > 0.5;
-- file_groups={8 groups: [[f00, f01], [f02, f03], [f04, f05], ...]} two
16M-row partitions
```
Without `WITH ORDER` (files without sorting metadata) the same query gets
`[[f00:0..43M], [f00:43M.., f01:0..14M], [f01:14M..57M], ...]`.
| Query | ordered | not ordered |
|---|---|---|
| `SELECT sum(a + b) FROM t WHERE b > 0.5` | 233 ms | 93 ms |
| `SELECT * FROM t WHERE b > 0.999999` | 32 ms | 13 ms |
Work stealing does not close the gap: with `enable_file_stream_work_stealing
= false` the unordered table still runs in about 91 ms.
### Expected behavior
An ordered scan should use the same size-balanced split whenever every new
group is still sorted. Cutting the concatenated groups into contiguous byte
ranges keeps the files in order; it only moves the group boundaries.
### Additional context
Found while reviewing the plain-`LIMIT` cost of #26434 with @adriangb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
