sagiba commented on issue #24272:
URL: https://github.com/apache/datafusion/issues/24272#issuecomment-6048319349
We hit this on real data: a sorted file of S3 object keys queried with
`WHERE key >= <prefix> ORDER BY key LIMIT 200`. The result was sorted but
skipped the matching rows in the first, partially matching row group.
Here is an end-to-end reproducer for `datafusion-cli`, run from an empty
directory:
```sql
SET datafusion.execution.target_partitions = 4;
SET datafusion.optimizer.repartition_file_min_size = 0;
SET datafusion.execution.parquet.pushdown_filters = true;
-- 40 row groups of 1,000 rows, k = 0..39999 ascending.
-- This COPY records sorting_columns for k ASC.
COPY (SELECT value AS k FROM generate_series(0, 39999))
TO 't.parquet' STORED AS PARQUET
OPTIONS ('format.max_row_group_size' '1000');
CREATE EXTERNAL TABLE t (k BIGINT)
STORED AS PARQUET LOCATION 't.parquet';
SELECT k FROM t WHERE k >= 500 ORDER BY k LIMIT 3;
-- expected 500, 501, 502
-- actual 1000, 1001, 1002
```
`repartition_file_min_size = 0` enables byte-range splitting for this small
file. The relevant plan is:
```text
SortPreservingMergeExec: [k@0 ASC NULLS LAST], fetch=3
DataSourceExec: file_groups={4 groups: [...]}, projection=[k], limit=3,
output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=k@0 >= 500,
...
```
Results for this reproducer:
- Wrong on `main` at `521ba0efaf` (2026-10-07) and with the Python package
54.1.0.
- Correct with #24273 at `3062d91`.
- Also wrong with `WITH ORDER (k ASC)` on a file written without
`sorting_columns`.
- Correct with any one of `pushdown_filters = false`, `target_partitions =
1`, or `repartition_file_scans = false`.
- On `main`, `datafusion.execution.parquet.pruning = false` alone still
gives the wrong result because page-level limit pruning remains active. Also
setting `datafusion.execution.parquet.enable_page_index = false` restores the
expected result.
This could complement the `.slt` regression in #24273 by covering a single
file split into byte ranges, with ordering inferred from Parquet metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
