zhuqi-lucas commented on PR #21828:
URL: https://github.com/apache/datafusion/pull/21828#issuecomment-4350744475
Thanks @alamb for the great discussion and testing!
**Update: multi-partition issue fixed**
The optimization now works on large files with byte-range partitioning. The
key fix was pushing offset through `CoalescePartitionsExec` to its
`DataSourceExec` child, using a shared `Arc<AtomicUsize>` counter so all
partitions coordinate offset consumption atomically.
Testing on the same 14GB `hits.parquet`:
```
> select * from 'hits.parquet' OFFSET 99000000 LIMIT 5;
....
5 row(s) fetched.
Elapsed 0.002 seconds.
```
```
> explain select * from 'hits.parquet' OFFSET 99000000 LIMIT 5;
CoalescePartitionsExec: fetch=99000005
DataSourceExec: file_groups={12 groups: [...]}, limit=99000005,
offset=99000000
-- No GlobalLimitExec needed
```
**On semantics**
I agree with you and @asolimando that `OFFSET` without `ORDER BY` has no
guaranteed ordering per SQL standard. This PR does **not** change the semantics
— it produces exactly the same results as the current `GlobalLimitExec` skip
(physical row order). The only difference is we skip rows at the parquet reader
level (RG pruning + RowSelection) instead of reading and discarding them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
