zhuqi-lucas opened a new pull request, #21828: URL: https://github.com/apache/datafusion/pull/21828
## Which issue does this PR close? Closes #19654 ## Rationale for this change `SELECT * FROM table LIMIT 5 OFFSET 59000000` on a 60M-row parquet file takes 182ms because DataFusion reads 59M+ rows then discards them in `GlobalLimitExec`. The parquet reader has no knowledge of the offset, so it cannot skip row groups. ## What changes are included in this PR? Push OFFSET from `GlobalLimitExec` down to `DataSourceExec`/parquet opener: 1. **`FileSource::supports_offset()`** — trait method, parquet returns true, others false 2. **`FileScanConfig.offset`** — new field, only set when source supports offset 3. **`LimitPushdown` optimizer** — pushes offset to DataSourceExec, eliminates `GlobalLimitExec` when accepted 4. **`prune_by_offset`** — skips leading fully-matched RGs whose cumulative rows fall within offset 5. **`RowSelection`** — for partial RG skip (remaining offset within first surviving RG) 6. **`effective_limit`** — decoder reads only `fetch` rows (`limit - offset`) ### Benchmark (60M rows, 1.5GB parquet, single partition) | OFFSET | Before | After | Speedup | |--------|--------|-------|---------| | 0 | 3ms | 2ms | - | | 1M | 4ms | 1ms | **4x** | | 30M | 98ms | **1ms** | **98x** | | 59M | 182ms | **<1ms** | **>182x** | ## Are these changes tested? - **6 unit tests** for `prune_by_offset` (boundary, partial, non-fully-matched, etc.) - **SLT Test N** (10 sub-tests): EXPLAIN, correctness, WHERE interaction, multi-partition, boundary cases - Updated `explain_analyze`, `push_down_filter_parquet`, `limit_pruning` SLTs for new `offset_pruned_row_groups` metric - All `limit.slt` tests pass (CSV/JSON offset still handled by `GlobalLimitExec`) ## Are there any user-facing changes? Performance only — faster OFFSET queries on parquet. No API changes visible to SQL users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
