alamb opened a new pull request, #27022: URL: https://github.com/apache/datafusion/pull/27022
## Which issue does this PR close? - Follow on to #26897 > [!NOTE] > This PR is stacked on #26897. The new tests are in the last commit: e85a89fc00 ## Rationale for this change #26897 adds an instrumented object store that shows the exact storage requests made by a Parquet scan. This PR uses the same test setup to show what I/O several pruning optimizations save, so changes to the parquet reader's I/O pattern are visible in review. ## What changes are included in this PR? New tests in `datafusion/core/tests/parquet/io/`, all using a shared file with three row groups (`three_row_group_file`): | row group | id | val | |-----------|---------|-----------| | 0 | 1, 2, 3 | 0, 5, 10 | | 1 | 4, 5, 6 | 0, 7, 10 | | 2 | 7, 8, 9 | 0, 5, 10 | 1. `row_group_pruning_io.rs`: the same file with different predicates on `id` (none, `id = 5`, `id > 5`, `id > 100`), showing that pruned row groups are never fetched. 2. `filter_pushdown_io.rs`: `val = 7` (which statistics cannot prune) with `pushdown_filters` off and on. With pushdown on, `val` is fetched for every row group, but `id` is fetched only for row group 1. 3. `topk_io.rs`: `ORDER BY id DESC LIMIT 2` with TopK dynamic filter pushdown on and off. With it on, sort pushdown reads row group 2 first, and the dynamic filter then prunes row groups 1 and 0 before they are fetched. With it off, all three row groups are fetched. ## What is the testing strategy for this PR? This PR only adds tests. I ran the TopK tests 15 times locally to check that they don't depend on timing (they use `target_partitions = 1`). ## Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
