cetra3 opened a new pull request, #26405: URL: https://github.com/apache/datafusion/pull/26405
## Which issue does this PR close? - Related to #21968. There is no dedicated issue for this change. ## Rationale for this change Building a pruning predicate for a Parquet file with a wide schema takes time proportional to the number of columns in the file, once for every comparison in the predicate, even when the predicate only references one column. A Parquet scan builds several pruning predicates for each file it opens (for example for row group statistics, the page index and bloom filters), so on wide tables this cost is paid many times per file. The cause is in `PruningExpressionBuilder::try_new`. For every comparison it converts the whole file schema into a `DFSchema`, which runs `DFSchema::check_names` and inserts every field name into a `BTreeSet`. The `DFSchema` is then only used through `schema.as_arrow()`, to get the data type of the expression inside a `CAST` or `TRY_CAST`. That is the Arrow schema the builder already has. In a profile of `wide_schema` Q01 (1024 columns, 256 files), `DFSchema::try_from` called from `PruningExpressionBuilder::try_new` was about 5% of CPU samples. With a file metadata cache large enough to hold every footer it was 9.8%. The `DFSchema` dates from #3422, when pruning rewrote logical `Expr`s and needed a `DFSchema` to resolve their types. #5419 moved pruning to physical expressions, which only need the Arrow schema, but kept the `DFSchema` parameter. ## What changes are included in this PR? - `rewrite_expr_to_prunable` and `rewrite_cast_child_to_prunable` take `&Schema` instead of a `DFSchema`. - `PruningExpressionBuilder::try_new` passes the schema it already has, rather than building a `DFSchema` for each comparison. - Two unit tests that built a `DFSchema` only to call `rewrite_expr_to_prunable` now pass the `Schema`. ## What is the testing strategy for this PR? The change is covered by existing tests, so this PR adds none. Both functions are private, and their behaviour is unchanged apart from the duplicate-name case described below. - `datafusion-pruning` tests, including `test_rewrite_expr_to_prunable` and `test_rewrite_expr_to_prunable_error` - `datafusion-datasource-parquet` lib tests - `datafusion` `parquet_integration` tests, which cover row group and page index pruning ## Are there any user-facing changes? There are no API changes, because both functions are private. There is one behaviour change. `DFSchema::try_from` fails when the schema has duplicate field names, so previously no comparison against such a schema could be used for pruning. Now pruning goes ahead and resolves the column by name to the first field with that name. The column lookup in the same function (`schema.column_with_name`) and the Parquet statistics lookup (`parquet_column`) already resolve names this way. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
