Rich-T-kid opened a new pull request, #26973: URL: https://github.com/apache/datafusion/pull/26973
## Which issue does this PR close? - Closes #26972. ## Rationale for this change `enable_rle_to_dictionary` promotes a column when any row group has a dictionary page. If a writer later falls back to plain pages, schema adaptation rebuilds dictionaries per batch. ClickBench Q23 spends most of its projection time in that cast. ## What changes are included in this PR? Dictionary promotion now requires page encoding statistics showing that every data page is dictionary encoded. Missing evidence keeps the plain type. When files disagree, directory schema inference also keeps the plain type instead of promoting the other files. The option documentation is updated to describe the narrower behavior. ## What is the testing strategy for this PR? Unit tests cover dictionary-only, mixed, missing-statistics, and missing-dictionary-page metadata, plus mixed-file schema normalization. Validated with: - `cargo test -p datafusion-datasource-parquet --lib` (296 tests) - `cargo test --test sqllogictests -- information_schema` - `cargo clippy --workspace --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` - `./dev/update_config_docs.sh` A controlled end-to-end ClickBench run is in progress on #26964 using the same DataFusion and Arrow revisions on both sides. ## Are there any user-facing changes? With `enable_rle_to_dictionary=true`, columns containing plain fallback pages, or lacking page encoding statistics, remain their plain string or binary type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
