Rich-T-kid commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5285046082
@adriangb your comments are addressed in https://github.com/apache/datafusion/pull/24227/commits/8bc183beaf0a59fcd5566598ee436910adb54c7f Previously the promotion only applied when the schema was inferred via `CREATE EXTERNAL TABLE` (no explicit columns). Explicit schemas (`CREATE EXTERNAL TABLE (col VARCHAR)`) and custom `TableProvider` implementations never touched `infer_schema` so the flag had no effect on those paths. `ParquetSource::with_table_parquet_options` now promotes all string/binary fields to `Dictionary(Int32, …)` at planning time regardless of how the schema was established, since every `ParquetSource` passes through this method. To keep planning and execution in sync, the opener also rewrites the physical read schema so arrow-rs produces dict arrays. Without the planning-time promotion, downstream operators (FilterExec, AggregateExec) would be compiled against `Utf8` while the scan produces `Dictionary(Int32, Utf8)` — a type mismatch at execution time. ## test Added tests covering: - Unit tests in `datasource-parquet`: flag propagation, RLE-encoded and plain-encoded files with the flag on/off - Integration tests in `core/tests/parquet/schema_coercion.rs`: scan → filter → project using `ParquetSource` directly (no `infer_schema`) to validate the explicit-schema path; end-to-end scan → filter → aggregate via `SessionContext` and SQL to validate the full planning pipeline - SLT (`parquet_rle_to_dictionary.slt`): `arrow_typeof` assertions with the flag on/off, GROUP BY over dict columns to confirm aggregation works correctly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
