manuzhang opened a new pull request, #25802: URL: https://github.com/apache/datafusion/pull/25802
## Which issue does this PR close? - Closes #25801. ## Rationale for this change A table can declare a string type for a column that a Parquet file stores as unannotated `BINARY`, for example with `binary_as_string`, or when an engine such as Apache Spark does its own schema inference. The opener then always tells the Parquet reader to decode the column into a string array. The reader only checks UTF-8 for columns with the `UTF8` annotation, so invalid bytes come back as an invalid `StringArray` in release builds and panic in debug builds. Callers can't choose a different conversion, because the coercion happens before the `PhysicalExprAdapter` runs. This PR adds an opt-out so that such columns are read as binary and converted by the adapter's cast. With the default adapter the cast rejects invalid UTF-8, and a custom adapter can replace it. Apache DataFusion Comet uses this to apply Spark's U+FFFD replacement while keeping the table schema, and therefore filter pushdown, typed as strings (apache/datafusion-comet#6246). ## What changes are included in this PR? - **New option.** `datafusion.execution.parquet.coerce_binary_to_string`, default `true`, so existing behaviour is unchanged. When it is `false`, the opener leaves binary file columns as binary. - **New public function.** `apply_file_schema_type_coercions_with_options` is `apply_file_schema_type_coercions` with the flag exposed. The flag only affects the binary to string cases, at the top level and in struct and list children. The regular to view type coercions still apply. - **Plumbing.** `ParquetSource` passes the option to the opener. `ParquetOptions` gets protobuf field 39, and the generated code is regenerated. `information_schema.slt` and `configs.md` are updated. Statistics collection in `metadata.rs` still coerces. It only feeds planning, and arrow-rs already drops min/max values that are not valid UTF-8. With the option off, statistics and bloom filter pruning don't apply to predicates on these columns, because `PruningPredicate` doesn't prune through a cast between binary and string types. That trade-off is why the default stays `true`. The protobuf field is a plain `bool`, like `enable_page_index`, so a plan serialized before this change deserializes with `false`. I can switch it to a `oneof` if reviewers prefer. ## What is the testing strategy for this PR? - `schema_coercion::tests::binary_to_string_coercion_can_be_disabled`: with the flag off, binary columns stay binary at the top level and in struct and list children, and the view coercion still applies. The default still coerces. - `opener::test::test_coerce_binary_to_string_disabled_casts_in_adapter`: with the option off, an unannotated binary column is read into a `Utf8` table column through the adapter's cast, and invalid UTF-8 fails with a cast error instead of producing an invalid string array. - `information_schema.slt` covers the new option's default and description. ## Are there any user-facing changes? There's a new config option, `datafusion.execution.parquet.coerce_binary_to_string`, which defaults to `true`, and a new public function, `apply_file_schema_type_coercions_with_options`. Nothing changes unless the option is turned off. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
