wyatt-herkamp opened a new pull request, #27021: URL: https://github.com/apache/datafusion/pull/27021
## Which issue does this PR close? - Closes #27020. ## Rationale for this change arrow-csv parses columns by position, so when reading multiple CSV files where some file is missing a column (that isn't the last one), values land in the wrong columns or parsing fails. ## What changes are included in this PR? - New `match_columns_by_header` CSV option (config, `CsvFormat`, `CsvSource`, and proto). - When enabled (requires `has_header`), each file's header is read and its columns are mapped to the table schema by name; missing columns are filled via `PhysicalExprAdapterFactory`. Works for both full-file and range (repartitioned) reads. ## Are these changes tested? Yes: unit tests in `datasource-csv/src/source.rs` and `core/src/datasource/physical_plan/csv.rs`, a proto round-trip test, and sqllogictest cases in `csv_files.slt`. ## Are there any user-facing changes? New opt-in `format.match_columns_by_header` option, documented in `format_options.md`. Default behavior is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
