charan-rathore opened a new pull request, #26494: URL: https://github.com/apache/datafusion/pull/26494
## Which issue does this PR close? - Closes #26490. ## Rationale for this change Parquet writers leave NaN out of floating-point min/max statistics (the spec requires it). So a column chunk holding one distinct finite value plus some NaNs has statistics `min == max` and `null_count == 0`, although its values are not constant. The constant-column substitution in the Parquet opener then treats the column as constant and returns the statistic for every NaN row: a plain `SELECT *` silently loses NaNs, and `COPY ... TO` writes the wrong values into the new files. pyarrow, DuckDB and Polars read the same files correctly (see #26490 for a reproducer). ## What changes are included in this PR? `constant_value_from_stats` in `datafusion-datasource-parquet` no longer derives a constant column from `min == max` statistics for `Float16`/`Float32`/`Float64` columns, since float min/max cannot prove the column is constant when NaN is excluded from the statistics. All-null columns (proven by `null_count == num_rows`) are still treated as constant null, and non-float types are unchanged. ## What is the testing strategy for this PR? New unit test `extract_constant_columns_float_nan_unsafe` in `datafusion/datasource-parquet/src/opener/mod.rs` covers Float64 and Float32 columns whose statistics look constant (`min == max`, `null_count == 0`) and asserts no constant is substituted, while the existing `extract_constant_columns_non_null` / `extract_constant_columns_all_null` tests pin the unchanged behavior for other types. The new test fails on current main (it returns the bogus constant) and passes with this change. Full `datafusion-datasource-parquet` lib suite locally: 290 passed, 0 failed (with the `parquet-testing` submodule checked out). ## Are there any user-facing changes? Bug fix only: `SELECT` and `COPY ... TO` on affected Parquet files now return the real NaN values instead of the statistics constant. No API changes. --- AI disclosure: this change was prepared with AI assistance under human direction. The fix and the test were run locally as described above (test confirmed failing on main and passing with the fix; full crate suite green). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
