AP24110011695 opened a new pull request, #4412:
URL: https://github.com/apache/iceberg-python/pull/4412

   ## What's changed
   
   Implements design A from #4396: `data_file_statistics_from_parquet_metadata` 
returned an empty `nan_value_counts` dict unconditionally, so data files 
written by the pyarrow writer never reported NaN counts — and NaN-aware scan 
pruning stayed silently unavailable for them (`_InclusiveMetricsEvaluator` 
degrades to `ROWS_MIGHT_MATCH` without nan counts, for `is_nan`, `not_nan` and 
range predicates via `_contains_nans_only`).
   
   pyarrow's statistics API does not expose NaN counts (checked on 26.0.1 — no 
`nan_count`/`has_nan_count`), so the writer now counts them from the written 
table itself:
   
   - `ParquetFormatWriter.write()` accumulates `pa.compute.is_nan(...)` sums 
per top-level float/double field (`is_nan` is null on nulls and `sum` skips 
nulls — matching the spec's NaN-count semantics of counting non-null NaN values)
   - the counts flow through a new optional `table_nan_value_counts` parameter 
of `data_file_statistics_from_parquet_metadata`
   - the metadata-only caller (`parquet_file_to_data_file`, used by 
`add_files`) keeps its previous behavior — no NaN counts from metadata-only 
input
   
   Nested float fields keep the previous (empty) behavior for now; the metrics 
plan and evaluator keys make them a natural follow-up once the top-level path 
is settled.
   
   ## Test
   
   `test_nan_value_counts_are_written` (mixed values + null) and 
`test_nan_value_counts_are_written_across_row_groups` (all-NaN, two `write()` 
calls) — both verified failing-first on main (`nan_value_counts == {}`) and 
passing with the change; the second also pins the pruning win (`not_nan` now 
prunes an all-NaN file via `_contains_nans_only`). Full 
`tests/io/test_pyarrow_stats.py` (22) green; the pre-existing `azure`-fixture 
failures in `test_pyarrow.py` are unrelated (missing optional dep in this 
environment).
   
   AI disclosure: drafted with AI assistance, verified locally by running the 
failing-first cycle.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to