AP24110011695 opened a new issue, #4396:
URL: https://github.com/apache/iceberg-python/issues/4396

   ### PyIceberg version
   main @ fce8185 (2026-11-15)
   
   ### Describe the bug
   
   `data_file_statistics_from_parquet_metadata` returns `nan_value_counts={}` 
unconditionally — the dict is initialized and returned but never populated. 
Data files written by the pyarrow writer (and files imported through 
`parquet_file_to_data_file`) therefore never report NaN counts, while the 
`DataFile` schema and the scan-planning metrics evaluator both fully support 
them.
   
   Probe on a table containing 2 NaNs per column (written with 
`pq.write_table`, metadata read back and passed through the function):
   
   ```
   null_value_counts: {1: 0, 2: 1}      # correct
   nan_value_counts: {}                 # always empty
   # true non-null NaNs: field 1 = 2, field 2 = 2
   ```
   
   **Consequence** — conservative, not wrong results: 
`_InclusiveMetricsEvaluator` uses `nan_counts` for pruning (`visit_is_nan`: 
`nan_counts.get(field_id) == 0 → ROWS_CANNOT_MATCH`; `_contains_nans_only` for 
`not_nan`/`less_than`/`greater_than`). With empty nan counts every check 
degrades to `ROWS_MIGHT_MATCH`, so:
   
   - `is_nan` filters can never prune files, even files with zero NaNs in that 
column are always scanned,
   - `not_nan`/range filters never prune all-NaN or NaN-heavy files (the 
sensor-data and ML-feature use case),
   - files written by parquet-mr/Java *do* carry `nan_value_counts` and get 
that pruning — the gap only affects pyiceberg-written tables.
   
   This looks adjacent to #4313 (drop bounds when a row group has no min/max) 
and #4385 (omit incomplete null counts): the same statistics-completeness 
family, except NaN counts are not partially accumulated — they are never 
accumulated at all.
   
   ### Fix options
   
   pyarrow 26's `ColumnStatistics` exposes no `nan_count`/`has_nan_count`, so a 
row-group-level accumulation mirroring #4385 is not possible with the public 
API today:
   
   **A. Compute at write time** — the writer sees the arrow table before 
`close()`; NaN counts per float column are one vectorized op 
(`pa.compute.is_nan(col)` sums to the non-null NaN count, which matches the 
spec's NaN-count semantics) and can be threaded into 
`data_file_statistics_from_parquet_metadata` (or merged into its returned 
`DataFileStatistics`).
   
   **B. Upstream** — expose `nan_count` from the parquet statistics in pyarrow 
(the thrift `Statistics` message carries it since parquet-format 2.5), then 
mirror the #4385 accumulation including a completeness guard.
   
   Happy to send A as a PR if that design is preferred.
   
   AI disclosure: this report was compiled with AI assistance; every claim 
above is reproducible from the probe snippet and the cited code paths 
(`pyiceberg/io/pyarrow.py` `data_file_statistics_from_parquet_metadata`, 
`pyiceberg/expressions/visitors.py` `_InclusiveMetricsEvaluator`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to