Anurag-S1ngh opened a new pull request, #4401:
URL: https://github.com/apache/iceberg-python/pull/4401

   ## Summary
   
   Data files written by the PyArrow writer always had an empty 
`nan_value_counts` map, because the counts were never computed. The scan 
planner supports NaN counts, so files written by pyiceberg could never be 
pruned on `is_nan`, `not_nan`, or range filters, while files written by 
parquet-mr could.
   
   ## Changes
   
   - Count the non-null NaNs of the top-level float and double columns from the 
Arrow table the writer receives, since PyArrow does not expose a NaN count on 
its Parquet column statistics.
   - Accumulate the counts across batches and merge them into the 
`DataFileStatistics` returned by `ParquetFormatWriter.close()`.
   - Leave a column out of the map when it cannot be counted, which only means 
the file cannot be pruned on NaN counts.
   
   Nested float and double columns are not counted yet; that needs flattening 
through lists, structs, and maps and can follow separately.
   
   ## Testing
   
   - Added a test asserting the NaN counts of a written file, and one asserting 
`d IS NAN` now prunes a file without NaNs.
   - `make test` passes (4694 passed, 5 skipped).
   
   Closes #4396


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to