Anurag-S1ngh commented on PR #4401:
URL: https://github.com/apache/iceberg-python/pull/4401#issuecomment-6957363577

   Thanks Fokko, all addressed.
   
   - Counts are now added across `write()` calls instead of replaced, with a 
test that writes two batches to one file.
   - The statistics plan is computed once in `__init__` and passed into the NaN 
counter, which skips columns whose metrics mode is `none`. 
`write.metadata.metrics.default=none` now gives an empty `nan_value_counts`. 
pyiceberg does not implement 
`write.metadata.metrics.max-inferred-column-defaults` yet, so there was nothing 
else to skip.
   - Dropped the `try/except` and the `count is not None` check. `pc.is_nan` 
and `pc.sum` do not raise for float or double columns, and `pc.sum` with 
`min_count=0` always returns a valid scalar.
   - The counts are passed into `data_file_statistics_from_parquet_metadata` as 
an optional argument, so the `add_files` path can reuse it later.
   - Added the tests you asked for and updated the merge manifests integration 
expectation.
   
   `add_files` is out of scope here and #4396 stays open; I noted it in the 
description. Thanks for the nested fields note, I will follow up with structs 
counted and fields inside lists and maps left out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to