nastra commented on PR #17609: URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5315974742
@xndai in https://github.com/apache/iceberg/pull/17413 we concluded that we need to always track `null_value_count`. Would you like to re-open this PR and update the spec accordingly? I think the diff should look something like this: ``` --- a/format/spec.md +++ b/format/spec.md @@ -822,10 +822,12 @@ Each stats struct holds statistics for one table field. It may contain the follo | _optional_ | 2 | `upper_bound` | Field type or `geo_upper` | all primitives or `variant` | Upper bound stored as the field's type, or `geo_upper` for geo types | | _optional_ | 3 | `tight_bounds` | `boolean` | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values | | _optional_ | 4 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) | -| _optional_ | 5 | `null_value_count` | `long` | optional fields | Number of null values in the column | +| _optional_ | 5 | `null_value_count` | `long` | all | Number of null values in the column | | _optional_ | 6 | `nan_value_count` | `long` | `float`, `double` | Number of NaN values in the column | | _optional_ | 7 | `avg_value_size_in_bytes` | `int` | `string`, `binary`, `variant`, `geometry`, `geography` | Avg value size in memory (uncompressed) in bytes over non-null values to estimate memory consumption | +`null_value_count` is included for `required` and `optional` fields because a field that is `required` in the table schema is **null** in rows where an `optional` parent struct is **null**. + For example, stats for a `required` `int` field named `id` with field-id `2` are stored using: @@ -834,8 +836,8 @@ For example, stats for a `required` `int` field named `id` with field-id `2` are 10_402: optional int upper_bound; // type matches the field type (int) 10_403: optional boolean tight_bounds; 10_404: optional long value_count; + 10_405: optional long null_value_count; - // null_value_count is only used for optional fields // nan_value_count is only used for float and double // avg_value_size_in_bytes is only used for variable length types } @@ -902,6 +904,7 @@ For example, stats for a table with a required int, `id`, and an optional string 10_402: optional int upper_bound; 10_403: optional boolean tight_bounds; 10_404: optional long value_count; + 10_405: optional long null_value_count; } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
