xndai commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5318588701
> @xndai in #17413 we concluded that we need to always track
`null_value_count`. Would you like to re-open this PR and update the spec
accordingly? I think the diff should look something like this:
>
> ```
> --- a/format/spec.md
> +++ b/format/spec.md
> @@ -822,10 +822,12 @@ Each stats struct holds statistics for one table
field. It may contain the follo
> | _optional_ | 2 | `upper_bound` | Field type or
`geo_upper` | all primitives or `variant` | Upper bound
stored as the field's type, or `geo_upper` for geo types |
> | _optional_ | 3 | `tight_bounds` | `boolean`
| all primitives except for `geometry` and `geography` | When true,
`lower_bound` and `upper_bound` must be equal to the min and max values |
> | _optional_ | 4 | `value_count` | `long`
| all | Number of values in the
column (including null and NaN values) |
> -| _optional_ | 5 | `null_value_count` | `long`
| optional fields | Number of null values in
the column |
> +| _optional_ | 5 | `null_value_count` | `long`
| all | Number of null values in
the column |
> | _optional_ | 6 | `nan_value_count` | `long`
| `float`, `double` | Number of NaN values in
the column |
> | _optional_ | 7 | `avg_value_size_in_bytes` | `int`
| `string`, `binary`, `variant`, `geometry`, `geography` | Avg value size
in memory (uncompressed) in bytes over non-null values to estimate memory
consumption |
>
> +`null_value_count` is included for `required` and `optional` fields
because a field that is `required` in the table schema is **null** in rows
where an `optional` parent struct is **null**.
> +
> For example, stats for a `required` `int` field named `id` with field-id
`2` are stored using:
>
>
> @@ -834,8 +836,8 @@ For example, stats for a `required` `int` field named
`id` with field-id `2` are
> 10_402: optional int upper_bound; // type matches the field type (int)
> 10_403: optional boolean tight_bounds;
> 10_404: optional long value_count;
> + 10_405: optional long null_value_count;
>
> - // null_value_count is only used for optional fields
> // nan_value_count is only used for float and double
> // avg_value_size_in_bytes is only used for variable length types
> }
> @@ -902,6 +904,7 @@ For example, stats for a table with a required int,
`id`, and an optional string
> 10_402: optional int upper_bound;
> 10_403: optional boolean tight_bounds;
> 10_404: optional long value_count;
> + 10_405: optional long null_value_count;
> }
> ```
Yes, I just re-open and push a new diff following your suggestion and remove
the `nullable` term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
