PFischbeck commented on code in PR #18416: URL: https://github.com/apache/iceberg/pull/18416#discussion_r4220849245
########## format/spec.md: ########## @@ -295,6 +297,12 @@ For details on how to serialize a schema to JSON, see Appendix C. [1001]: <https://portal.ogc.org/files/?artifact_id=25355> "OGC Simple feature access" +##### Vector + +Every non-null `vector(E, D)` value must contain exactly D non-null elements of type E. Floating-point elements must not be `NaN`, positive infinity, or negative infinity. The vector value itself may be null when the containing field is optional. + +No type promotion to or from `vector` is defined, including changes to its element type or dimension. Lower and upper bounds are not defined for vector values. Review Comment: Concerning lower/upper bounds, we should also adjust the table in the "Field Statistics" section to mention vectors are excluded. ########## format/spec.md: ########## @@ -1888,6 +1901,7 @@ The binary single-value serialization can be used to store the lower and upper b | **`binary`** | **`JSON string`** | `"000102ff"` | Stored as a hexadecimal string | | **`struct`** | **`JSON object by field ID`** | `{"1": 1, "2": "bar"}` | Stores struct fields using the field ID as the JSON field name; field values are stored using this JSON single-value format | | **`list`** | **`JSON array of values`** | `[1, 2, 3]` | Stores a JSON array of values that are serialized using this JSON single-value format | +| **`vector(E, D)`** | **`JSON array of values`** | `[1.0, 2.0, 3.0]` | Stores exactly D values using E's JSON single-value format | Review Comment: What would this be used for? ########## format/spec.md: ########## @@ -295,6 +297,12 @@ For details on how to serialize a schema to JSON, see Appendix C. [1001]: <https://portal.ogc.org/files/?artifact_id=25355> "OGC Simple feature access" +##### Vector + +Every non-null `vector(E, D)` value must contain exactly D non-null elements of type E. Floating-point elements must not be `NaN`, positive infinity, or negative infinity. The vector value itself may be null when the containing field is optional. Review Comment: Are you suggesting we might only allow one representation of the number zero? If yes, do you have use-cases in mind? ########## format/spec.md: ########## @@ -1507,6 +1515,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`binary`**|`bytes`|| |**`struct`**|`record`|| |**`list`**|`array`|| +|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E and must not be null. Readers and writers must enforce the vector dimension and element requirements.| Review Comment: Would it make sense to add an Avro logical type for vector? ########## format/spec.md: ########## @@ -1594,6 +1604,7 @@ When reading an `unknown` column, any corresponding column must be ignored and r | **`binary`** | `binary` | | | | **`struct`** | `struct` | | | | **`list`** | `array` | | | +| **`vector(E, D)`** | `array` | | Array elements must use the ORC mapping for E and must not be null. Readers and writers must enforce the vector dimension and element requirements. | Review Comment: I'm not too familiar with ORC, but does it make sense to also add new ORC type attributes to mark it as vector? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
