RussellSpitzer commented on code in PR #18416:
URL: https://github.com/apache/iceberg/pull/18416#discussion_r4557115865


##########
format/spec.md:
##########
@@ -1934,6 +1945,7 @@ Lists must use the [3-level 
representation](https://github.com/apache/parquet-fo
 | **`binary`**       | `binary`                                                
                                                                                
     |                                             |                            
                                    |
 | **`struct`**       | `group`                                                 
                                                                                
     |                                             |                            
                                    |
 | **`list`**         | `3-level list`                                          
                                                                                
     | `LIST`                                      | See Parquet docs for 
3-level representation.                   |
+| **`vector(E, D)`** | `3-level list`                                          
                                                                                
     | `LIST`                                      | The outer group must 
contain a single repeated group named `list`. The `list` group must contain a 
single primitive field named `element`, marked `required` in the Parquet 
schema. The `element` field must use the same Parquet physical type as an 
Iceberg column of type E, following the mapping in this table. Each non-null 
vector must contain exactly D non-null, finite elements. |

Review Comment:
   I would agree on the "must" unless we want to do possible up/down casting 
which I'm beginning to think would just encourage type promotion. I think it's 
probably simpler (if we want to include float16 and treat vector elements as 
sepererate from the Iceberg Type System). 
   
   I would just detach the element completely from the Iceberg Type and define 
it them in their own little table.
   
   ```
   | **`vector(E, D)`** | `group` | `FINITE_VECTOR(D)` | The `element` field 
must use the Parquet type for E listed in the vector element table below. See 
Parquet docs for 
[`FINITE_VECTOR`](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#finite-vectors).
 |
   ```
   
   ```
   Vector elements are stored in Parquet using the following types:
   | Element type  | Parquet physical type     | Logical type |
   |---------------|---------------------------|--------------|
   | **`int`**     | `int32`                   |              |
   | **`long`**    | `int64`                   |              |
   | **`float16`** | `fixed_len_byte_array[2]` | `FLOAT16`    |
   | **`float`**   | `float`                   |              |
   | **`double`**  | `double`                  |              |
   ```
   
   
   We Could do something similar in the Iceberg Vector Definitions above
   ```
   | [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of D 
finite numeric values with element type E | E must be a [vector element 
type](#vector); D must be positive |
   ```
   
   ```
   ##### Vector
   
   A `vector(E, D)` contains exactly D elements of element type E, and every 
element is non-null. The vector value may be null only when the vector field is 
optional.
   
   E must be one of the following vector element types:
   
   | Element type  | Element values                                             
            | Value type |
   
|---------------|------------------------------------------------------------------------|------------|
   | **`int`**     | 32-bit signed integers                                     
            | `int`      |
   | **`long`**    | 64-bit signed integers                                     
            | `long`     |
   | **`float16`** | `float` values that are exactly representable as IEEE 754 
binary16     | `float`    |
   | **`float`**   | 32-bit IEEE 754 floating point                             
            | `float`    |
   | **`double`**  | 64-bit IEEE 754 floating point                             
            | `double`   |
   
   Vector element types are valid only inside a `vector`; `float16` is not a 
primitive type and cannot be used as the type of a field. Element values follow 
the rules of the value type, including equality and JSON single-value 
serialization. Floating-point elements must be finite: `NaN`, `+Infinity`, and 
`-Infinity` are invalid.
   
   A `vector` must not be promoted to another type. No type other than 
`unknown` may be promoted to `vector`. The element type and dimension must not 
change. A `list` of element type E is not a `vector(E, D)`, even if every list 
instance contains exactly D elements.
   
   Equality of vector values is element-wise, using the equality rules of the 
value type.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to