yyanyy commented on code in PR #18416:
URL: https://github.com/apache/iceberg/pull/18416#discussion_r4545245809


##########
format/spec.md:
##########
@@ -282,6 +283,7 @@ Supported primitive types are defined in the table below. 
Primitive types added
 |                  | **`uuid`**         | Universally unique identifiers       
                                    | Should use 16-byte fixed                  
       |
 |                  | **`fixed(L)`**     | Fixed-length byte array of length L  
                                    |                                           
       |
 |                  | **`binary`**       | Arbitrary-length byte array          
                                    |                                           
       |
+| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite 
numeric values with element type E and dimension D | E must be `int`, `long`, 
`float`, or `double`; D must be positive |

Review Comment:
   Following some research and implementation feedback from 
https://github.com/apache/iceberg/pull/18676/changes, I’m leaning toward 
keeping `vector(E, D)` as an Iceberg primitive.
   
   The implementation in [#18676](https://github.com/apache/iceberg/pull/18676) 
strengthens that preference. `VectorType` fits into the existing primitive-type 
machinery, with `E` and `D` as type parameters and the existing schema 
serialization path handling the type. A vector field has its normal field ID, 
with no additional element field ID. This seems natural for a single schema 
value, similar to `fixed(L)`.
   
   Other databases provide precedents for both approaches:
   
   - Oracle explicitly classifies VECTOR as a [scalar type family in 
PL/SQL](https://docs.oracle.com/en/database/oracle/oracle-database/26/lnpls/sql-data-types.html).
   - [pgvector defines vector as a PostgreSQL base 
type](https://github.com/pgvector/pgvector/blob/master/sql/vector.sql), which 
PostgreSQL describes as a [base/scalar 
type](https://www.postgresql.org/docs/current/sql-createtype.html#SQL-CREATETYPE-BASETYPE).
   - Cassandra includes VECTOR among [native 
types](https://cassandra.apache.org/doc/stable/cassandra/developing/cql/types.html),
 separately from collections.
   - On the other side, DuckDB uses fixed-size arrays, explicitly classified as 
[nested types](https://duckdb.org/docs/2.0/sql/data_types/overview), and 
BigQuery’s 
[VECTOR_SEARCH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#vector_search)
 uses `ARRAY<FLOAT64>`.
   - ClickHouse uses numeric arrays for [conventional vector 
search](https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/annindexes),
 alongside its specialized [QBit 
type](https://clickhouse.com/docs/reference/data-types/qbit).
   
   The database conventions vary. For Iceberg, the schema model and the 
straightforward API integration demonstrated in #18676 seem like good reasons 
to retain the primitive classification.
   
   For now I'll keep the PR to treat vector as a primitive type, and we can 
discuss this in next sync meeting. Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to