yyanyy commented on code in PR #18416:
URL: https://github.com/apache/iceberg/pull/18416#discussion_r4545245809
##########
format/spec.md:
##########
@@ -282,6 +283,7 @@ Supported primitive types are defined in the table below.
Primitive types added
| | **`uuid`** | Universally unique identifiers
| Should use 16-byte fixed
|
| | **`fixed(L)`** | Fixed-length byte array of length L
|
|
| | **`binary`** | Arbitrary-length byte array
|
|
+| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite
numeric values with element type E and dimension D | E must be `int`, `long`,
`float`, or `double`; D must be positive |
Review Comment:
Following some research and implementation feedback from
https://github.com/apache/iceberg/pull/18676/changes, I’m leaning toward
keeping `vector(E, D)` as an Iceberg primitive.
The implementation in [#18676](https://github.com/apache/iceberg/pull/18676)
strengthens that preference. `VectorType` fits into the existing primitive-type
machinery, with `E` and `D` as type parameters and the existing schema
serialization path handling the type. A vector field has its normal field ID,
with no additional element field ID. This seems natural for a single schema
value, similar to `fixed(L)`.
Other databases provide precedents for both approaches:
- Oracle explicitly classifies VECTOR as a [scalar type family in
PL/SQL](https://docs.oracle.com/en/database/oracle/oracle-database/26/lnpls/sql-data-types.html).
- [pgvector defines vector as a PostgreSQL base
type](https://github.com/pgvector/pgvector/blob/master/sql/vector.sql), which
PostgreSQL describes as a [base/scalar
type](https://www.postgresql.org/docs/current/sql-createtype.html#SQL-CREATETYPE-BASETYPE).
- Cassandra includes VECTOR among [native
types](https://cassandra.apache.org/doc/stable/cassandra/developing/cql/types.html),
separately from collections.
- On the other side, DuckDB uses fixed-size arrays, explicitly classified as
[nested types](https://duckdb.org/docs/2.0/sql/data_types/overview), and
BigQuery’s
[VECTOR_SEARCH](https://docs.cloud.google.com/bigquery/docs/reference/standard-sql/search_functions#vector_search)
uses `ARRAY<FLOAT64>`.
- ClickHouse uses numeric arrays for [conventional vector
search](https://clickhouse.com/docs/reference/engines/table-engines/mergetree-family/annindexes),
alongside its specialized [QBit
type](https://clickhouse.com/docs/reference/data-types/qbit).
The database conventions vary. For Iceberg, the schema model and the
straightforward API integration demonstrated in #18676 seem like good reasons
to retain the primitive classification.
For now I'll keep the PR to treat vector as a primitive type, and we can
discuss this in next sync meeting. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]