pvary commented on PR #18485: URL: https://github.com/apache/iceberg/pull/18485#issuecomment-6289640778
@danielcweeks: We would like to bring Vector embeddings natively to Iceberg tables. We have FileType now, so we can store the raw data and we would like to store the embeddings next to it. The readers of the Vectors need to know the size of the data. When an engine reads an int[1024] vector, it needs to know at planning time the size of the data in memory. int32 vs int8 is a sizeable difference we need a type or some other schema/table level markers. float16 uses different encoding so we need a different type. These are the vector types used in AI workloads, so I think we definitely need a way to natively support them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
