[
https://issues.apache.org/jira/browse/LUCENE-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17579831#comment-17579831
]
Julie Tibshirani commented on LUCENE-10471:
-------------------------------------------
bq. It makes sense to me to increase the limit to the point where we would see
actual bugs/failures, or where the large numbers might prevent us from making
some future optimization, rather than trying to determine where the performance
stops being acceptable - that's a question for users to decide for themselves.
Mike's perspective makes sense to me too. I'd be supportive of increasing the
limit to an upper bound. Maybe we could run a test with ~1 million synthetic
vectors with the proposed max dimension (~16K) to check there are no failures
or unexpected behavior?
> Increase the number of dims for KNN vectors to 2048
> ---------------------------------------------------
>
> Key: LUCENE-10471
> URL: https://issues.apache.org/jira/browse/LUCENE-10471
> Project: Lucene - Core
> Issue Type: Wish
> Reporter: Mayya Sharipova
> Priority: Trivial
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The current maximum allowed number of dimensions is equal to 1024. But we see
> in practice a couple well-known models that produce vectors with > 1024
> dimensions (e.g
> [mobilenet_v2|https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/1]
> uses 1280d vectors, OpenAI / GPT-3 Babbage uses 2048d vectors). Increasing
> max dims to `2048` will satisfy these use cases.
> I am wondering if anybody has strong objections against this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
