[
https://issues.apache.org/jira/browse/LUCENE-10592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566445#comment-17566445
]
Julie Tibshirani edited comment on LUCENE-10592 at 7/13/22 6:16 PM:
--------------------------------------------------------------------
This change makes sense to me too, and I like the direction the PR is going!
The one downside is that the index sorting logic becomes more complicated.
Specifically, after building the graph, we need to remap all the ordinals to
account for the sorting. I don't see a good way around this, maybe we just need
to accept that this becomes more complex?
was (Author: julietibs):
This change makes sense to me too, and I like the direction the PR is going!
The one downside is that the indexing sorting logic becomes more complicated.
Specifically, we after building the graph, we need to remap all the ordinals to
account for the sorting. I don't see a good way around this, maybe we just need
to accept that this becomes more complex?
> Should we build HNSW graph on the fly during indexing
> -----------------------------------------------------
>
> Key: LUCENE-10592
> URL: https://issues.apache.org/jira/browse/LUCENE-10592
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Mayya Sharipova
> Assignee: Mayya Sharipova
> Priority: Minor
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Currently, when we index vectors for KnnVectorField, we buffer those vectors
> in memory and on flush during a segment construction we build an HNSW graph.
> As building an HNSW graph is very expensive, this makes flush operation take
> a lot of time. This also makes overall indexing performance quite
> unpredictable (as the number of flushes are defined by memory used, and the
> presence of concurrent searches), e.g. some indexing operations return almost
> instantly while others that trigger flush take a lot of time.
> Building an HNSW graph on the fly as we index vectors allows to avoid this
> problem, and spread a load of HNSW graph construction evenly during indexing.
> This will also supersede LUCENE-10194
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
