mccullocht commented on issue #14758:
URL: https://github.com/apache/lucene/issues/14758#issuecomment-4290418224
@mikemccand I had started playing with the idea of linking fields a doc
values field for partitioning but don't yet have something useful enough to
share as a strawman implementation, but I have some observations:
* `Sorted{,Set}DocValues` does feel like the right data structure because it
handles a lot of the grouping aspects that are needed here.
* The KNN writer can consume DocValues with a bit of plumbing. In the
ingestion path it forces us to wait until the segment is flushed to construct
the graph. In the merge path it feels like we're duplicating at least some of
the work required to merge DocValues, although I guess if we introduced some
constraints on field ordering in merge we could use a merged view.
* In addition to a graph per doc value we also need a secondary map that
either maps docids <-> vector ordinals, or maps global vector ordinals <->
local vector ordinals. This is probably true in any solution.
* The search interface would still need to take `Optional<BytesRef>` or
similar to choose the right graph.
Maybe the right solution is to compose it the other way -- extend the
interface to accept labels at ingestion and query time and use DocValues
underneath?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
