jimczi commented on PR #16571: URL: https://github.com/apache/lucene/pull/16571#issuecomment-5443964635
> Interesting. I miss what are the benefits/ use cases for this? Mostly range routing. You route documents by a key, a tenant id for instance, and sort the index on it. The index sort gives you contiguity inside a segment, but a tenant's documents are still spread over all of them, and merging can't fix that. A merge takes N segments and writes one, so the output covers the union of the ranges its inputs covered. It only gets wider, never narrower. With M outputs you can cut on the key instead, so each output owns a disjoint range, and a tenant's data ends up in a bounded set of segments and stays that way as merging continues. That's what makes resharding possible. When an index gets too big for one shard you split it by key range, and if the segments already own those ranges you hand whole segments to each side. Otherwise you copy everything and delete what doesn't belong, on both sides. This is how LSM stores already work. HBase regions are key ranges and a split is just a boundary operation, the daughter regions reference the parent files instead of copying them. Cassandra orders by token so a range can be streamed rather than rebuilt. What makes it work is that a compaction reads N files and writes M, cut on key boundaries. Lucene is the one where a merge always collapses to a single output. The merge policy in the PR is just something concrete to look at, it doesn't need a sort and splits by position. I'm also curious whether a partition following the vector space would help vector search, but I haven't tried it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
