kaivalnp opened a new pull request, #16938: URL: https://github.com/apache/lucene/pull/16938
### Description Partially addresses #16595. The sandboxed dedup vector format (#15979) de-duplicates across all documents in a segment. This PR _restricts_ de-duplication to within a single document because of the following reasons: - During merge, avoids read-back of vectors for full equality comparison on hash collisions across documents. This can lead to expensive IO when raw vectors overflow beyond memory. When limited to a document, de-duplication only occurs during flush when vectors are in-memory, and merging of vectors becomes a simpler sequential read + copy, similar to the flat format. - If vectors are attributable to a single document, they can be sorted on-disk (in index sort order). This change is accompanied by appropriate hinting of the access pattern during merge (sequential), similar to the flat format. Cons: duplicates across documents are no longer collapsed, leading to larger index sizes than ideal. Note: this does not change the on-disk layout (and consequently, no minor version bump). Future work: the `fieldOrdToGroupOrd` mapping is now monotonically increasing, and can be replaced by more appropriate data structures? Disclaimer: some of the code, tests and benchmarking is AI-assisted with a human (me) in the loop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
