hack4chang commented on code in PR #13470:
URL: https://github.com/apache/lucene/pull/13470#discussion_r1632393474
##########
lucene/core/src/java/org/apache/lucene/search/TopDocs.java:
##########
@@ -350,4 +354,38 @@ private static TopDocs mergeAux(
return new TopFieldDocs(totalHits, hits, sort.getSort());
}
}
+
+ /** Reciprocal Rank Fusion method. */
+ public static TopDocs rrf(int TopN, int k, TopDocs[] hits) {
+ Map<Integer, Float> rrfScore = new HashMap<>();
+ long minHits = Long.MAX_VALUE;
+ for (TopDocs topDoc : hits) {
+ minHits = Math.min(minHits, topDoc.totalHits.value);
+ Map<Integer, Float> scoreMap = new HashMap<>();
+ for (ScoreDoc scoreDoc : topDoc.scoreDocs) {
+ scoreMap.put(scoreDoc.doc, scoreDoc.score);
+ }
+
+ List<Map.Entry<Integer, Float>> scoreList = new
ArrayList<>(scoreMap.entrySet());
+ scoreList.sort(Map.Entry.comparingByValue());
+
+ int rank = 1;
+ for (ScoreDoc scoreDoc : topDoc.scoreDocs) {
+ rrfScore.put(scoreDoc.doc, rrfScore.getOrDefault(scoreDoc.doc, 0.0f) +
1.0f / (rank + k));
+ rank++;
+ }
+ }
+
+ List<Map.Entry<Integer, Float>> rrfScoreRank = new
ArrayList<>(rrfScore.entrySet());
+ rrfScoreRank.sort(
+ Map.Entry.<Integer, Float>comparingByValue().reversed()); // Sort in
descending order
+
+ ScoreDoc[] rrfScoreDocs = new ScoreDoc[Math.min(TopN,
rrfScoreRank.size())];
+ for (int i = 0; i < rrfScoreDocs.length; i++) {
+ rrfScoreDocs[i] = new ScoreDoc(rrfScoreRank.get(i).getKey(),
rrfScoreRank.get(i).getValue());
Review Comment:
If we do the rrf by setting the unique key as docid and shardIndex, what
would be the difference between TopDocs#rrf and TopDocs#merge? I think giving
an example could express better. Suppose that we have two Shards, and we want
to retrieve top 3 results from each shards and do rrf on top of them. There's
three documents A, B and C. In Shard1, the top 3 is A -> B -> C. In Shard2,
it's B -> C -> A. The original rrf method would calculate the rank by
aggregating the docid, assume the constant k is 1. Top 3 results would be B
```(1/(k+2) + 1/(k+1))``` - A ```(1/(k+1) + 1/(k+3))``` - C ```(1/(k+3) +
1/(k+2))```.
If we are going to consider the shardIndex as a unique key as well, how
should the rrf rank to be presented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
