[
https://issues.apache.org/jira/browse/LUCENE-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104642#comment-17104642
]
Dumitru Daniliuc commented on LUCENE-7368:
------------------------------------------
[~jpountz], I was wondering if you could help with a question about this patch.
The javadocs for TFIDFSimilarity say that the IDF factor is squared in the
final score. And before this patch (Lucene 6.6.6), it looks like it was:
{noformat}
private final class TFIDFSimScorer extends SimScorer {
...
TFIDFSimScorer(IDFStats stats, NumericDocValues norms) throws IOException {
this.stats = stats;
this.weightValue = stats.value; // <--- stats.value = queryNorm * boost
* idf.getValue() * idf.getValue()
this.norms = norms;
}
}
private static class IDFStats extends SimWeight {
...
public IDFStats(String field, Explanation idf) {
// TODO: Validate?
this.field = field;
this.idf = idf;
normalize(1f, 1f);
}
...
@Override
public void normalize(float queryNorm, float boost) {
this.boost = boost;
this.queryNorm = queryNorm;
queryWeight = queryNorm * boost * idf.getValue();
value = queryWeight * idf.getValue(); // idf for document
}
}
{noformat}
After this patch though (Lucene 7.0.0 and beyond), it looks like we lost an IDF
factor in this code:
{noformat}
private final class TFIDFSimScorer extends SimScorer {
....
TFIDFSimScorer(IDFStats stats, NumericDocValues norms, float[] normTable)
throws IOException {
this.stats = stats;
this.weightValue = stats.queryWeight; // <--- stats.queryWeight = boost
* idf.getValue()
this.norms = norms;
this.normTable = normTable;
}
}
static class IDFStats extends SimWeight {
...
public IDFStats(String field, float boost, Explanation idf, float[]
normTable) {
// TODO: Validate?
this.field = field;
this.idf = idf;
this.boost = boost;
this.queryWeight = boost * idf.getValue();
this.normTable = normTable;
}
}
{noformat}
Was this change intentional? If so, I was wondering if you could point us to
the location where the second IDF factor is supposed to come from now.
For a bit more context: we've been running on an old Lucene version for a
while, and we're working now on getting to the latest Lucene version (one major
version at a time), and we've noticed that the scores for our results have lost
an IDF factor when upgrading from Lucene 6.6.6 to 7.0.0, and this patch seems
relevant.
Thanks!
> Remove queryNorm
> ----------------
>
> Key: LUCENE-7368
> URL: https://issues.apache.org/jira/browse/LUCENE-7368
> Project: Lucene - Core
> Issue Type: Sub-task
> Reporter: Adrien Grand
> Assignee: Adrien Grand
> Priority: Major
> Fix For: 7.0
>
> Attachments: LUCENE-7368.patch
>
>
> Splitting LUCENE-7347 into smaller tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]