[ 
https://issues.apache.org/jira/browse/LUCENE-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104642#comment-17104642
 ] 

Dumitru Daniliuc commented on LUCENE-7368:
------------------------------------------

[~jpountz], I was wondering if you could help with a question about this patch. 
The javadocs for TFIDFSimilarity say that the IDF factor is squared in the 
final score. And before this patch (Lucene 6.6.6), it looks like it was:
{noformat}
  private final class TFIDFSimScorer extends SimScorer {
    ...
    TFIDFSimScorer(IDFStats stats, NumericDocValues norms) throws IOException {
      this.stats = stats;
      this.weightValue = stats.value;  // <--- stats.value = queryNorm * boost 
* idf.getValue() * idf.getValue()
      this.norms = norms;
    }
  }

  private static class IDFStats extends SimWeight {
    ...
    public IDFStats(String field, Explanation idf) {
      // TODO: Validate?
      this.field = field;
      this.idf = idf;
      normalize(1f, 1f);
    }
    ...
    @Override
    public void normalize(float queryNorm, float boost) {
      this.boost = boost;
      this.queryNorm = queryNorm;
      queryWeight = queryNorm * boost * idf.getValue();
      value = queryWeight * idf.getValue();         // idf for document
    }
  }
{noformat}

After this patch though (Lucene 7.0.0 and beyond), it looks like we lost an IDF 
factor in this code:
{noformat}
  private final class TFIDFSimScorer extends SimScorer {
    ....    
    TFIDFSimScorer(IDFStats stats, NumericDocValues norms, float[] normTable) 
throws IOException {
      this.stats = stats;
      this.weightValue = stats.queryWeight;  // <--- stats.queryWeight = boost 
* idf.getValue()
      this.norms = norms;
      this.normTable = normTable;
    }
  }

  static class IDFStats extends SimWeight {
    ...    
    public IDFStats(String field, float boost, Explanation idf, float[] 
normTable) {
      // TODO: Validate?
      this.field = field;
      this.idf = idf;
      this.boost = boost;
      this.queryWeight = boost * idf.getValue();
      this.normTable = normTable;
    }
  }  
{noformat}

Was this change intentional? If so, I was wondering if you could point us to 
the location where the second IDF factor is supposed to come from now.

For a bit more context: we've been running on an old Lucene version for a 
while, and we're working now on getting to the latest Lucene version (one major 
version at a time), and we've noticed that the scores for our results have lost 
an IDF factor when upgrading from Lucene 6.6.6 to 7.0.0, and this patch seems 
relevant.

Thanks!

> Remove queryNorm
> ----------------
>
>                 Key: LUCENE-7368
>                 URL: https://issues.apache.org/jira/browse/LUCENE-7368
>             Project: Lucene - Core
>          Issue Type: Sub-task
>            Reporter: Adrien Grand
>            Assignee: Adrien Grand
>            Priority: Major
>             Fix For: 7.0
>
>         Attachments: LUCENE-7368.patch
>
>
> Splitting LUCENE-7347 into smaller tasks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to