zacharymorn commented on pull request #418:
URL: https://github.com/apache/lucene/pull/418#issuecomment-985229215
I just added two logging statements in `ImpactsDISI` to print out `upTo` and
max score comparison to examine the docs pruning results:
```
diff --git a/lucene/core/src/java/org/apache/lucene/search/ImpactsDISI.java
b/lucene/core/src/java/org/apache/lucene/search/ImpactsDISI.java
index 843bb4ad57b..3930ae2f50d 100644
--- a/lucene/core/src/java/org/apache/lucene/search/ImpactsDISI.java
+++ b/lucene/core/src/java/org/apache/lucene/search/ImpactsDISI.java
@@ -110,8 +110,10 @@ public final class ImpactsDISI extends DocIdSetIterator
{
assert upTo >= target;
if (maxScore >= minCompetitiveScore) {
+ System.out.println("Keeping - target: " + target + " upTo: " + upTo
+ " minCompScore: " + minCompetitiveScore + " maxScore: " + maxScore);
return target;
}
+ System.out.println("Skipping - target: " + target + " upTo: " + upTo
+ " minCompScore: " + minCompetitiveScore + " maxScore: " + maxScore);
if (upTo == NO_MORE_DOCS) {
return NO_MORE_DOCS;
```
and the following query was used:
```
CombinedFieldQuery query =
new CombinedFieldQuery.Builder()
.addField("titleTokenized", (float) 20.0)
.addField("body", (float) 1.0)
.addTerm(new BytesRef("three"))
.addTerm(new BytesRef("geronimo"))
.build();
```
For the implementation that uses mostly weighted field and lowest
`Impacts#getDocIdUpTo` to drive scoring, `upTo` and `maxScore` were high, hence
there was no pruning.
```
1> Keeping - target: 152 upTo: 232464 minCompScore: 0.17629458 maxScore:
2.1560352
1> Keeping - target: 159 upTo: 232464 minCompScore: 0.18669213 maxScore:
2.1560352
1> Keeping - target: 160 upTo: 232464 minCompScore: 0.20146967 maxScore:
2.1560352
1> Keeping - target: 161 upTo: 232464 minCompScore: 0.21165837 maxScore:
2.1560352
1> Keeping - target: 167 upTo: 232464 minCompScore: 0.21696068 maxScore:
2.1560352
1> Keeping - target: 170 upTo: 232464 minCompScore: 0.23223151 maxScore:
2.1560352
1> Keeping - target: 171 upTo: 232464 minCompScore: 0.26476994 maxScore:
2.1560352
1> Keeping - target: 172 upTo: 232464 minCompScore: 0.27027848 maxScore:
2.1560352
1> Keeping - target: 174 upTo: 232464 minCompScore: 0.27216592 maxScore:
2.1560352
1> Keeping - target: 175 upTo: 232464 minCompScore: 0.27311972 maxScore:
2.1560352
1> Keeping - target: 176 upTo: 232464 minCompScore: 0.28871515 maxScore:
2.1560352
1> Keeping - target: 179 upTo: 232464 minCompScore: 0.2955102 maxScore:
2.1560352
1> Keeping - target: 185 upTo: 232464 minCompScore: 0.31796065 maxScore:
2.1560352
1> Keeping - target: 187 upTo: 232464 minCompScore: 0.32868698 maxScore:
2.1560352
1> Keeping - target: 189 upTo: 232464 minCompScore: 0.33091953 maxScore:
2.1560352
1> Keeping - target: 190 upTo: 232464 minCompScore: 0.353465 maxScore:
2.1560352
1> Keeping - target: 191 upTo: 232464 minCompScore: 0.36112097 maxScore:
2.1560352
1> Keeping - target: 194 upTo: 232464 minCompScore: 0.3743663 maxScore:
2.1560352
1> Keeping - target: 195 upTo: 232464 minCompScore: 0.38150516 maxScore:
2.1560352
1> Keeping - target: 197 upTo: 232464 minCompScore: 0.39127985 maxScore:
2.1560352
1> Keeping - target: 198 upTo: 232464 minCompScore: 0.39738885 maxScore:
2.1560352
1> Keeping - target: 208 upTo: 232464 minCompScore: 0.39789525 maxScore:
2.1560352
1> Keeping - target: 216 upTo: 232464 minCompScore: 0.41046718 maxScore:
2.1560352
1> Keeping - target: 219 upTo: 232464 minCompScore: 0.4148362 maxScore:
2.1560352
1> Keeping - target: 224 upTo: 232464 minCompScore: 0.4204301 maxScore:
2.1560352
1> Keeping - target: 227 upTo: 232464 minCompScore: 0.43892494 maxScore:
2.1560352
1> Keeping - target: 234 upTo: 232464 minCompScore: 0.4551008 maxScore:
2.1560352
1> Keeping - target: 235 upTo: 232464 minCompScore: 0.47162262 maxScore:
2.1560352
1> Keeping - target: 238 upTo: 232464 minCompScore: 0.47382453 maxScore:
2.1560352
1> Keeping - target: 239 upTo: 232464 minCompScore: 0.48034182 maxScore:
2.1560352
1> Keeping - target: 243 upTo: 232464 minCompScore: 0.48332027 maxScore:
2.1560352
1> Keeping - target: 245 upTo: 232464 minCompScore: 0.4903498 maxScore:
2.1560352
1> Keeping - target: 246 upTo: 232464 minCompScore: 0.5026699 maxScore:
2.1560352
...
```
For the implementation that uses mostly weighted field and lowest cost / doc
freq to drive scoring, `upTo` turned out to be the integer value of no more
docs for that query, and there was no pruning either.
```
1> Keeping - target: 151 upTo: 2147483647 minCompScore: 0.15862586
maxScore: 2.1560352
1> Keeping - target: 152 upTo: 2147483647 minCompScore: 0.17629458
maxScore: 2.1560352
1> Keeping - target: 159 upTo: 2147483647 minCompScore: 0.18669213
maxScore: 2.1560352
1> Keeping - target: 160 upTo: 2147483647 minCompScore: 0.20146967
maxScore: 2.1560352
1> Keeping - target: 161 upTo: 2147483647 minCompScore: 0.21165837
maxScore: 2.1560352
1> Keeping - target: 167 upTo: 2147483647 minCompScore: 0.21696068
maxScore: 2.1560352
1> Keeping - target: 170 upTo: 2147483647 minCompScore: 0.23223151
maxScore: 2.1560352
1> Keeping - target: 171 upTo: 2147483647 minCompScore: 0.26476994
maxScore: 2.1560352
1> Keeping - target: 172 upTo: 2147483647 minCompScore: 0.27027848
maxScore: 2.1560352
1> Keeping - target: 174 upTo: 2147483647 minCompScore: 0.27216592
maxScore: 2.1560352
1> Keeping - target: 175 upTo: 2147483647 minCompScore: 0.27311972
maxScore: 2.1560352
1> Keeping - target: 176 upTo: 2147483647 minCompScore: 0.28871515
maxScore: 2.1560352
1> Keeping - target: 179 upTo: 2147483647 minCompScore: 0.2955102
maxScore: 2.1560352
1> Keeping - target: 185 upTo: 2147483647 minCompScore: 0.31796065
maxScore: 2.1560352
1> Keeping - target: 187 upTo: 2147483647 minCompScore: 0.32868698
maxScore: 2.1560352
1> Keeping - target: 189 upTo: 2147483647 minCompScore: 0.33091953
maxScore: 2.1560352
1> Keeping - target: 190 upTo: 2147483647 minCompScore: 0.353465 maxScore:
2.1560352
1> Keeping - target: 191 upTo: 2147483647 minCompScore: 0.36112097
maxScore: 2.1560352
1> Keeping - target: 194 upTo: 2147483647 minCompScore: 0.3743663
maxScore: 2.1560352
1> Keeping - target: 195 upTo: 2147483647 minCompScore: 0.38150516
maxScore: 2.1560352
1> Keeping - target: 197 upTo: 2147483647 minCompScore: 0.39127985
maxScore: 2.1560352
1> Keeping - target: 198 upTo: 2147483647 minCompScore: 0.39738885
maxScore: 2.1560352
1> Keeping - target: 208 upTo: 2147483647 minCompScore: 0.39789525
maxScore: 2.1560352
1> Keeping - target: 216 upTo: 2147483647 minCompScore: 0.41046718
maxScore: 2.1560352
1> Keeping - target: 219 upTo: 2147483647 minCompScore: 0.4148362
maxScore: 2.1560352
1> Keeping - target: 224 upTo: 2147483647 minCompScore: 0.4204301
maxScore: 2.1560352
1> Keeping - target: 227 upTo: 2147483647 minCompScore: 0.43892494
maxScore: 2.1560352
1> Keeping - target: 234 upTo: 2147483647 minCompScore: 0.4551008
maxScore: 2.1560352
1> Keeping - target: 235 upTo: 2147483647 minCompScore: 0.47162262
maxScore: 2.1560352
1> Keeping - target: 238 upTo: 2147483647 minCompScore: 0.47382453
maxScore: 2.1560352
1> Keeping - target: 239 upTo: 2147483647 minCompScore: 0.48034182
maxScore: 2.1560352
1> Keeping - target: 243 upTo: 2147483647 minCompScore: 0.48332027
maxScore: 2.1560352
1> Keeping - target: 245 upTo: 2147483647 minCompScore: 0.4903498
maxScore: 2.1560352
1> Keeping - target: 246 upTo: 2147483647 minCompScore: 0.5026699
maxScore: 2.1560352
1> Keeping - target: 247 upTo: 2147483647 minCompScore: 0.50923806
maxScore: 2.1560352
1> Keeping - target: 249 upTo: 2147483647 minCompScore: 0.50958425
maxScore: 2.1560352
1> Keeping - target: 259 upTo: 2147483647 minCompScore: 0.51977855
maxScore: 2.1560352
1> Keeping - target: 265 upTo: 2147483647 minCompScore: 0.5250303
maxScore: 2.1560352
1> Keeping - target: 266 upTo: 2147483647 minCompScore: 0.53061455
maxScore: 2.1560352
1> Keeping - target: 267 upTo: 2147483647 minCompScore: 0.53076476
maxScore: 2.1560352
1> Keeping - target: 269 upTo: 2147483647 minCompScore: 0.53151745
maxScore: 2.1560352
1> Keeping - target: 275 upTo: 2147483647 minCompScore: 0.56218606
maxScore: 2.1560352
1> Keeping - target: 279 upTo: 2147483647 minCompScore: 0.57041436
maxScore: 2.1560352
1> Keeping - target: 280 upTo: 2147483647 minCompScore: 0.58023125
maxScore: 2.1560352
1> Keeping - target: 296 upTo: 2147483647 minCompScore: 0.592781 maxScore:
2.1560352
1> Keeping - target: 298 upTo: 2147483647 minCompScore: 0.5975618
maxScore: 2.1560352
1> Keeping - target: 314 upTo: 2147483647 minCompScore: 0.5996211
maxScore: 2.1560352
1> Keeping - target: 318 upTo: 2147483647 minCompScore: 0.6027052
maxScore: 2.1560352
1> Keeping - target: 319 upTo: 2147483647 minCompScore: 0.6080974
maxScore: 2.1560352
1> Keeping - target: 329 upTo: 2147483647 minCompScore: 0.6116286
maxScore: 2.1560352
1> Keeping - target: 335 upTo: 2147483647 minCompScore: 0.6127903
maxScore: 2.1560352
1> Keeping - target: 336 upTo: 2147483647 minCompScore: 0.6186264
maxScore: 2.1560352
1> Keeping - target: 339 upTo: 2147483647 minCompScore: 0.62189597
maxScore: 2.1560352
1> Keeping - target: 342 upTo: 2147483647 minCompScore: 0.63768846
maxScore: 2.1560352
1> Keeping - target: 346 upTo: 2147483647 minCompScore: 0.63877517
maxScore: 2.1560352
1> Keeping - target: 352 upTo: 2147483647 minCompScore: 0.66235477
maxScore: 2.1560352
1> Keeping - target: 354 upTo: 2147483647 minCompScore: 0.6642242
maxScore: 2.1560352
1> Keeping - target: 355 upTo: 2147483647 minCompScore: 0.6710866
maxScore: 2.1560352
1> Keeping - target: 358 upTo: 2147483647 minCompScore: 0.6718084
maxScore: 2.1560352
1> Keeping - target: 362 upTo: 2147483647 minCompScore: 0.67271274
maxScore: 2.1560352
1> Keeping - target: 365 upTo: 2147483647 minCompScore: 0.6831766
maxScore: 2.1560352
1> Keeping - target: 367 upTo: 2147483647 minCompScore: 0.683592 maxScore:
2.1560352
1> Keeping - target: 371 upTo: 2147483647 minCompScore: 0.6896402
maxScore: 2.1560352
...
```
For the implementation that uses lowest `Impacts#getDocIdUpTo` across all
fields, we would get lower `upTo` and max score in general, and hence some docs
do get pruned.
```
1> Keeping - target: 152 upTo: 195 minCompScore: 0.17629458 maxScore:
1.3767262
1> Keeping - target: 159 upTo: 195 minCompScore: 0.18669213 maxScore:
1.3767262
...
1> Keeping - target: 219 upTo: 436 minCompScore: 0.4148362 maxScore:
1.7731527
1> Keeping - target: 224 upTo: 436 minCompScore: 0.4204301 maxScore:
1.7731527
...
1> Keeping - target: 764 upTo: 902 minCompScore: 0.9138265 maxScore:
1.65851
1> Keeping - target: 767 upTo: 902 minCompScore: 0.91438395 maxScore:
1.65851
1> Keeping - target: 779 upTo: 902 minCompScore: 0.91606015 maxScore:
1.65851
...
1> Keeping - target: 1278 upTo: 1384 minCompScore: 1.0459963 maxScore:
1.5856887
1> Keeping - target: 1280 upTo: 1384 minCompScore: 1.0520688 maxScore:
1.5856887
...
1> Keeping - target: 7144 upTo: 7227 minCompScore: 1.428069 maxScore:
1.7997875
1> Keeping - target: 7228 upTo: 7453 minCompScore: 1.4284092 maxScore:
1.4454873
1> Skipping - target: 7454 upTo: 7702 minCompScore: 1.4284092 maxScore:
1.3662372
1> Keeping - target: 7703 upTo: 7921 minCompScore: 1.4284092 maxScore:
1.6103871
1> Keeping - target: 7922 upTo: 8131 minCompScore: 1.4284092 maxScore:
1.5153265
...
1> Keeping - target: 17039 upTo: 17327 minCompScore: 1.5600302 maxScore:
1.6949782
1> Keeping - target: 17328 upTo: 17873 minCompScore: 1.5600302 maxScore:
1.8440268
1> Keeping - target: 17874 upTo: 18438 minCompScore: 1.5600302 maxScore:
1.8541759
1> Skipping - target: 18439 upTo: 18774 minCompScore: 1.5600302 maxScore:
1.5445734
1> Keeping - target: 18775 upTo: 19012 minCompScore: 1.5600302 maxScore:
2.1527436
1> Keeping - target: 19001 upTo: 19012 minCompScore: 1.5605642 maxScore:
2.1527436
...
1> Keeping - target: 19408 upTo: 19541 minCompScore: 1.5636915 maxScore:
1.927903
1> Skipping - target: 19542 upTo: 19834 minCompScore: 1.5636915 maxScore:
1.5145609
1> Keeping - target: 19835 upTo: 20077 minCompScore: 1.5636915 maxScore:
1.699799
1> Keeping - target: 20066 upTo: 20077 minCompScore: 1.5694537 maxScore:
1.699799
...
1> Keeping - target: 27159 upTo: 27396 minCompScore: 1.6439995 maxScore:
2.1519704
1> Keeping - target: 27182 upTo: 27396 minCompScore: 1.6449014 maxScore:
2.1519704
1> Keeping - target: 27397 upTo: 27627 minCompScore: 1.6449014 maxScore:
2.1505308
1> Keeping - target: 27500 upTo: 27627 minCompScore: 1.6467081 maxScore:
2.1505308
1> Skipping - target: 27628 upTo: 27859 minCompScore: 1.6467081 maxScore:
1.5844319
1> Keeping - target: 27860 upTo: 28118 minCompScore: 1.6467081 maxScore:
1.7705853
1> Keeping - target: 27976 upTo: 28118 minCompScore: 1.6487088 maxScore:
1.7705853
1> Skipping - target: 28119 upTo: 28370 minCompScore: 1.6487088 maxScore:
1.576628
1> Keeping - target: 28371 upTo: 28619 minCompScore: 1.6487088 maxScore:
2.0417855
1> Keeping - target: 28620 upTo: 28954 minCompScore: 1.6487088 maxScore:
2.1356478
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
