[
https://issues.apache.org/jira/browse/SOLR-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977881#comment-16977881
]
Chris M. Hostetter commented on SOLR-13946:
-------------------------------------------
Background...
Jenkins recently identified a reproducing seed failure for
SpellCheckCollatorTest.testEstimatedHitCounts on branch_8_3 that reproduces on
branch_8x...
{noformat}
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts
-Dtests.seed=AFA731DEE618DA14 -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true -Dtests.badapples=true
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.3/test-data/enwiki.random.lines.txt
-Dtests.locale=ga-IE -Dtests.timezone=Canada/Atlantic -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 0.16s J2 | SpellCheckCollatorTest.testEstimatedHitCounts <<<
[junit4] > Throwable #1: java.lang.RuntimeException: Exception during
query
[junit4] > at
__randomizedtesting.SeedInfo.seed([AFA731DEE618DA14:9E1C8FEB4327CAC4]:0)
[junit4] > at
org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:998)
[junit4] > at
org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:958)
[junit4] > at
org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:569)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > Caused by: java.lang.RuntimeException: REQUEST FAILED:
xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/long[@name='hits'
and 3 <= . and . <= 13]
[junit4] > xml response was: <?xml version="1.0" encoding="UTF-8"?>
[junit4] > <response>
[junit4] > <lst name="responseHeader"><int name="status">0</int><int
name="QTime">2</int></lst><result name="response" numFound="0"
start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst
name="everother"><int name="numFound">1</int><int
name="startOffset">9</int><int name="endOffset">18</int><arr
name="suggestion"><str>everyother</str></arr></lst></lst><lst
name="collations"><lst name="collation"><str
name="collationQuery">teststop:everyother</str><long name="hits">14</long><lst
name="misspellingsAndCorrections"><str
name="everother">everyother</str></lst></lst></lst></lst>
[junit4] > </response>
[junit4] > request
was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=/spellCheckCompRH&q=teststop:everother&spellcheck.collateMaxCollectDocs=5
[junit4] > at
org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:991)
[junit4] > ... 41 more
{noformat}
...that seed did not reproduce on branch_8_0, so i attempted to use "git
bisect" to idenfity when exactly it started to fail, but that lead to a false
end (651f41e21bd3df98f70d2673295db29506e3d2e6) where a new variable was
introduced in RandomCodec skewing the results. For completeness, i manually
tried overriding the random FSTLoadMode in RandomCodec, after consuming hte
same amount of randomness, and got a failure from this seed for all possible
values – supporting the theory that this bug existed prior to this commit, it
just doesn't manifest with the same seed prior to this.
Going one commit prior to when the randomization changed (80a6590c52b) I then
attempted to beast this test w/diff seeds to try and find a new permutation
that would fail...
{noformat}
ant beast -Dbeast.iters=1000 -Dtestcase=SpellCheckCollatorTest
-Dtests.method=testEstimatedHitCounts -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=et-EE
-Dtests.timezone=Australia/Canberra -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
{noformat}
...and found this...
{noformat}
[beaster] 2> 5212 INFO
(TEST-SpellCheckCollatorTest.testEstimatedHitCounts-seed#[989A055D5EA05EC3]) [
] o.a.s.SolrTestCaseJ4 ###Ending testEstimatedHitCounts
[beaster] 2> NOTE: reproduce with: ant test
-Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts
-Dtests.seed=989A055D5EA05EC3 -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=et-EE
-Dtests.timezone=Australia/Canberra -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[beaster] [14:21:03.465] ERROR 0.40s |
SpellCheckCollatorTest.testEstimatedHitCounts <<<
[beaster] > Throwable #1: java.lang.RuntimeException: Exception during
query
[beaster] > at
__randomizedtesting.SeedInfo.seed([989A055D5EA05EC3:A921BB68FB9F4E13]:0)
[beaster] > at
org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:934)
[beaster] > at
org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:894)
[beaster] > at
org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:569)
...
[beaster] > Caused by: java.lang.RuntimeException: REQUEST FAILED:
xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/long[@name='hits'
and 3 <= . and . <= 13]
[beaster] > xml response was: <?xml version="1.0" encoding="UTF-8"?>
[beaster] > <response>
[beaster] > <lst name="responseHeader"><int name="status">0</int><int
name="QTime">3</int></lst><result name="response" numFound="0"
start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst
name="everother"><int name="numFound">1</int><int
name="startOffset">9</int><int name="endOffset">18</int><arr
name="suggestion"><str>everyother</str></arr></lst></lst><lst
name="collations"><lst name="collation"><str
name="collationQuery">teststop:everyother</str><long name="hits">14</long><lst
name="misspellingsAndCorrections"><str
name="everother">everyother</str></lst></lst></lst></lst>
[beaster] > </response>
[beaster] >
[beaster] > request
was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=/spellCheckCompRH&q=teststop:everother&spellcheck.collateMaxCollectDocs=5
[beaster] > at
org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:927)
[beaster] > ... 41 more
{noformat}
A quick check confirmed that this seed reproduced even as far back as
branch_8_0 (93d1e67886e75623f3f72526284bdbc3b1fef7e6), but did *NOT* reproduce
when this assert was last (substantively) modified
(213a2a1791e4557afd2542a25e94eec65e29a42d) as part of SOLR-5344 (in 2016)
So back to git bisect...
{noformat}
93d1e67886e75623f3f72526284bdbc3b1fef7e6 is the first bad commit
commit 93d1e67886e75623f3f72526284bdbc3b1fef7e6
Author: jimczi <jim...@apache.org>
Date: Mon Jan 28 22:43:20 2019 +0100
LUCENE-8660: TopDocsCollectors now return an accurate count (instead of a
lower bound) if the total hit count is equal to the provided threshold.
:040000 040000 7bd5d8ab9e691e1253979cf8698fa88b2c7b9fb1
ec6791f9fc718f27eb7ffd9451bff3595e25250b M lucene
bisect run success
{noformat}
...I'm not really clear on how this change could affect ths test given that
AFAIK every usage of TopDocsCollectors in solr _should_ be insisting on an
exact count of total hits (ie threshold=MAX_INT) but since this commit has no
modifications to the test-framework randomness that could otherwise impact this
test, the fact that it's first failing seed means *something* about these
changes impacted the underlying code being tested.
> SpellCheckCollatorTest.testEstimatedHitCounts reproducing failure seed
> ----------------------------------------------------------------------
>
> Key: SOLR-13946
> URL: https://issues.apache.org/jira/browse/SOLR-13946
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Chris M. Hostetter
> Priority: Major
>
> The following seed reliably fails on branch_8x. based on the results of git
> bisect, it appears that this in some way relates to changes made in
> LUCENE-8660 (TopDocsCollectors's accuracy in reporting totalHits when dealing
> with totalHitsThreshold) even though all usage in sol should be requesting
> {{totalHitsThreshold=Integer.MAX_VALUE}}
> {noformat}
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts
> -Dtests.seed=AFA731DEE618DA14 -Dtests.multiplier=2 -Dtests.nightly=true
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ga-IE
> -Dtests.timezone=Canada/Atlantic -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> [junit4] ERROR 0.36s | SpellCheckCollatorTest.testEstimatedHitCounts <<<
> [junit4] > Throwable #1: java.lang.RuntimeException: Exception during
> query
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([AFA731DEE618DA14:9E1C8FEB4327CAC4]:0)
> [junit4] > at
> org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1001)
> [junit4] > at
> org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:961)
> [junit4] > at
> org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:569)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > Caused by: java.lang.RuntimeException: REQUEST FAILED:
> xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/long[@name='hits'
> and 3 <= . and . <= 13]
> [junit4] > xml response was: <?xml version="1.0" encoding="UTF-8"?>
> [junit4] > <response>
> [junit4] > <lst name="responseHeader"><int name="status">0</int><int
> name="QTime">2</int></lst><result name="response" numFound="0"
> start="0"></result><lst name="spellcheck"><lst name="suggestions"><lst
> name="everother"><int name="numFound">1</int><int
> name="startOffset">9</int><int name="endOffset">18</int><arr
> name="suggestion"><str>everyother</str></arr></lst></lst><lst
> name="collations"><lst name="collation"><str
> name="collationQuery">teststop:everyother</str><long
> name="hits">14</long><lst name="misspellingsAndCorrections"><str
> name="everother">everyother</str></lst></lst></lst></lst>
> [junit4] > </response>
> [junit4] > request
> was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=/spellCheckCompRH&q=teststop:everother&spellcheck.collateMaxCollectDocs=5
> [junit4] > at
> org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:994)
> [junit4] > ... 41 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
