[
https://issues.apache.org/jira/browse/SOLR-15071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263587#comment-17263587
]
Christine Poerschke commented on SOLR-15071:
--------------------------------------------
{quote}... why does SFS not (potentially) have two-phase iterator support?
Looking at the code, it seems it simply delegates to the Scorer it wraps, ...
{quote}
Yes, that's what I meant by "potentially" i.e. if the wrapped scorer supports
it then SFS claims to support it too. If the wrapped scorer does not support it
then SFS also does not support it.
{quote}... if that Scorer supports it, shouldn't SFS likewise? ...
{quote}
In principle yes but
[observationally|https://github.com/apache/lucene-solr/pull/2196#issuecomment-758599976]
in {{MinShouldMatchSumScorer}} at least after the use of the two-phase
iterator the {{score}} method can no longer be called. Haven't yet looked
further into the exact details.
Prior to 8.6 i.e.
[https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.2/solr/contrib/ltr/src/java/org/apache/solr/ltr/feature/SolrFeature.java#L269-L294]
SFS did not have two-phase iterator support. I would suggest to scope of this
ticket here to _re-instate the non-support_ of two-phase iteration and the
_addition of two-phase support_ would be a separate future enhancement.
bq. ... A real unit test would be ideal. Are there obstacles make this
difficult?
I agree.
[https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.7.0/solr/contrib/ltr/src/test/org/apache/solr/ltr/feature/TestEdisMaxSolrFeature.java]
is an existing test that could be extended or a new test similar to it could
be created. [~florin.babes] and/or [~dizzu333] would you be interested in
having a go at that?
> Bug on LTR when using solr 8.6.3 - index out of bounds
> DisiPriorityQueue.add(DisiPriorityQueue.java:102)
> --------------------------------------------------------------------------------------------------------
>
> Key: SOLR-15071
> URL: https://issues.apache.org/jira/browse/SOLR-15071
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - LTR
> Affects Versions: 8.6, 8.7
> Reporter: Florin Babes
> Assignee: Christine Poerschke
> Priority: Major
> Labels: ltr
> Fix For: 8.8
>
> Attachments: featurestore+model+sample_documents.zip
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Hello,
> We are trying to update Solr from 8.3.1 to 8.6.3. On Solr 8.3.1 we are
> using LTR in production using a MultipleAdditiveTrees model. On Solr 8.6.3
> we receive an error when we try to compute some SolrFeatures. We didn't
> find any pattern of the queries that fail.
> Example:
> We have the following query raw parameters:
> q=lg cx 4k oled 120 hz -> just of many examples
> term_dq=lg cx 4k oled 120 hz
> rq={!ltr model=model reRankDocs=1000 store=feature_store
> efi.term=${term_dq}}
> defType=edismax,
> mm=2<75%
> The features are something like this:
> {
> "name":"similarity_query_fileld_1",
> "class":"org.apache.solr.ltr.feature.SolrFeature",
> "params":\{"q":"{!dismax qf=query_field_1 mm=1}${term}"},
> "store":"feature_store"
> },
> {
> "name":"similarity_query_field_2",
> "class":"org.apache.solr.ltr.feature.SolrFeature",
> "params":\{"q":"{!dismax qf=query_field_2 mm=5}${term}"},
> "store":"feature_store"
> }
> We are testing ~6300 production queries and for about 1% of them we receive
> that following error message:
> "metadata":[
> "error-class","org.apache.solr.common.SolrException",
> "root-error-class","java.lang.ArrayIndexOutOfBoundsException"],
> "msg":"java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds
> for length 2",
> The stacktrace is :
> org.apache.solr.common.SolrException:
> java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds for length 2
> at org.apache.solr.search.ReRankCollector.topDocs(ReRankCollector.java:154)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:159
> 9)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1413
> )
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:596)
> at
> org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryC
> omponent.java:1513)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:403
> )
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.
> java:360)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java
> :214)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2627)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:795)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:568)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.jav
> a:1596)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235
> )
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:161
> 0)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233
> )
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:130
> 0)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580
> )
> at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215
> )
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerC
> ollection.java:221)
> at
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java
> :177)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java
> :146)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnectio
> n.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.jav
> a:336)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.j
> ava:313)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.
> java:171)
> at
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:12
> 9)
> at
> org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(Reserved
> ThreadExecutor.java:375)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:
> 938)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 2 out of bounds
> for length 2
> at
> org.apache.lucene.search.DisiPriorityQueue.add(DisiPriorityQueue.java:102)
> at
> org.apache.lucene.search.MinShouldMatchSumScorer.advanceTail(MinShouldMatchSumSc
> orer.java:246)
> at
> org.apache.lucene.search.MinShouldMatchSumScorer.updateFreq(MinShouldMatchSumSco
> rer.java:312)
> at
> org.apache.lucene.search.MinShouldMatchSumScorer.score(MinShouldMatchSumScorer.j
> ava:320)
> at
> org.apache.solr.ltr.feature.SolrFeature$SolrFeatureWeight$SolrFeatureScorer.scor
> e(SolrFeature.java:242)
> at
> org.apache.solr.ltr.LTRScoringQuery$ModelWeight$ModelScorer$SparseModelScorer.sc
> ore(LTRScoringQuery.java:595)
> at
> org.apache.solr.ltr.LTRScoringQuery$ModelWeight$ModelScorer.score(LTRScoringQuer
> y.java:540)
> at org.apache.solr.ltr.LTRRescorer.scoreFeatures(LTRRescorer.java:183)
> at org.apache.solr.ltr.LTRRescorer.rescore(LTRRescorer.java:122)
> at org.apache.solr.search.ReRankCollector.topDocs(ReRankCollector.java:119)
>
> Based on [~cpoerschke] suggestions we detailed the following:
> Reproducibility: We can reproduce the same queries on multiple runs, with the
> same error.
> Data as a factor: Our setup is single-sharded, so we can't investigate
> further on this.
> Feature vs. Model: We've also tried a dummy LinearModel with only two
> features and the problem still occurs.
> Identification of the troublesome feature(s): We've narrowed our model to
> only two features and the problem always occurs (for some queries, not all)
> when we have a feature with a mm=1 and a feature with a mm>=3. The problem
> also occurs when we only do feature extraction and the problem seems to
> always occur on the feature with the bigger mm. The errors seem to be related
> to the size of the head DisiPriorityQueue created here:
> https://github.com/apache/lucene-solr/blob/branch_8_6/lucene/core/src/java/org/apache/lucene/search/MinShouldMatchSumScorer.java#L107
> as the error changes as we change the mm for the second feature:
>
> 1 feature with mm=1 and one with mm=3 -> Index 4 out of bounds for length 4
> 1 feature with mm=1 and one with mm=5 -> Index 2 out of bounds for length 2
>
> You can find below the dummy feature-store.
>
> [
> {
> "store": "dummystore",
> "name": "similarity_name_mm_1",
> "class": "org.apache.solr.ltr.feature.SolrFeature",
> "params": {
> "q": "\{!dismax qf=name mm=1}${term}"
> }
> },
> {
> "store": "dummystore",
> "name": "similarity_names_mm_3",
> "class": "org.apache.solr.ltr.feature.SolrFeature",
> "params": {
> "q": "\{!dismax qf=name mm=3}${term}"
> }
> }
> ]
>
> The problem starts occuring in Solr 8.6.0, as we tried multiple versions <
> 8.6 and >= 8.6 and the problem started on 8.6.0 and we tend to believe it's
> because of the following changes:
> https://issues.apache.org/jira/browse/SOLR-14364 as they're the only major
> changes related to LTR which were introduced in Solr 8.6.0.
>
> Thanks.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
