Hi everyone,
Environment:
Solr 7.5.0, cloud mode (but problem should be identical in multiple
versions, at least in 7.x)
Summary:
We have a Solr configuration that returns suggestions in the course of a
normal search call (i.e., we have a 'suggest' component added to the
'last-components' for '/select' request handler). This does not work in
cloud mode, where we get an NPE in QueryComponent. This problem seems to
have been reported in various forms in the past -- see for example [1]
and [2] (links at the end of this email) -- but we couldn't find any
resolution (or in-depth discussion for that matter).
In more detail:
We have a suggest component configured as follows:
<searchComponent name="suggest" class="solr.SuggestComponent" >
<lst name="suggester">
<str name="name">default</str>
<str
name="classname">org.apache.solr.spelling.suggest.Suggester</str>
<str
name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
<str name="storeDir">dict_default</str>
<str name="suggestAnalyzerFieldType">text_suggest</str>
<str name="field">text_suggest</str>
<str name="buildOnStartup">true</str>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
</lst>
<lst name="suggester">
<str name="name">suggest_phrase</str>
<str
name="lookupImpl">org.apache.solr.spelling.suggest.fst.AnalyzingLookupFactory</str>
<str name="storeDir">dict_suggest_phrase</str>
<str
name="suggestAnalyzerFieldType">text_suggest_phrase</str>
<str name="field">suggest_phrase</str>
<str name="buildOnStartup">true</str>
<str name="buildOnCommit">true</str>
<str name="buildOnOptimize">true</str>
</lst>
<lst name="suggester">
<str name="name">suggest_infix_shingle</str>
<str name="lookupImpl">AnalyzingInfixLookupFactory</str>
<str name="indexPath">suggestInfixShingleDir</str>
<str
name="suggestAnalyzerFieldType">text_suggest_phrase</str>
<str name="field">suggest_phrase</str>
<str name="buildOnOptimize">true</str>
<str name="buildOnCommit">true</str>
<str name="buildOnStartup">true</str>
</lst>
<lst name="suggester">
<str name="name">suggest_prefix</str>
<str name="classname">Suggester</str>
<str name="lookupImpl">AnalyzingLookupFactory</str>
<str
name="suggestAnalyzerFieldType">text_suggest_prefix</str>
<str name="field">suggest_prefix</str>
<str name="buildOnOptimize">true</str>
<str name="buildOnCommit">true</str>
<str name="buildOnStartup">true</str>
</lst>
</searchComponent>
This component works without issued both in standalone and cloud mode,
when used as the sole component in a handler, such as in the following
excerpt:
<requestHandler name="/suggest" class="solr.SearchHandler"
startup="lazy">
<lst name="defaults">
<str name="suggest.dictionary">default</str>
<str name="suggest.dictionary">suggest_phrase</str>
<str
name="suggest.dictionary">suggest_infix_shingle</str>
<str name="suggest.dictionary">suggest_prefix</str>
<str name="suggest">true</str>
<str name="suggest.count">10</str>
<str name="suggest.highlight">false</str>
</lst>
<arr name="components">
<str>suggest</str>
</arr>
</requestHandler>
It also works when used along with other component in standalone mode,
such as in the following excerpt, where we use the suggest component to
get suggestions during a "normal" search call:
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">text_search</str>
<str name="defType">edismax</str>
<str name="qf">title^5.0 subtitle^3.0
abstract^2.0 text_search</str>
<str name="pf">title^5.0 subtitle^3.0
abstract^2.0 text_search</str>
<str name="ps">4</str>
<str name="spellcheck">on</str>
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.alternativeTermCount">5</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str
name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">10</str>
<str name="spellcheck.maxCollations">5</str>
<str name="suggest.dictionary">default</str>
<str name="suggest.dictionary">suggest_phrase</str>
<str
name="suggest.dictionary">suggest_infix_shingle</str>
<str name="suggest.dictionary">suggest_prefix</str>
<str name="suggest">true</str>
<str name="suggest.count">10</str>
<str name="suggest.highlight">false</str>
</lst>
<arr name="last-components">
<str>suggest</str>
<str>spellcheck</str>
</arr>
</requestHandler>
However, the above configuration does not work in cloud mode, where we
get an NPE if a search call is made:
o.a.s.s.HttpSolrCall null:java.lang.NullPointerException
at
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1034)
at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:885)
at
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585)
at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2541)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:709)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
As far as we can tell, this is due to the fact that the failing code in
QueryComponent assumes that the shard response always contains a
'sort_values' section:
NamedList sortFieldValues =
(NamedList)(srsp.getSolrResponse().getResponse().get("sort_values"));
NamedList unmarshalledSortFieldValues = unmarshalSortValues(ss,
sortFieldValues, schema);
(the exception occurs in unmarshalSortValues(), which assumes the
sortFieldValues parameter is not null)
Are we doing something wrong that is preventing 'sort_values' from being
present in the shard response? Or is their absence OK, but the code
fails to account for it? Our guess is the second, because
umarshalSortValues() does check if the values are empty or not, and
handles that.
Can anyone help us with this issue? Thanks in advance for any pointers!
Alex
[1]
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201711.mbox/%3CCACKWnqOv=hfqwtxd7sweqstj-im+43fe+c9_zbtcpmwsp8v...@mail.gmail.com%3E
[2] https://issues.apache.org/jira/browse/SOLR-12060
