solr 3.4 with nTiers >= 2: usage of ids param causes NullPointerException (NPE)

[Dmitry Kan]

Hello,

Hopefully this question is not too complex to handle, but I'm currently
stuck with it.

We have a system with nTiers, that is:

Solr front base ---> Solr front --> shards

Inside QueryComponent there is a method createRetrieveDocs(ResponseBuilder
rb) which collects doc ids of each shard and sends them in different
queries using the ids parameter:

[code]
sreq.params.add(ShardParams.IDS, StrUtils.join(ids, ','));
[/code]

This actually produces NPE (same as in
https://issues.apache.org/jira/browse/SOLR-1477) in the first tier, because
Solr front (on the second tier) fails to process such a query. I have tried
to fix this by using a unique field with a value of ids ORed (the following
code substitutes the code above):

[code]
      StringBuffer idsORed = new StringBuffer();
      for (Iterator<String> iterator = ids.iterator(); iterator.hasNext();
) {
        String next = iterator.next();

        if (iterator.hasNext()) {
          idsORed.append(next).append(" OR ");
        } else {
          idsORed.append(next);
        }
      }

      sreq.params.add(rb.req.getSchema().getUniqueKeyField().getName(),
idsORed.toString());
[/code]

This works perfectly if for rows=n there is n or less hits from a
distributed query. However, if there are more than 2*n hits, the querying
fails with an NPE in a completely different component, which is
HighlightComponent (highlights are requested in the same query with
hl=true&hl.fragsize=50000&hl.requireFieldMatch=true&hl.fl=targetTextField):

SEVERE: java.lang.NullPointerException
        at
org.apache.solr.handler.component.HighlightComponent.finishStage(HighlightComponent.java:161)
        at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)
        at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1368)
        at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
        at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:619)

It sounds like the ids of documents somehow get shuffled and the
instruction (only a hypothesis)

[code]
ShardDoc sdoc = rb.resultIds.get(id);
[/code]

returns sdoc=null, which causes the next line of code to fail with an NPE:

[code]
int idx = sdoc.positionInResponse;
[/code]

Am I missing anything? Can something be done for solving this issue?

Thanks.

-- 
Regards,

Dmitry Kan