Elevation with distributed search causes NPE
Hi all,
I'm facing the problem that Solr is throwing a NullPointerException when
performing a distributed search with multiple shards having elevation
configured where one or more shards do have elevated results but others do not.
We are using Solr 8.2 and have the QueryElevationComponent configured with
"last-components" of the default search handler "/select". But the problem also
occurs when using the explicit "/elevate" search handler.
...
elevator
...
string
elevate.xml
### Steps to reproduce:
(1) Add entries to the elevate.xml of each core to elevate a specific document
for the text "searchTerm":
core1:
...
core2:
...
(2) Execute query (we use port 9983):
http://localhost:9983/solr/web/select?q=elevatedTerm&lowercaseOperators=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/solr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&rows=10&start=0
Now as both shards have elevated documents for the requested "searchTerm" the
search results are as expected:
response: {
numFound: 5192,
start: 0,
maxScore: 1.9032197,
docs: [{
area: "press",
id: "core1docId1",
[elevated]: true,
[shard]: "localhost:9983/solr/core1"
}, {
area: "products",
id: "core2docId1",
[elevated]: true,
[shard]: "localhost:9983/solr/core2"
}, {
area: "press",
id: "core1docId2",
[elevated]: false,
[shard]: "localhost:9983/solr/core1"
},
...
(3) Remove the elevation entry for that "searchTerm" from one of the cores,
e.g. via comment
core2:
...
(4) Reload the modified core:
http://localhost:9983/solr/admin/cores?action=RELOAD&core=core2
(5) Request same query again and you get the NPE:
error: {
trace: "java.lang.NullPointerException
at
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1068)
at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
at
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:566)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
...
I want to ask the community if I'm missing something or if this really is a bug.
Thanks in advance,
Marc
Marc Linden
Developer
Virtual Identity AG
Grünwälderstraße 10-14
79098 Freiburg
+49 761 20758-422
--
marc.lin...@virtual-identity.com
http://www.virtual-identity.com
Freiburg | München | Wien | Porto
Virtual Identity AG
Grünwälderstraße 10-14
79098 Freiburg
Amtsgericht Freiburg, HRB 6218
Vorstand: Ralf Heller
Vorsitzende des Aufsichtsrates: Kirsten Heller
Umsatzsteuer-ID: DE208002218
AW: Elevation with distributed search causes NPE
Thanks Erick for your fast response.
I've checked out adding the sort param and yes that vanished the problem but it
also disables elevation if I'm not mistaken. So after adding
forceElevation=true to the query then a ClassCastException was thrown:
http://localhost:9983/solr/core1/select?q=elevatedTerm&lowercaseOperators=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/solr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&rows=10&start=0&sort=area%20asc&forceElevation=true
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.String
at
org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.java:1229)
at org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
at
org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1092)
at
org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
at
org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
at
org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:431)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
...
Best regards,
Marc
-Ursprüngliche Nachricht-
Von: Erick Erickson
Gesendet: Mittwoch, 15. Juli 2020 14:32
An: solr-user@lucene.apache.org
Betreff: Re: Elevation with distributed search causes NPE
Hmmm, looking at the code that line looks like this:
sortSpec.getSort().getSort();
I’m curious what happens if you specify a sort on the query? If that makes the
problem go away, it’s a smoking gun.
Whether or not adding sorting makes the problem go away, this looks like
something that’s a legitimate JIRA, please go ahead and raise one.
Best,
Erick
> On Jul 15, 2020, at 4:34 AM, Marc Linden
> wrote:
>
> Hi all,
>
> I'm facing the problem that Solr is throwing a NullPointerException when
> performing a distributed search with multiple shards having elevation
> configured where one or more shards do have elevated results but others do
> not.
>
> We are using Solr 8.2 and have the QueryElevationComponent configured with
> "last-components" of the default search handler "/select". But the problem
> also occurs when using the explicit "/elevate" search handler.
> ...
>
> elevator
>
>
> ...
> >
> name="queryFieldType">string
> elevate.xml
>
>
> ### Steps to reproduce:
>
> (1) Add entries to the elevate.xml of each core to elevate a specific
> document for the text "searchTerm":
>
> core1:
>
> ...
>
>
> core2:
>
> ...
>
>
>
> (2) Execute query (we use port 9983):
> http://localhost:9983/solr/web/select?q=elevatedTerm&lowercaseOperator
> s=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/so
> lr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&rows=
> 10&start=0
>
> Now as both shards have elevated documents for the requested "searchTerm" the
> search results are as expected:
>
> response: {
> numFound: 5192,
> start: 0,
> maxScore: 1.9032197,
> docs: [{
> area: "press",
> id: "core1docId1",
> [elevated]: true,
> [shard]: "localhost:9983/solr/core1"
> }, {
> area: "products",
> id: "core2docId1",
> [elevated]: true,
> [shard]: "localhost:9983/solr/core2"
> }, {
> area: "press",
> id: "core1docId2",
> [elevated]: false,
> [shard]: "localhost:9983/solr/core1"
> },
> ...
>
> (3) Remove the elevation entry for that "searchTerm" from one of the
> cores, e.g. via comment
>
> core2:
>
> ...
>
>
>
>
> (4) Reload the modified core:
> http://localhost:9983/solr/admin/cores?action=RELOAD&core=core2
>
> (5) Request same query again and you get the NPE:
>
> error: {
> trace: "java.lang.NullPointerException
> at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(QueryComponent.java:1068)
> at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:917)
> at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:613)
> at
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:592)
>
AW: Elevation with distributed search causes NPE
There it is: https://issues.apache.org/jira/browse/SOLR-14662. I'd love to dig
deeper into that and provide a patch for it, but I'm fully occupied with our
own project.
Thanks and best regards,
Marc
-Ursprüngliche Nachricht-
Von: Erick Erickson
Gesendet: Donnerstag, 16. Juli 2020 15:06
An: solr-user@lucene.apache.org
Betreff: Re: Elevation with distributed search causes NPE
Can you raise a JIRA? If you’re ambitious, you can add a patch too ;)
> On Jul 16, 2020, at 2:52 AM, Marc Linden
> wrote:
>
> Thanks Erick for your fast response.
>
> I've checked out adding the sort param and yes that vanished the problem but
> it also disables elevation if I'm not mistaken. So after adding
> forceElevation=true to the query then a ClassCastException was thrown:
>
> http://localhost:9983/solr/core1/select?q=elevatedTerm&lowercaseOperat
> ors=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/
> solr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&row
> s=10&start=0&sort=area%20asc&forceElevation=true
>
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.String at
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.ja
> va:1229) at
> org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
> at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(Q
> ueryComponent.java:1092) at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryCompone
> nt.java:917) at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponse
> s(QueryComponent.java:613) at
> org.apache.solr.handler.component.QueryComponent.handleResponses(Query
> Component.java:592) at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
> chHandler.java:431) at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rBase.java:199) at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
> ...
>
> Best regards,
> Marc
>
> -Ursprüngliche Nachricht-
> Von: Erick Erickson
> Gesendet: Mittwoch, 15. Juli 2020 14:32
> An: solr-user@lucene.apache.org
> Betreff: Re: Elevation with distributed search causes NPE
>
> Hmmm, looking at the code that line looks like this:
>
> sortSpec.getSort().getSort();
>
> I’m curious what happens if you specify a sort on the query? If that makes
> the problem go away, it’s a smoking gun.
>
> Whether or not adding sorting makes the problem go away, this looks like
> something that’s a legitimate JIRA, please go ahead and raise one.
>
> Best,
> Erick
>
>> On Jul 15, 2020, at 4:34 AM, Marc Linden
>> wrote:
>>
>> Hi all,
>>
>> I'm facing the problem that Solr is throwing a NullPointerException when
>> performing a distributed search with multiple shards having elevation
>> configured where one or more shards do have elevated results but others do
>> not.
>>
>> We are using Solr 8.2 and have the QueryElevationComponent configured with
>> "last-components" of the default search handler "/select". But the problem
>> also occurs when using the explicit "/elevate" search handler.
>> ...
>>
>> elevator
>>
>>
>> ...
>> >>
>> > name="queryFieldType">string
>> elevate.xml
>>
>>
>> ### Steps to reproduce:
>>
>> (1) Add entries to the elevate.xml of each core to elevate a specific
>> document for the text "searchTerm":
>>
>> core1:
>>
>> ...
>>
>> core2:
>>
>> ...
>>
>>
>> (2) Execute query (we use port 9983):
>> http://localhost:9983/solr/web/select?q=elevatedTerm&lowercaseOperato
>> r
>> s=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/s
>> o
>> lr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&rows
>> =
>> 10&start=0
>>
>> Now as both shards have elevated documents for the requested "searchTerm"
>> the search results are as expected:
>>
>> response: {
>> numFound: 5192,
>> start: 0,
>> maxScore: 1.9032197,
>> docs: [{
>> area: "press",
>> id: "core1docId1",
>> [elevated]: true,
>> [shard]: "localhost:9983/solr/core1"
>> }, {
>> area: "products",
>> id: "core2docId1",
>> [elevated]: true,
>> [shard]: "localhost:9983/solr/core2&q
AW: Elevation with distributed search causes NPE
Hi Erick,
could you tell me or point me to the documentation on how the process around
raised JIRA issues usually works. I could not find anything that helped me.
Thanks in advance.
Best,
Marc
-Ursprüngliche Nachricht-
Von: Marc Linden
Gesendet: Freitag, 17. Juli 2020 11:35
An: solr-user@lucene.apache.org
Betreff: AW: Elevation with distributed search causes NPE
There it is: https://issues.apache.org/jira/browse/SOLR-14662. I'd love to dig
deeper into that and provide a patch for it, but I'm fully occupied with our
own project.
Thanks and best regards,
Marc
-Ursprüngliche Nachricht-
Von: Erick Erickson
Gesendet: Donnerstag, 16. Juli 2020 15:06
An: solr-user@lucene.apache.org
Betreff: Re: Elevation with distributed search causes NPE
Can you raise a JIRA? If you’re ambitious, you can add a patch too ;)
> On Jul 16, 2020, at 2:52 AM, Marc Linden
> wrote:
>
> Thanks Erick for your fast response.
>
> I've checked out adding the sort param and yes that vanished the problem but
> it also disables elevation if I'm not mistaken. So after adding
> forceElevation=true to the query then a ClassCastException was thrown:
>
> http://localhost:9983/solr/core1/select?q=elevatedTerm&lowercaseOperat
> ors=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/
> solr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&row
> s=10&start=0&sort=area%20asc&forceElevation=true
>
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.String at
> org.apache.solr.schema.FieldType.unmarshalStringSortValue(FieldType.ja
> va:1229) at
> org.apache.solr.schema.StrField.unmarshalSortValue(StrField.java:122)
> at
> org.apache.solr.handler.component.QueryComponent.unmarshalSortValues(Q
> ueryComponent.java:1092) at
> org.apache.solr.handler.component.QueryComponent.mergeIds(QueryCompone
> nt.java:917) at
> org.apache.solr.handler.component.QueryComponent.handleRegularResponse
> s(QueryComponent.java:613) at
> org.apache.solr.handler.component.QueryComponent.handleResponses(Query
> Component.java:592) at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(Sear
> chHandler.java:431) at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandle
> rBase.java:199) at
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2578)
> at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:780)
> ...
>
> Best regards,
> Marc
>
> -Ursprüngliche Nachricht-
> Von: Erick Erickson
> Gesendet: Mittwoch, 15. Juli 2020 14:32
> An: solr-user@lucene.apache.org
> Betreff: Re: Elevation with distributed search causes NPE
>
> Hmmm, looking at the code that line looks like this:
>
> sortSpec.getSort().getSort();
>
> I’m curious what happens if you specify a sort on the query? If that makes
> the problem go away, it’s a smoking gun.
>
> Whether or not adding sorting makes the problem go away, this looks like
> something that’s a legitimate JIRA, please go ahead and raise one.
>
> Best,
> Erick
>
>> On Jul 15, 2020, at 4:34 AM, Marc Linden
>> wrote:
>>
>> Hi all,
>>
>> I'm facing the problem that Solr is throwing a NullPointerException when
>> performing a distributed search with multiple shards having elevation
>> configured where one or more shards do have elevated results but others do
>> not.
>>
>> We are using Solr 8.2 and have the QueryElevationComponent configured with
>> "last-components" of the default search handler "/select". But the problem
>> also occurs when using the explicit "/elevate" search handler.
>> ...
>>
>> elevator
>>
>>
>> ...
>> >>
>> > name="queryFieldType">string
>> elevate.xml
>>
>>
>> ### Steps to reproduce:
>>
>> (1) Add entries to the elevate.xml of each core to elevate a specific
>> document for the text "searchTerm":
>>
>> core1:
>>
>> ...
>>
>> core2:
>>
>> ...
>>
>>
>> (2) Execute query (we use port 9983):
>> http://localhost:9983/solr/web/select?q=elevatedTerm&lowercaseOperato
>> r
>> s=false&df=text_en&defType=edismax&fq=lang:en&shards=localhost:9983/s
>> o
>> lr/core1,localhost:9983/solr/core2&fl=[elevated],[shard],area,id&rows
>> =
>> 10&start=0
>>
>> Now as both shards have elevated documents for the requested "searchTerm"
>> the search results are as expected:
>>
>> response: {
>> numFound: 5192,
>> start:
