[
https://issues.apache.org/jira/browse/SOLR-14520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-14520:
--------------------------------------
Attachment: SOLR-14520.patch
Status: Open (was: Open)
attaching a patch file with small changes to TestJsonFacetRefinement that
easily demonstrates one of these types of problems...
{noformat}
[junit4] 2> 8945 ERROR (qtp561920109-35) [ x:collection1 ]
o.a.s.h.RequestHandlerBase
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at null: java.lang.NullPointerException
[junit4] 2> at
org.apache.solr.search.facet.SlotAcc$SumSlotAcc.collect(SlotAcc.java:401)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessor$SpecialSlotAcc.collect(FacetFieldProcessor.java:778)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArrayDV.collect(FacetFieldProcessorByArrayDV.java:335)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArrayDV.collectDocs(FacetFieldProcessorByArrayDV.java:266)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArrayDV.collectDocs(FacetFieldProcessorByArrayDV.java:161)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.calcFacets(FacetFieldProcessorByArray.java:90)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.process(FacetFieldProcessorByArray.java:62)
[junit4] 2> at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:414)
[junit4] 2> at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:478)
[junit4] 2> at
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:434)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessor.refineBucket(FacetFieldProcessor.java:922)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessor.refineFacets(FacetFieldProcessor.java:887)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.calcFacets(FacetFieldProcessorByArray.java:70)
[junit4] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.process(FacetFieldProcessorByArray.java:62)
[junit4] 2> at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:414)
[junit4] 2> at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:478)
[junit4] 2> at
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:434)
[junit4] 2> at
org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:65)
[junit4] 2> at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:414)
[junit4] 2> at
org.apache.solr.search.facet.FacetModule.process(FacetModule.java:150)
[junit4] 2> at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:329)
[junit4] 2> at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209)
[junit4] 2> at
org.apache.solr.core.SolrCore.execute(SolrCore.java:2591)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:803)
[junit4] 2> at
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:582)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:415)
[junit4] 2> at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
[junit4] 2> at
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
[junit4] 2> at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
[junit4] 2> at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
[junit4] 2> at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
[junit4] 2> at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
[junit4] 2> at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
[junit4] 2> at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
[junit4] 2> at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
[junit4] 2> at
org.eclipse.jetty.server.Server.handle(Server.java:500)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
[junit4] 2> at
org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:335)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
[junit4] 2> at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2>
[junit4] 2> at
org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:721)
[junit4] 2> at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:410)
[junit4] 2> at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:740)
[junit4] 2> at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
[junit4] 2> at
org.apache.solr.handler.component.HttpShardHandler.request(HttpShardHandler.java:208)
[junit4] 2> at
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:170)
[junit4] 2> at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[junit4] 2> at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[junit4] 2> at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[junit4] 2> at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
[junit4] 2> at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[junit4] 2> at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2>
[junit4] 2> 8945 INFO (qtp561920109-35) [ x:collection1 ]
o.a.s.c.S.Request [collection1] webapp=/solr path=/select
params={cat_i=cat_i&date=cat_dt&indent=true&cat_s=cat_s&xy_s=xy_s&version=2.2&er_s=er_s&num_d=num_d&qw_s=qw_s&shards=127.0.0.1:40101/solr/collection1,127.0.0.1:33705/solr/collection1,127.0.0.1:37179/solr/collection1&shards=debugQuery&shards=false&q=*:*&json.facet={++cat0:{${terms}+type:terms,+field:${cat_s},+limit:1,+overrequest:0,+refine:false,+allBuckets:true,+facet:{++xy:{${terms}+type:terms,+field:${xy_s},+limit:1,+overrequest:0,+allBuckets:true,+refine:false}++}++},+cat1:{${terms}+type:terms,+field:${cat_s},+limit:1,+overrequest:0,+refine:true,+allBuckets:true,+sort:'f+asc',+facet:{++f:'min(${num_d})'+}++},+cat2:{${terms}+type:terms,+field:${cat_s},+limit:1,+overrequest:0,+refine:true+,+allBuckets:true,+facet:{++xy:{${terms}+type:terms,+field:${xy_s},+limit:1,+overrequest:0,+allBuckets:true,+refine:true+}++}++},+cat3:{${terms}+type:terms,+field:${cat_s},+limit:1,+overrequest:0,+refine:true+,+allBuckets:true,+facet:{++xy:{${terms}+type:terms,+field:${xy_s},+limit:1,+overrequest:0,+allBuckets:true,+refine:true+,+facet:{f:'sum(${num_d})'}+++}++}++},+cat4:{${terms}+type:terms,+field:${cat_s},+limit:1,+overrequest:0,+refine:true+,+allBuckets:true,+facet:{++xy:{${terms}+type:terms,+field:${xy_s},+limit:1,+overrequest:0,+allBuckets:true,+refine:true+,+sort:'f+asc',+facet:{f:'sum(${num_d})'}+++}++}++}}&wt=json}
hits=8 status=500 QTime=13
...
[junit4] ERROR 0.62s | TestJsonFacetRefinement.testBasicRefinement <<<
[junit4] > Throwable #1:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error
from server at http://127.0.0.1:40101/solr/collection1: Server Error
[junit4] > request:
http://127.0.0.1:40101/solr/collection1/select?shards=127.0.0.1:40101/solr/collection1,127.0.0.1:33705/solr/collection1,127.0.0.1:37179/solr/collection1&shards=debugQuery&shards=false&cat_s=cat_s&cat_i=cat_i&date=cat_dt&xy_s=xy_s&num_d=num_d&qw_s=qw_s&er_s=er_s&q=*:*&json.facet={
cat0:{${terms} type:terms, field:${cat_s}, limit:1, overrequest:0,
refine:false, allBuckets:true, facet:{ xy:{${terms} type:terms, field:${xy_s},
limit:1, overrequest:0, allBuckets:true, refine:false} } }, cat1:{${terms}
type:terms, field:${cat_s}, limit:1, overrequest:0, refine:true,
allBuckets:true, sort:'f asc', facet:{ f:'min(${num_d})' } }, cat2:{${terms}
type:terms, field:${cat_s}, limit:1, overrequest:0, refine:true ,
allBuckets:true, facet:{ xy:{${terms} type:terms, field:${xy_s}, limit:1,
overrequest:0, allBuckets:true, refine:true } } }, cat3:{${terms} type:terms,
field:${cat_s}, limit:1, overrequest:0, refine:true , allBuckets:true, facet:{
xy:{${terms} type:terms, field:${xy_s}, limit:1, overrequest:0,
allBuckets:true, refine:true , facet:{f:'sum(${num_d})'} } } },
cat4:{${terms} type:terms, field:${cat_s}, limit:1, overrequest:0, refine:true
, allBuckets:true, facet:{ xy:{${terms} type:terms, field:${xy_s}, limit:1,
overrequest:0, allBuckets:true, refine:true , sort:'f asc',
facet:{f:'sum(${num_d})'} } } }}&wt=json&indent=true&version=2.2
[junit4] > at
__randomizedtesting.SeedInfo.seed([BBFA81426E659210:A96E13741F03B528]:0)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:663)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
[junit4] > at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
[junit4] > at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
[junit4] > at
org.apache.solr.SolrTestCaseHS.getQueryResponse(SolrTestCaseHS.java:229)
[junit4] > at
org.apache.solr.SolrTestCaseHS.getJSON(SolrTestCaseHS.java:250)
[junit4] > at
org.apache.solr.SolrTestCaseHS.assertJQ(SolrTestCaseHS.java:144)
[junit4] > at
org.apache.solr.SolrTestCaseHS$Client$Tester.assertJQ(SolrTestCaseHS.java:280)
[junit4] > at
org.apache.solr.SolrTestCaseHS$Client.testJQ(SolrTestCaseHS.java:326)
[junit4] > at
org.apache.solr.search.facet.TestJsonFacetRefinement.doBasicRefinement(TestJsonFacetRefinement.java:1365)
[junit4] > at
org.apache.solr.search.facet.TestJsonFacetRefinement.testBasicRefinement(TestJsonFacetRefinement.java:1025)
[junit4] > at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
{noformat}
The other type of failure i've seen i have not yet been able to reproduce in
small scale ... you can be see an example found by looking at some of the beast
log attachments in SOLR-14467,but those are based on the ptach in that jira, so
some of hte line numbers won't be exactly accurate.
one example is ...
{noformat}
[beaster] 2> 10513 ERROR (qtp144184226-64) [n:127.0.0.1:43085_solr
c:org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection s:shard2
r:core_node4
x:org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection_shard2_replica_n2
] o.a.s.h.RequestHandlerBase java.lang.ArrayIndexOutOfBoundsException: Index 7
out of bounds for length 1
[beaster] 2> at
org.apache.solr.search.facet.SlotAcc$CountSlotArrAcc.incrementCount(SlotAcc.java:636)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArrayDV.collect(FacetFieldProcessorByArrayDV.java:330)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArrayDV.collectDocs(FacetFieldProcessorByArrayDV.java:303)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArrayDV.collectDocs(FacetFieldProcessorByArrayDV.java:171)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.calcFacets(FacetFieldProcessorByArray.java:90)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.process(FacetFieldProcessorByArray.java:62)
[beaster] 2> at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:414)
[beaster] 2> at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:478)
[beaster] 2> at
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:434)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessor.refineBucket(FacetFieldProcessor.java:938)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessor.refineFacets(FacetFieldProcessor.java:903)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.calcFacets(FacetFieldProcessorByArray.java:70)
[beaster] 2> at
org.apache.solr.search.facet.FacetFieldProcessorByArray.process(FacetFieldProcessorByArray.java:62)
[beaster] 2> at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:414)
[beaster] 2> at
org.apache.solr.search.facet.FacetProcessor.processSubs(FacetProcessor.java:478)
[beaster] 2> at
org.apache.solr.search.facet.FacetProcessor.fillBucket(FacetProcessor.java:434)
[beaster] 2> at
org.apache.solr.search.facet.FacetQueryProcessor.process(FacetQuery.java:65)
[beaster] 2> at
org.apache.solr.search.facet.FacetRequest.process(FacetRequest.java:414)
[beaster] 2> at
org.apache.solr.search.facet.FacetModule.process(FacetModule.java:150)
[beaster] 2> at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:329)
...
[beaster] 2> 10513 INFO (qtp144184226-64) [n:127.0.0.1:43085_solr
c:org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection s:shard2
r:core_node4
x:org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection_shard2_replica_n2
] o.a.s.c.S.Request
[org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection_shard2_replica_n2]
webapp=/solr path=/select
params={df=text&distrib=false&_facet_={"refine":{"facet_1":{"_p":[[29,{"facet_2":{"_l":["23","31","42","43","50","54"]}}],[10,{"facet_2":{"_l":["23","31","34","42","43","50"]}}],[14,{"facet_2":{"_l":["13","14","18","53","8"]}}],[24,{"facet_2":{"_l":["15","23","34","42","47","48","7"]}}],[48,{"facet_2":{"_l":["44","47","48"]}}],[13,{"facet_2":{"_l":["54"]}}],[23,{"facet_2":{"_l":["54"]}}],[35,{"facet_2":{"_l":["15","23","29","30","31","36","42","43","48","50"]}}]]}}}&shards.purpose=2097152&back=(field_9_multi_ss:58+OR+field_6_multi_ss:32+OR+field_11_multi_sdsS:15)&shard.url=http://127.0.0.1:43085/solr/org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv_collection_shard2_replica_n2/&rows=0&fore=(field_8_multi_sdsS:29+OR+field_14_multi_sdsS:30+OR+field_7_multi_sds:26+OR+field_2_multi_sdsS:9+OR+field_1_multi_sds:11+OR+field_3_multi_ss:15+OR+field_10_multi_sds:45)&version=2&q=(field_2_multi_sdsS:6+OR+field_2_multi_sdsS:33+OR+field_2_multi_sdsS:26+OR+field_8_multi_sdsS:10+OR+field_6_multi_ss:50+OR+field_6_multi_ss:19+OR+field_12_multi_ss:50+OR+field_12_multi_ss:56+OR+field_3_multi_ss:23+OR+field_4_multi_sds:5)&json.facet={"facet_1":{"method":"smart","sort":"skg+asc","allBuckets":true,"refine":true,"type":"terms","field":"field_0_multi_is","facet":{"skg":{"type":"func","func":"relatedness($fore,$back)"},"facet_2":{"method":"smart","limit":63,"overrequest":5,"allBuckets":true,"refine":true,"type":"terms","field":"field_4_multi_sds","facet":{"skg":{"type":"func","func":"relatedness($fore,$back)"},"skg0":{"min_popularity":"0.001","type":"func","func":"relatedness($fore,$back)"},"sum":"sum(field_9_multi_is)"}}}},"facet_3":{"method":"smart","limit":25,"overrequest":0,"sort":"count+asc","allBuckets":false,"refine":true,"type":"terms","field":"field_1_solo_id","facet":{"skg":{"type":"func","func":"relatedness($fore,$back)"}}}}&omitHeader=false&NOW=1590620938809&isShard=true&facet=false&wt=javabin}
hits=22 status=500 QTime=1
{noformat}
The meat of that problem seems aparent when looking at the imple of
{{FacetFieldProcessorByArray.calcFacets()}} ...
{code:java}
if (refineResult != null) {
if (freq.allBuckets) {
createAccs(nDocs, 1); // ##### NOTE X ###########################
allBucketsAcc = new SpecialSlotAcc(fcontext, null, -1, accs, 0);
collectDocs(); // ##### NOTE Y ###########################
SimpleOrderedMap<Object> allBuckets = new SimpleOrderedMap<>();
allBuckets.add("count", allBucketsAcc.getSpecialCount());
allBucketsAcc.setValues(allBuckets, -1); // -1 slotNum is unused for
SpecialSlotAcc
refineResult.add("allBuckets", allBuckets);
return refineResult;
}
}
{code}
"NOTE Y" is "{{calcFacets(FacetFieldProcessorByArray.java:90)}}" from the stack
trace above, and results in FacetFieldProcessorByArrayDV trying call
{{collect(int,int,LongValues)}} over *all* of the DV terms (using the ord as
the slotNum) not just for {{allBucketsAcc}} but also {{countAcc}} ... even
though the call to {{createAccs(nDocs, 1)}} at "NOTE X" results in {{countAcc}}
only having a single slot
I'm not really sure how this is _suppose_ to work, or why it doesn't fail more
often, let alone what the best "fix" is.
> json.facets: allBucket:true can cause server errors when combined with
> refine:true
> ----------------------------------------------------------------------------------
>
> Key: SOLR-14520
> URL: https://issues.apache.org/jira/browse/SOLR-14520
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Facet Module
> Reporter: Chris M. Hostetter
> Priority: Major
> Attachments: SOLR-14520.patch
>
>
> Another bug that was discovered while testing SOLR-14467...
> In some situations, using {{allBuckets:true}} in conjunction with
> {{refine:true}} can cause server errors during the "refinement" requests to
> the individual shards -- either NullPointerExceptions from some (nested)
> SlotAccs when SpecialSlotAcc tries to collect them, or
> ArrayIndexOutOfBoundsException from CountSlotArrAcc.incrementCount because
> it's asked to collect to "large" slot# values even though it's been
> initialized with a size of '1'
> NOTE: these problems may be specific to FacetFieldProcessorByArrayDV - i have
> not yet seen similar failures from FacetFieldProcessorByArrayUIF (those are
> the only 2 used when doing refinement) but that may just be a fluke of
> testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
