Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions
2019-07-09 17:37:30.053 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore [promotion_rules-v1_shard1_replica_n1] **CLOSING SolrCore org.apache.solr.core.SolrCore**@44de0f89 2019-07-09 17:37:30.054 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.promotion_rules-v1.shard1.replica_n1, tag=44de0f89 2019-07-09 17:37:30.056 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@36a60356: rootName = null, domain = solr.core.promotion_rules-v1.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.promotion_rules-v1.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@79be463d 2019-07-09 17:37:30.148 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.promotion_rules-v1.shard1.leader, tag=44de0f89 2019-07-09 17:37:30.159 INFO (facetExecutor-67-thread-22-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.u.p.DocExpirationUpdateProcessorFactory Triggering Graceful close of DocExpiration Executor 2019-07-09 17:38:03.062 **ERROR (facetExecutor-67-thread-26-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-4] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org** 2019-07-09 17:38:03.067 ERROR (facetExecutor-67-thread-28-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] **o.a.s.c.SolrCore Too many close [count:-5] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org** 2019-07-09 17:38:06.176 ERROR (facetExecutor-67-thread-25-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-7] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:38:07.602 ERROR (facetExecutor-67-thread-24-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-6] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:38:11.868 ERROR (facetExecutor-67-thread-23-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-8] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:38:32.522 ERROR (facetExecutor-67-thread-4-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] **o.a.s.c.SolrCore Too many close [count:-9] on** org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:39:08.862 ERROR (facetExecutor-67-thread-3-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-10] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:39:08.988 ERROR (facetExecutor-67-thread-6-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-12] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:39:09.048 ERROR (facetExecutor-67-thread-5-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-13] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:39:10.474 ERROR (facetExecutor-67-thread-2-processing-n:127.0.0.1:8983_solr x:products_comp-v1_shard1_replica_n21 c:products_comp-v1 s:shard1 r:core_node22) [ ] o.a.s.c.SolrCore Too many close [count:-11] on org.apache.solr.core.SolrCore@44de0f89. Please report this exception to solr-user@lucene.apache.org 2019-07-09 17:39:10.572 ERROR (facetExecutor-67-thread-7-processing-n
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions
SolrQuery used:
q=*:*&defType=edismax&fl=comboid,skuid&start=0&rows=15&facet.limit=100&variant=A&debugQuery=true&debug.explain.structured=true&fq=!status_facet:inactive&fq=!mode:AF_ONLY&fq=**{!join+from=skuid+to=skuid+fromIndex=promotion_rules}id:promo-opera-test-1000_promo-opera-test-1000**&enableManagedResourceRange=true&facet=true&facet.mincount=1&**facet.threads=32**&facet.field=dynamiccategory_facet&facet.recommend.maxfacets=30&facet.recommend.debug=true&sort=ratingandreviewsort+desc&qt=/promo&timeAllowed=5000
We are migrating to 7.6. We have been using joins with multithreaded facets
for years now.
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions
Few deatils added.
public void close() {
int count = refCount.decrementAndGet();
if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
if (count < 0) {
log.error("Too many close [count:{}] on {}. Please report this
exception to solr-user@lucene.apache.org", count, this );
assert false : "Too many closes on SolrCore";
return;
}
log.info("{} CLOSING SolrCore {}", logid, this);
// stop reporting metrics
try {
coreMetricManager.close();
} catch (Throwable e) {
SolrException.log(log, e);
if (e instanceof Error) {
throw (Error) e;
}
}}
The above code has refCount in SolrCore.java this atomicRef is in negative
when we do multi threaded facets and joins.
Can anyone explain the semaphore implemenattion here. What would be scenario
when this can go below 0. I can clearly see each thread of multithreaded
facet trying to close the core.
Every Thread of facet tries to call createWeight of JoinQParserPlugin and
attaches closeHooks which getscalled from SolrRequestInfo.clearRequestInfo()
.
if (fromRef != null) {
final RefCounted ref = fromRef;
info.addCloseHook(new Closeable() {
@Override
public void close() {
ref.decref();
}
});
}
info.addCloseHook(new Closeable() {
@Override
public void close() {
fromCore.close();
}
});
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions
Few deatils added.
public void close() {
int count = refCount.decrementAndGet();
if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
if (count < 0) {
log.error("Too many close [count:{}] on {}. Please report this
exception to solr-user@lucene.apache.org", count, this );
assert false : "Too many closes on SolrCore";
return;
}
log.info("{} CLOSING SolrCore {}", logid, this);
// stop reporting metrics
try {
coreMetricManager.close();
} catch (Throwable e) {
SolrException.log(log, e);
if (e instanceof Error) {
throw (Error) e;
}
}}
The above code has refCount in SolrCore.java this atomicRef is in negative
when we do multi threaded facets and joins.
Can anyone explain the semaphore implemenattion here. What would be scenario
when this can go below 0. I can clearly see each thread of multithreaded
facet trying to close the core.
Every Thread of facet tries to call createWeight of JoinQParserPlugin and
attaches closeHooks which getscalled from SolrRequestInfo.clearRequestInfo()
.
if (fromRef != null) {
final RefCounted ref = fromRef;
info.addCloseHook(new Closeable() {
@Override
public void close() {
ref.decref();
}
});
}
info.addCloseHook(new Closeable() {
@Override
public void close() {
fromCore.close();
}
});
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr 7.6 JoinQueryParser with Muti Threaded Facet stops solr core with Exceptions
I dint quite get your question.Our Solr collection under contention is not shareded. Data being joint is collocated.INfact multithreaded facet help our performance. Afer debugging more solr 7 calls JoinQParserPlugin createWeight is called by each thread of multithreaded facets but not in solr 6. Is this something to do with changes to ExecutorUtils awaitTermination? -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Solr behaves wonky when zookeeper quorom is messed up.
In our PROD SOLR cluster(7.6 and ZK:3.4.9) when Zookeeper leader fails Zookeeper enter an infinite leader election loop which makes SOLR instable. Solr Fails to index(as Expected with error "Remote error message: Cannot talk to ZooKeeper - Updates are disabled") and CPU spikes up. This is a know ZK bug https://issues.apache.org/jira/browse/ZOOKEEPER-2164. below are my question: 1. Solr behaving Wonky when ZK quorom is affected is expected but is there a work around? 2. The ZK bug is fixed in 3.5.6 is solr7.6 compatible with ZK 3.5.6? 3. Anyone have an opinion around "fast leader election" vs "original UDP-based" of ZK. Will cahnge election algorithm version from 3->0 solve the issue? -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Solr behaves wonky when zookeeper quorom is messed up.
If ZK loses quorum, Solr goes read-only. That's how it's designed to work. I don't know of any workaround for that. That makes sense. CPU spiking in solr is because solr's index calls are holding threads because zookeeper is down as per solr? -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
