Do you have firewall rules between the servers, like only allow these servers to communicate to each other on these solr/zookeeper ports. Is it possible these rules may have been deactivated somehow, i.e. OS patches resetting iptables.
-----Original Message----- From: Shawn Heisey <apa...@elyograg.org> Sent: Saturday, August 15, 2020 7:33 AM To: solr-user@lucene.apache.org Subject: Re: Solr ping taking 600 seconds On 8/14/2020 3:39 PM, Susheel Kumar wrote: > One of our Solr 6.6.2 DR cluster (target CDCR) which even doesn't have > any live search load seems to be taking 600000 ms many times for the > ping / health check calls. Anyone has seen this before/suggestion what > could be wrong. The collection has 8 shards/3 replicas and 64GB memory > and index seems to fit in memory. Below solr log entries. 10 minutes for an all docs query is a REALLY long time.On a properly sized and tuned system, that query should complete in far less than one second, and when Solr has the query cached, it might have a QTime in the single digits. What happens if you manually send a query to the standard handler (usually /select) where the whole q parameter is just the "*:*" text, with no other parameters? At the following location in one of our wiki pages, there are instructions for getting a screenshot showing some detailed information about the system: https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue The rest of that wiki page has some good info about performance problems. It's worth reading. Disclaimer: I wrote it. I typically see one of two problems causing such massive performance issues: 1) Huge GC pauses, usually caused by a heap that's too small. 2) Too little memory in the system available for disk caching. The screenshot described at the link above will tell us a LOT about your system, usually enough to pinpoint the cause of major performance problems. Once you have the screenshot, please be aware that sending it as an email attachment will not work. The mailing list strips most attachments as it processes your email, so you will need to upload the file to some kind of hosting and include a link to it in your email. I normally use dropbox for that, and there are many other services. Thanks, Shawn
