Hello,
One of our Solr 6.6.2 DR cluster (target CDCR) which even doesn't have any
live search load seems to be taking 600000 ms many times for the ping /
health check calls. Anyone has seen this before/suggestion what could be
wrong. The collection has 8 shards/3 replicas and 64GB memory and index
seems to fit in memory. Below solr log entries.
solr.log.26:2020-08-13 14:03:20.827 INFO (qtp1775120226-46486) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2}
hits=62569458 status=0 QTime=600113
solr.log.26:2020-08-13 14:03:20.827 WARN (qtp1775120226-46486) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2}
hits=62569458 status=0 QTime=600113
solr.log.26:2020-08-13 14:03:20.827 INFO (qtp1775120226-46486) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2} status=0
QTime=600113
solr.log.26:2020-08-13 14:03:20.827 WARN (qtp1775120226-46486) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2} status=0
QTime=600113
solr.log.38:2020-08-08 15:01:45.640 INFO (qtp1775120226-46254) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2}
hits=62221186 status=0 QTime=600092
solr.log.38:2020-08-08 15:01:45.640 WARN (qtp1775120226-46254) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2}
hits=62221186 status=0 QTime=600092
solr.log.38:2020-08-08 15:01:45.640 INFO (qtp1775120226-46254) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2} status=0
QTime=600092
solr.log.38:2020-08-08 15:01:45.640 WARN (qtp1775120226-46254) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.SolrCore slow:
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2} status=0
QTime=600092
solr.log.39:2020-08-08 13:20:12.117 INFO (qtp1775120226-46254) [c:COLL
s:shard1 r:core_node19 x:COLL_shard1_replica1] o.a.s.c.S.Request
[COLL_shard1_replica1] webapp=/solr path=/admin/ping
params={distrib=true&_stateVer_=COLL:3032&wt=javabin&version=2}
hits=63094900 status=0 QTime=600095
server1:/home/kumar # curl --location --request GET '
http://server1:8080/solr/COLL/admin/ping?distrib=true'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><bool name="zkConnected">true</bool><int
name="status">0</int><int name="QTime">600095</int><lst name="params"><str
name="q">{!lucene}*:*</str><str name="distrib">true</str><str
name="df">wordTokens</str><str name="preferLocalShards">false</str><str
name="rows">10</str><str name="echoParams">all</str></lst></lst><str
name="status">OK</str>
</response>
