Are you stuck in iowait during that commit?
On Fri, Dec 4, 2020 at 6:28 AM raj.yadav <rajkum...@cse.ism.ac.in> wrote: > > Hi everyone, > > As per suggestions in previous post (by Erick and Shawn) we did following > changes. > > OLD CACHE CONFIG > <filterCache class="solr.CaffeineCache" > size="32768" > initialSize="6000" > autowarmCount="6000"/> > > <queryResultCache class="solr.CaffeineCache" > size="25600" > initialSize="6000" > autowarmCount="0"/> > > <documentCache class="solr.CaffeineCache" > size="32768" > initialSize="6144" > autowarmCount="0"/> > > NEW CACHE CONFIG > <filterCache class="solr.CaffeineCache" > size="8192" > initialSize="512" > autowarmCount="512"/> > > <queryResultCache class="solr.CaffeineCache" > size="8192" > initialSize="3000" > autowarmCount="0"/> > > <documentCache class="solr.CaffeineCache" > size="8192" > initialSize="3072" > autowarmCount="0"/> > > > *Reduced JVM heap size from 30GB to 26GB* > > > > *Currently query request rate on the system is zero. > But still, commit with openSearcher=true is taking 25 mins.* > > We looked into solr logs, and observed the following things: > > 1. /Once the commit is issued, five (shard1_0, shard1_1, shard2_2, shard3_0, > shard3_1) of the six shards have immediately started processing commit but > on one shard (shard2_1) we are getting follwing error:/ > > 2020-12-03 12:29:17.518 ERROR > (updateExecutor-5-thread-6-processing-n:solr_132:8389_solr > x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1 > r:core_node22) [c:my_collection s:shard2_1 r:core_node22 > x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: > http://solr_198:8389/solr/my_collection_shard3_1_replica_n23/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753) > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369) > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) > at > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344) > at > org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.concurrent.TimeoutException > at > org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398) > ... 13 more > > 2020-12-03 12:29:17.518 ERROR > (updateExecutor-5-thread-2-processing-n:solr_132:8389_solr > x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1 > r:core_node22) [c:my_collection s:shard2_1 r:core_node22 > x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: > http://solr_199:8389/solr/my_collection_shard1_0_replica_n7/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753) > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369) > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) > at > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344) > at > org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.concurrent.TimeoutException > at > org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398) > ... 13 more > > 2020-12-03 12:29:17.518 ERROR > (updateExecutor-5-thread-3-processing-n:solr_132:8389_solr > x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1 > r:core_node22) [c:my_collection s:shard2_1 r:core_node22 > x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: > http://solr_200:8389/solr/my_collection_shard1_1_replica_n19/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753) > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369) > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) > at > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344) > at > org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.concurrent.TimeoutException > at > org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398) > ... 13 more > > 2020-12-03 12:29:17.518 ERROR > (updateExecutor-5-thread-5-processing-n:solr_132:8389_solr > x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1 > r:core_node22) [c:my_collection s:shard2_1 r:core_node22 > x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: > http://solr_133:8389/solr/my_collection_shard3_0_replica_n15/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753) > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369) > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) > at > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344) > at > org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.concurrent.TimeoutException > at > org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398) > ... 13 more > > 2020-12-03 12:29:17.518 ERROR > (updateExecutor-5-thread-4-processing-n:solr_132:8389_solr > x:my_collection_shard2_1_replica_n21 c:my_collection s:shard2_1 > r:core_node22) [c:my_collection s:shard2_1 r:core_node22 > x:my_collection_shard2_1_replica_n21] o.a.s.u.SolrCmdDistributor > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: > http://solr_254:8389/solr/my_collection_shard2_0_replica_n11/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_132%3A8389%2Fsolr%2Fmy_collection_shard2_1_replica_n21%2F > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753) > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369) > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290) > at > org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344) > at > org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.util.concurrent.TimeoutException > at > org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216) > at > org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398) > ... 13 more > > 2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection > s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21] > o.a.s.u.SolrCmdDistributor SolrCmdDistributor found 5 errors > 2020-12-03 12:29:17.522 INFO (qtp2034610694-29830) [c:my_collection > s:shard2_1 r:core_node22 x:my_collection_shard2_1_replica_n21] > o.a.s.u.DirectUpdateHandler2 start > commit{_version_=1685060119452188672,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} > -- snipped -- > > > > -- > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
