Hello. Recently we upgraded to SOLR 8.4.1 from 6.4. After the upgrade we started getting errors during our indexing process.
We have 3 node solr cloud cluster with main-2 (main-index via alias) collection replicated on each node. We have standalone app running in cycle, it will query our DB to load all docs and put them into the main-index collection. After all docs are indexed we switch main-index into main-query and vice versa and start the process again. This worked fine on 6.4. After upgrade to 8.4.1 we started getting socket timeouts in our indexing logic. ERROR c.i.w.s.i.c.i.IndexerServiceImpl - Cannot perform reindex com.idc.webchannel.search.indexer.dao.TraversableResult$ResultProcessingException: org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: http://solr-prod2.aws.idc.com:8983/solr/main-index When I go to logs in the node in solr I get 2020-04-08 08:20:57.559 WARN (qtp2082351661-23) [c:main-2 s:shard1 r:core_node6 x:main-2_shard1_replica_n5] o.a.s.u.SolrCmdDistributor Unable to finish sending updates => java.io.IOException: Task queue processing has stalled for 20124 ms with 0 remaining elements to process. at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501) java.io.IOException: Task queue processing has stalled for 20124 ms with 0 remaining elements to process. at org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.blockUntilFinished(ConcurrentUpdateHttp2SolrClient.java:501) ~[?:?] at org.apache.solr.update.StreamingSolrClients.blockUntilFinished(StreamingSolrClients.java:87) ~[?:?] at org.apache.solr.update.SolrCmdDistributor.blockAndDoRetries(SolrCmdDistributor.java:265) ~[?:?] at org.apache.solr.update.SolrCmdDistributor.finish(SolrCmdDistributor.java:99) ~[?:?] We ran 8.4.1 on QA for a while and did some load testing, etc and it never happened there. Is something installed incorrectly? Any idea here would help as I do not know what is happening. -- We also slightingly increased the amount of data being index, but again it worked fine on QA. -- CPU utilization of our solr nodes went up. [cid:image003.jpg@01D60D94.880ECAC0] Thanks. Pavel
