Updating the thread with further findings: So turns out that the nodes hosting Solr are VMs with Virtual disks. Additionally, a Windows system process (the infamous PID 4) is hogging a lot of disk. This is indicated by disk reponse times in excess of 100 ms and a disk drive queue length of 5 which would be considered very high. The indexing is running in two parallel threads each sending a batch of 50 docs per request. I would like to believe this is not too high (?). The docs are not too heavy either, only containing metadata fields.So the disk IO seems to be the bottleneck at this point, causing commits and merges to take more time than they should. This is causing update routing to the leader replica to take more than 10 mins, resulting into read time outs, and eventually failed updates. I could not find anything alarming in the GC logs I shared earlier.
Will update the thread with more findings as I have them and the attempted solutions. At this point I am considering increasing the Jetty timeout and increasing the distribUpdateConnTimeout to a higher value to let the indexing proceed slowly but successfully. In the meantime, would greatly appreciate any other ideas/measures. Thanks, Rahul On Thu, Jun 6, 2019 at 11:00 AM Rahul Goswami <rahul196...@gmail.com> wrote: > Thank you for your responses. Please find additional details about the > setup below: > > We are using Solr 7.2.1 > > > I have a solrcloud setup on Windows server with below config: > > 3 nodes, > > 24 shards with replication factor 2 > > Each node hosts 16 cores. > > 16 CPU cores, or 16 Solr cores? The info may not be all that useful > either way, but just in case, it should be clarified. > > *OP Reply:* 16 Solr cores (i.e. replicas) > > > Index size is 1.4 TB per node > > Xms 8 GB , Xmx 24 GB > > Directory factory used is SimpleFSDirectoryFactory > > How much total memory in the server? Is there other software using > significant levels of memory? > > *OP Reply* : Total 48 GB per node... I couldn't see another software > using a lot of memory. > I am honestly not sure about the reason for change of directory factory to > SimpleFSDirectoryFactory. But I was told that with mmap at one point we > started to see the shared memory usage on Windows go up significantly, > intermittently freezing the system. > Could the choice of DirectoryFactory here be a factor for the long > updates/frequent merges? > > > How many total documents (maxDoc, not numDoc) are in that 1.4 TB of > space? > *OP Reply:* Also, there are nearly 12.8 million total docs (maxDoc, NOT > numDoc) in that 1.4 TB space > > > Can you share the GC log that Solr writes? > *OP Reply:* Please find the GC logs and thread dumps at this location > https://drive.google.com/open?id=1slsYkAcsH7OH-7Pma91k6t5T72-tIPlw > > Another observation is that the CPU usage reaches around 70% (through > manual monitoring) when the indexing starts and the merges are observed. It > is well below 50% otherwise. > > Also, should something be altered with the mergeScheduler setting ? > "mergeScheduler":{ > "class":"org.apache.lucene.index.ConcurrentMergeScheduler", > "maxMergeCount":2, > "maxThreadCount":2}, > > Thanks, > Rahul > > > On Wed, Jun 5, 2019 at 4:24 PM Shawn Heisey <apa...@elyograg.org> wrote: > >> On 6/5/2019 9:39 AM, Rahul Goswami wrote: >> > I have a solrcloud setup on Windows server with below config: >> > 3 nodes, >> > 24 shards with replication factor 2 >> > Each node hosts 16 cores. >> >> 16 CPU cores, or 16 Solr cores? The info may not be all that useful >> either way, but just in case, it should be clarified. >> >> > Index size is 1.4 TB per node >> > Xms 8 GB , Xmx 24 GB >> > Directory factory used is SimpleFSDirectoryFactory >> >> How much total memory in the server? Is there other software using >> significant levels of memory? >> >> Why did you opt to change the DirectoryFactory away from Solr's default? >> The default is chosen with care ... any other choice will probably >> result in lower performance. The default in recent versions of Solr is >> NRTCachingDirectoryFactory, which uses MMap for file access. >> >> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html >> >> The screenshot described here might become useful for more in-depth >> troubleshooting: >> >> >> https://wiki.apache.org/solr/SolrPerformanceProblems#Process_listing_on_Windows >> >> How many total documents (maxDoc, not numDoc) are in that 1.4 TB of space? >> >> > The cloud is all nice and green for the most part. Only when we start >> > indexing, within a few seconds, I start seeing Read timeouts and socket >> > write errors and replica recoveries thereafter. We are indexing in 2 >> > parallel threads in batches of 50 docs per update request. After >> examining >> > the thread dump, I see segment merges happening. My understanding is >> that >> > this is the cause, and the timeouts and recoveries are the symptoms. Is >> my >> > understanding correct? If yes, what steps could I take to help the >> > situation. I do see that the difference between "Num Docs" and "Max >> Docs" >> > is about 20%. >> >> Segment merges are a completely normal part of Lucene's internal >> operation. They should never cause problems like you have described. >> >> My best guess is that a 24GB heap is too small. Or possibly WAY too >> large, although with the index size you have mentioned, that seems >> unlikely. >> >> Can you share the GC log that Solr writes? The problem should occur >> during the timeframe covered by the log, and the log should be as large >> as possible. You'll need to use a file sharing site -- attaching it to >> an email is not going to work. >> >> What version of Solr? >> >> Thanks, >> Shawn >> >
