Hi. We're running Solr 6.6.1 (SolrCloud, 3 nodes). Recently I noticed it became significantly slower to respond and did some basic checks on servers. There is little IO, a bit of CPU usage (110% user, 3090% idle), but one thing is very strange - the resident memory usage of the Solr.
Despite the -Xms=8G and -Xmx=8G and despite that Solr UI shows only 4Gb heap used, the top and atop shows RSIZE=100Gb+ used by Solr. We are not using direct memory. We have not increased the java allowed direct memory. Further digging show that solr has 130K+ open files: $ sudo lsof | grep 5514 | grep REG | wc -l 132104 What is even more worriesome is that some files are literally open thousand times, and even old tlog files are still kept open: $ sudo lsof | grep 5514 | grep REG | grep tlog | headjava 5514 solr 142u REG 252,0 2918073 71565947 /var/db/solr/data/regulatory_shard1_replica1/data/tlog/tlog.00000000000 00038952 java 5514 solr 147u REG 252,0 2794 71565408 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.00000000000031447 00 java 5514 solr 153u REG 252,0 2869 71565602 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.00000000000031446 97 java 5514 solr 160u REG 252,0 2869 71565823 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.00000000000031446 99 java 5514 solr 161u REG 252,0 428385 71566321 /var/db/solr/data/WebResource_shard1_replica2/data/tlog/tlog.0000000000 075888387 java 5514 solr 162u REG 252,0 11518 71567726 /var/db/solr/data/RSS_shard2_replica1/data/tlog/tlog.000000000000005521 5 java 5514 solr 163u REG 252,0 1676 71566426 /var/db/solr/data/jobs_shard2_replica2/data/tlog/tlog.00000000000031439 93 java 5514 solr 176u REG 252,0 1199 71565773 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.00000000000031447 10 java 5514 solr 179u REG 252,0 1769 71565833 /var/db/solr/data/jobs_shard2_replica2/data/tlog/tlog.00000000000031439 92 java 5514 solr 180u REG 252,0 2006034 71565631 /var/db/solr/data/WebResource_shard3_replica1/data/tlog/tlog.0000000000 075897473 $ sudo lsof | grep 5514 | grep REG | grep tlog | grep /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.00000000000031447 00 | wc -l 98 So this old tlog file, which (as I understand) supposed to be closed, is still open 98 times. I wonder if that is how Solr is supposed to work (I really doubt it). Same picture on Solr 6.6.2, tested on various Oracle JVMs ranging from 1.8.0_171 to 1.8.0_171 Boris. ––––––––––––––––––––––––––––––––––––– The information contained in this message and any attachments may be confidential and/or restricted and protected from disclosure. If the reader of this message is not the intended recipient, disclosure, copying, use, or distribution of the information included in this message is prohibited - please destroy all electronic and paper copies and notify the sender immediately.
