Hi. We're running Solr 6.6.1 (SolrCloud, 3 clusters). Recently I noticed it became significantly slower to respond and did some basic checks on servers. There is little IO, a bit of CPU usage (110% user, 3090% idle), but one thing is very strange - the resident memory usage of the Solr.
Despite the -Xms=8G and -Xmx=8G and despite that Solr UI shows only 4Gb heap used, the top and atop shows RSIZE=100Gb+ used by Solr. We are not using direct memory. We have not increased the java allowed direct memory. Further digging show that solr has 130K+ open files: $ sudo lsof | grep 5514 | grep REG | wc -l 132104 What is even more worriesome is that some files are literally open thousand times, and even old tlog files are still kept open: $ sudo lsof | grep 5514 | grep REG | grep tlog | headjava 5514 solr 142u REG 252,0 2918073 71565947 /var/db/solr/data/regulatory_shard1_replica1/data/tlog/tlog.0000000000000038952 java 5514 solr 147u REG 252,0 2794 71565408 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.0000000000003144700 java 5514 solr 153u REG 252,0 2869 71565602 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.0000000000003144697 java 5514 solr 160u REG 252,0 2869 71565823 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.0000000000003144699 java 5514 solr 161u REG 252,0 428385 71566321 /var/db/solr/data/WebResource_shard1_replica2/data/tlog/tlog.0000000000075888387 java 5514 solr 162u REG 252,0 11518 71567726 /var/db/solr/data/RSS_shard2_replica1/data/tlog/tlog.0000000000000055215 java 5514 solr 163u REG 252,0 1676 71566426 /var/db/solr/data/jobs_shard2_replica2/data/tlog/tlog.0000000000003143993 java 5514 solr 176u REG 252,0 1199 71565773 /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.0000000000003144710 java 5514 solr 179u REG 252,0 1769 71565833 /var/db/solr/data/jobs_shard2_replica2/data/tlog/tlog.0000000000003143992 java 5514 solr 180u REG 252,0 2006034 71565631 /var/db/solr/data/WebResource_shard3_replica1/data/tlog/tlog.0000000000075897473 $ sudo lsof | grep 5514 | grep REG | grep tlog | grep /var/db/solr/data/jobs_shard1_replica1/data/tlog/tlog.0000000000003144700 | wc -l 98 So this old tlog file, which (as I understand) supposed to be closed, is still open 98 times. I wonder if that is how Solr is supposed to work (I really doubt it). Boris. ––––––––––––––––––––––––––––––––––––– The information contained in this message and any attachments may be confidential and/or restricted and protected from disclosure. If the reader of this message is not the intended recipient, disclosure, copying, use, or distribution of the information included in this message is prohibited - please destroy all electronic and paper copies and notify the sender immediately.
