Cool, please let us know what you find out.
On Tue, May 31, 2016 at 8:34 AM, Gavin Harcourt <gharco...@brainboxdigital.com> wrote: > Those two bugs would make sense as we have been reloading the cores quite > frequently recently to apply new config and schema changes. I'll keep an eye > on the situation now our reload spree has ended and see if it recurs. > > Thanks, > Gavin. > > On 31/05/16 16:14, Erick Erickson wrote: >> >> Possibly: >> SOLR-9116 or SOLR-9117? Note those two require that the core be >> reloaded, so you have to be doing something a bit unusual for them to >> be the problem. >> >> Best, >> Erick >> >> On Tue, May 31, 2016 at 5:41 AM, Gavin Harcourt >> <gharco...@brainboxdigital.com> wrote: >>> >>> Hi All, >>> >>> I've noticed on some of my solr nodes that the disk usage is increasing >>> over >>> time. After checking the output of lsof I found hundreds of references to >>> deleted index files being held by solr. This totaled 24GB on a 16GB >>> index. A >>> restart of solr can obviously fix this but this is not an ideal solution. >>> We >>> are running solr 5.4.0 on OpenJDK 1.8.0_91. We are using the Concurrent >>> Mark Sweep GC although I've also seen the same problem on nodes using the >>> G1 >>> GC. Our update handler has autoCommit and softAutoCommit enabled (at >>> different intervals). We are using solr cloud and have multiple shards >>> with >>> 2 nodes each in our collections. I've not seen any pattern between this >>> appearing on leaders or replicas. Not all my nodes appear to be >>> exhibiting >>> the problem either. Our usage pattern does involve a lot of churn in our >>> index with the majority of documents being updated/deleted every day. >>> >>> Searching JIRA and the web in general I could only find references to >>> this >>> sort of problem when running solr in tomcat. Can anyone suggest a reason >>> why >>> this might be happening or a way I can manage it without needing to >>> restart >>> solr? >>> >>> Example lsof output: >>> java 1100 s123 DEL REG 202,3 8919406 >>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3m9s.fdt >>> java 1100 s123 DEL REG 202,3 8919159 >>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk.tvd >>> java 1100 s123 DEL REG 202,3 8919150 >>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mnk_Lucene50_0.tim >>> java 1100 s123 DEL REG 202,3 8919094 >>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1_Lucene50_0.tim >>> java 1100 s123 DEL REG 202,3 8919103 >>> /home/s123/solr/data/uk_shard2_replica1/data/index/_3mq1.tvd >>> >>> Regards, >>> Gavin. >>> >
