On 2/26/2017 6:40 AM, SOLR4189 wrote: > Shawn, you are right. > * OS vendor and version > CentosOS 6.5 > > * Java vendor and version > OpenJDK version 1.8.0_20 > OpenJDK 64-bit Server VM (build 25.20-b23) > > * Servlet container used to start Solr. > Catalina(tomcat7) > > * Total amount of memory in the server. > 30 GB > > * Max heap size for Solr. > 8GB(JVM) > > * An idea of exactly what is running on the server. > On our servers runs solr service only and splunk forwarder > > * Total index size and document count being handled by Solr (add up all > indexes). > 60GB and 2.6 milion on one shard
You say that you've got 60GB of index, but the screenshot seems to indicate that you've actually got 160 to 180GB of index on the machine. With approximately 22GB of memory available for caching (30GB total minus 8GB for Solr's heap), you don't have enough memory for good performance. Your commits are probabl taking so long to finish that additional commits are coming in and trying to open new searchers before the previous commits are done. Increasing the memory or splitting the index onto more machines might help performance. With 2.6 million documents in an index shard, whenever the system creates a filterCache entry for that shard, it will be 325000 bytes. If enough of these entries are created, a huge amount of heap memory will be required. It will not be a memory leak, though. You've got an early Java 8 release. There have been some memory leaks in Java itself fixed in later releases. Consider upgrading to the latest Java 8. The only thing I can say about the container (tomcat) is that it is an untested environment. The only container that actually gets tested is Jetty. It's not very likely that running in Tomcat is the problm, though. > * A screen shot of a process list sorted by memory usage. > <http://lucene.472066.n3.nabble.com/file/n4322362/20170226_102812.jpg> The display for htop is NOT the same as top. If I had wanted htop, that would have been what I mentioned. The standard top utility shows everything I was wanting to see. The display for htop can be useful, and has answered one question, but doesn't contain everything that I was after. Can you share a screenshot of the Solr dashboard, and one of the standard top utility sorted by memory usage? > * A screenshot showing total system memory allocations > <http://lucene.472066.n3.nabble.com/file/n4322362/20170226_102007.jpg> This file is not available. Nabble says "file not found." Thanks, Shawn
