On 10/8/2014 4:02 AM, Simon Fairey wrote: > I'm currently setting up jconsole but as I have to remotely monitor (no gui > capability on the server) I have to wait before I can restart solr with a JMX > port setup. In the meantime I looked at top and given the calculations you > said based on your top output and this top of my java process from the node > that handles the querying, the indexing node has a similar memory profile: > > https://www.dropbox.com/s/pz85dm4e7qpepco/SolrTop.png?dl=0 > > It would seem I need a monstrously large heap in the 60GB region? > > We do use a lot of navigators/filters so I have set the caches to be quite > large for these, are these what are using up the memory?
With a VIRT size of 189GB and a RES size of 73GB, I believe you probably have more than 45GB of index data. This might be a combination of old indexes and the active index. Only the indexes (cores) that are being actively used need to be considered when trying to calculate the total RAM needed. Other indexes will not affect performance, even though they increase your virtual memory size. With MMap, part of the virtual memory size is the size of the index data that has been opened on the disk. This is not memory that's actually allocated. There's a very good reason that mmap has been the default in Lucene and Solr for more than two years. http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html You stated originally that you have 25 million document and 45GB of index data on each node. With those numbers and a conservative configuration, I would expect that you need about 4GB of heap, maybe as much as 8GB. I cannot think of any reason that you would NEED a heap 60GB or larger. Each field that you sort on, each field that you facet on with the default facet.method of fc, and each filter that you cache will use a large block of memory. The size of that block of memory is almost exclusively determined by the number of documents in the index. With 25 million documents, each filterCache entry will be approximately 3MB -- one bit for every document. I do not know how big each FieldCache entry is for a sort field and a facet field, but assume that they are probably larger than the 3MB entries on the filterCache. I've got a filterCache sized at 64, with an autowarmCount of 4. With larger autowarmCount values, I was seeing commits take 30 seconds or more, because each of those filters can take a few seconds to execute. Cache sizes in the thousands are rarely necessary, and just chew up a lot of memory with no benefit. Large autowarmCount values are also rarely necessary. Every time a new searcher is opened by a commit, add up all your autowarmCount values and realize that the searcher likely needs to execute that many queries before it is available. If you need to set up remote JMX so you can remotely connect jconsole, I have done this in the redhat init script I've built -- see JMX_OPTS here: http://wiki.apache.org/solr/ShawnHeisey#Init_script It's never a good idea to expose Solr directly to the internet, but if you use that JMX config, *definitely* don't expose it to the Internet. It doesn't use any authentication. We might need to back up a little bit and start with the problem that you are trying to figure out, not the numbers that are being reported. http://people.apache.org/~hossman/#xyproblem Your original note said that you're sanity checking. Toward that end, the only insane thing that jumps out at me is that your max heap is *VERY* large, and you probably don't have the proper GC tuning. My recommendations for initial action are to use -Xmx8g on the servlet container startup and include the GC settings you can find on the wiki pages I've given you. It would be a very good idea to set up remote JMX so you can use jconsole or jvisualvm remotely. Thanks, Shawn
