On 9/25/2015 8:53 AM, Keith L wrote: > Using: > - JDK 1.8u40 > - UseG1GC, ParallelRefProcEnabled, Xmx12g,Xms12g > - Solr 4.10.4
My own testing has not been extremely rigorous, and I have not spent a lot of time looking at the fine details in the GC logs. The details of your message that I omitted here indicate that you have looked much closer than I have. > *Application Details:* > - Heavy indexing: > - ~4000 bulk updates per minute, ~20 documents each update or 80k > documents per minute > - ~100 fields per document, mostly small strings, TrieInt, TrieDouble > - Usage of docValues (1-5 fields per document) > - Some fields multi-value TrieInt (precisionStep=0) fields could > potentially have hundreds of values > - 1s auto soft commit openSearcher=true, 15s auto hard commit > openSearcher=false Heavy indexing creates a lot of garbage, so it keeps GC algorithms busy. It is a good way to find GC problems. > *Questions*: > 1) What is causing the JNI Weak References? > - Is it from nio? Usage of MMaps, DirectBuffers, etc? > 2) Why does it become worse during the lifetime of the application? > - Is there a leak? > - Are we getting into a situation where we begin promoting Searchers > along with > their IndexReaders/Writers too early. i.e. do we have a lot expensive > objects moving to > tenured generation just before they become dereferenced. > - Tune G1NewSizePercent? > 3) Are the JNI Weak References being visited but not cleaned during GC? > - Why not? tune G1HeapWastePercent? I don't have any specific answers for you, but I can point you at the results of my own work on GC tuning. https://wiki.apache.org/solr/ShawnHeisey There are both G1 and CMS settings there. For G1, the exact size to use for the G1HeapRegionSize parameter will depend on how many documents you've got in each index. There's a few paragraphs about it on the page. The start script for Solr 5.x, if left with its default settings, includes GC tuning that's very similar to the CMS settings on my wiki page. My initial testing on 5.2.1 has shown that general performance is a little bit higher than the 4.9 version. There is one nagging problem that I know about, which will hopefully be addressed before the 5.4 release. There are at least two issues with that problem as a root cause: https://issues.apache.org/jira/browse/SOLR-8088 https://issues.apache.org/jira/browse/SOLR-8096 If you are not using grouping or facets, then you wouldn't need to worry about that problem. I am finding a nice rabbit hole to get lost in starting at this page and diving into the "Related issues" links at the bottom: http://www.evanjones.ca/jvm-mmap-pause.html Thanks, Shawn
