32 GB is a pretty big heap. If the working set is really smaller than that, the extra heap just makes a full GC take longer.
How much heap is used after a full GC? Take the largest value you see there, then add a bit more, maybe 25% more or 2 GB more. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Apr 28, 2016, at 8:50 AM, Nick Vasilyev <nick.vasily...@gmail.com> wrote: > > mmfr_exact is a string field. key_phrases is a multivalued string field. > > On Thu, Apr 28, 2016 at 11:47 AM, Yonik Seeley <ysee...@gmail.com> wrote: > >> What about the field types though... are they single valued or multi >> valued, string, text, numeric? >> >> -Yonik >> >> >> On Thu, Apr 28, 2016 at 11:43 AM, Nick Vasilyev >> <nick.vasily...@gmail.com> wrote: >>> Hi Yonik, >>> >>> I forgot to mention that the index is approximately 50 million docs split >>> across 4 shards (replication factor 2) on 2 solr replicas. >>> >>> This particular script will filter items based on a category >> (10-~1,000,000 >>> items in each) and run facets on top X terms for particular fields. Query >>> looks like this: >>> >>> { >>> q => "cat:$code", >>> rows => 0, >>> facet => 'true', >>> 'facet.field' => [ 'key_phrases', 'mmfr_exact' ], >>> 'f.key_phrases.facet.limit' => 100, >>> 'f.mmfr_exact.facet.limit' => 20, >>> 'facet.mincount' => 5, >>> distrib => 'false', >>> } >>> >>> I know it can be re-worked some, especially considering there are >> thousands >>> of similar requests going out. However we didn't have this issue before >> and >>> I am worried that it may be a symptom of a larger underlying problem. >>> >>> On Thu, Apr 28, 2016 at 11:34 AM, Yonik Seeley <ysee...@gmail.com> >> wrote: >>> >>>> On Thu, Apr 28, 2016 at 11:29 AM, Nick Vasilyev >>>> <nick.vasily...@gmail.com> wrote: >>>>> Hello, >>>>> >>>>> We recently upgraded to Solr 5.2.1 with jre1.8.0_74 and are seeing >> long >>>> GC >>>>> pauses when running jobs that do some hairy faceting. The same jobs >>>> worked >>>>> fine with our previous 4.6 Solr. >>>> >>>> What does a typical request look like, and what are the field types >>>> that faceting is done on? >>>> >>>> -Yonik >>>> >>>> >>>>> The JVM is configured with 32GB heap with default GC settings, however >>>> I've >>>>> been tweaking the GC settings to no avail. The latest version had the >>>>> following differences from the default config: >>>>> >>>>> XX:ConcGCThreads and XX:ParallelGCThreads are increased from 4 to 7 >>>>> >>>>> XX:CMSInitiatingOccupancyFraction increased from 50 to 70 >>>>> >>>>> >>>>> Here is a sample output from the gc_log >>>>> >>>>> 2016-04-28T04:36:47.240-0400: 27905.535: Total time for which >> application >>>>> threads were stopped: 0.1667520 seconds, Stopping threads took: >> 0.0171900 >>>>> seconds >>>>> {Heap before GC invocations=2051 (full 59): >>>>> par new generation total 6990528K, used 2626705K >> [0x00002b16c0000000, >>>>> 0x00002b18c0000000, 0x00002b18c0000000) >>>>> eden space 5592448K, 44% used [0x00002b16c0000000, >> 0x00002b17571b9948, >>>>> 0x00002b1815560000) >>>>> from space 1398080K, 10% used [0x00002b1815560000, >> 0x00002b181e8cac28, >>>>> 0x00002b186aab0000) >>>>> to space 1398080K, 0% used [0x00002b186aab0000, >> 0x00002b186aab0000, >>>>> 0x00002b18c0000000) >>>>> concurrent mark-sweep generation total 25165824K, used 25122205K >>>>> [0x00002b18c0000000, 0x00002b1ec0000000, 0x00002b1ec0000000) >>>>> Metaspace used 41840K, capacity 42284K, committed 42680K, >> reserved >>>>> 43008K >>>>> 2016-04-28T04:36:49.828-0400: 27908.123: [GC (Allocation Failure) >>>>> 2016-04-28T04:36:49.828-0400: 27908.124: >>>> [CMS2016-04-28T04:36:49.912-0400: >>>>> 27908.207: [CMS-concurr >>>>> ent-abortable-preclean: 5.615/5.862 secs] [Times: user=17.70 sys=2.77, >>>>> real=5.86 secs] >>>>> (concurrent mode failure): 25122205K->15103706K(25165824K), 8.5567560 >>>>> secs] 27748910K->15103706K(32156352K), [Metaspace: >>>> 41840K->41840K(43008K)], >>>>> 8.5657830 secs] [ >>>>> Times: user=8.56 sys=0.01, real=8.57 secs] >>>>> Heap after GC invocations=2052 (full 60): >>>>> par new generation total 6990528K, used 0K [0x00002b16c0000000, >>>>> 0x00002b18c0000000, 0x00002b18c0000000) >>>>> eden space 5592448K, 0% used [0x00002b16c0000000, >> 0x00002b16c0000000, >>>>> 0x00002b1815560000) >>>>> from space 1398080K, 0% used [0x00002b1815560000, >> 0x00002b1815560000, >>>>> 0x00002b186aab0000) >>>>> to space 1398080K, 0% used [0x00002b186aab0000, >> 0x00002b186aab0000, >>>>> 0x00002b18c0000000) >>>>> concurrent mark-sweep generation total 25165824K, used 15103706K >>>>> [0x00002b18c0000000, 0x00002b1ec0000000, 0x00002b1ec0000000) >>>>> Metaspace used 41840K, capacity 42284K, committed 42680K, >> reserved >>>>> 43008K >>>>> } >>>>> 2016-04-28T04:36:58.395-0400: 27916.690: Total time for which >> application >>>>> threads were stopped: 8.5676090 seconds, Stopping threads took: >> 0.0003930 >>>>> seconds >>>>> >>>>> I read the instructions here, >> https://wiki.apache.org/solr/ShawnHeisey, >>>> but >>>>> they seem to be specific to Java 7. Are there any updated >> recommendations >>>>> for Java 8? >>>> >>
