Hi Yonik,
I forgot to mention that the index is approximately 50 million docs split
across 4 shards (replication factor 2) on 2 solr replicas.
This particular script will filter items based on a category (10-~1,000,000
items in each) and run facets on top X terms for particular fields. Query
looks like this:
{
q => "cat:$code",
rows => 0,
facet => 'true',
'facet.field' => [ 'key_phrases', 'mmfr_exact' ],
'f.key_phrases.facet.limit' => 100,
'f.mmfr_exact.facet.limit' => 20,
'facet.mincount' => 5,
distrib => 'false',
}
I know it can be re-worked some, especially considering there are thousands
of similar requests going out. However we didn't have this issue before and
I am worried that it may be a symptom of a larger underlying problem.
On Thu, Apr 28, 2016 at 11:34 AM, Yonik Seeley <ysee...@gmail.com> wrote:
> On Thu, Apr 28, 2016 at 11:29 AM, Nick Vasilyev
> <nick.vasily...@gmail.com> wrote:
> > Hello,
> >
> > We recently upgraded to Solr 5.2.1 with jre1.8.0_74 and are seeing long
> GC
> > pauses when running jobs that do some hairy faceting. The same jobs
> worked
> > fine with our previous 4.6 Solr.
>
> What does a typical request look like, and what are the field types
> that faceting is done on?
>
> -Yonik
>
>
> > The JVM is configured with 32GB heap with default GC settings, however
> I've
> > been tweaking the GC settings to no avail. The latest version had the
> > following differences from the default config:
> >
> > XX:ConcGCThreads and XX:ParallelGCThreads are increased from 4 to 7
> >
> > XX:CMSInitiatingOccupancyFraction increased from 50 to 70
> >
> >
> > Here is a sample output from the gc_log
> >
> > 2016-04-28T04:36:47.240-0400: 27905.535: Total time for which application
> > threads were stopped: 0.1667520 seconds, Stopping threads took: 0.0171900
> > seconds
> > {Heap before GC invocations=2051 (full 59):
> > par new generation total 6990528K, used 2626705K [0x00002b16c0000000,
> > 0x00002b18c0000000, 0x00002b18c0000000)
> > eden space 5592448K, 44% used [0x00002b16c0000000, 0x00002b17571b9948,
> > 0x00002b1815560000)
> > from space 1398080K, 10% used [0x00002b1815560000, 0x00002b181e8cac28,
> > 0x00002b186aab0000)
> > to space 1398080K, 0% used [0x00002b186aab0000, 0x00002b186aab0000,
> > 0x00002b18c0000000)
> > concurrent mark-sweep generation total 25165824K, used 25122205K
> > [0x00002b18c0000000, 0x00002b1ec0000000, 0x00002b1ec0000000)
> > Metaspace used 41840K, capacity 42284K, committed 42680K, reserved
> > 43008K
> > 2016-04-28T04:36:49.828-0400: 27908.123: [GC (Allocation Failure)
> > 2016-04-28T04:36:49.828-0400: 27908.124:
> [CMS2016-04-28T04:36:49.912-0400:
> > 27908.207: [CMS-concurr
> > ent-abortable-preclean: 5.615/5.862 secs] [Times: user=17.70 sys=2.77,
> > real=5.86 secs]
> > (concurrent mode failure): 25122205K->15103706K(25165824K), 8.5567560
> > secs] 27748910K->15103706K(32156352K), [Metaspace:
> 41840K->41840K(43008K)],
> > 8.5657830 secs] [
> > Times: user=8.56 sys=0.01, real=8.57 secs]
> > Heap after GC invocations=2052 (full 60):
> > par new generation total 6990528K, used 0K [0x00002b16c0000000,
> > 0x00002b18c0000000, 0x00002b18c0000000)
> > eden space 5592448K, 0% used [0x00002b16c0000000, 0x00002b16c0000000,
> > 0x00002b1815560000)
> > from space 1398080K, 0% used [0x00002b1815560000, 0x00002b1815560000,
> > 0x00002b186aab0000)
> > to space 1398080K, 0% used [0x00002b186aab0000, 0x00002b186aab0000,
> > 0x00002b18c0000000)
> > concurrent mark-sweep generation total 25165824K, used 15103706K
> > [0x00002b18c0000000, 0x00002b1ec0000000, 0x00002b1ec0000000)
> > Metaspace used 41840K, capacity 42284K, committed 42680K, reserved
> > 43008K
> > }
> > 2016-04-28T04:36:58.395-0400: 27916.690: Total time for which application
> > threads were stopped: 8.5676090 seconds, Stopping threads took: 0.0003930
> > seconds
> >
> > I read the instructions here, https://wiki.apache.org/solr/ShawnHeisey,
> but
> > they seem to be specific to Java 7. Are there any updated recommendations
> > for Java 8?
>
