Actually at about that time the replication finished and added about 20-30gb to the index from the master. My current set up goes Indexing master -> indexer slave/production master (only replicated on command)-> three search slaves (replicate each 15 minutes)
We added about 2.3m docs, then I replicated it to the production master and since there was a change it replicated out to the slave node the gc came from I’ll set one of the slaves to 31/31 and force all load to that one and see how she does. Thanks! > On Dec 6, 2019, at 1:02 AM, Shawn Heisey <apa...@elyograg.org> wrote: > > On 12/5/2019 12:57 PM, David Hastings wrote: >> That probably isnt enough data, so if youre interested: >> https://gofile.io/?c=rZQ2y4 > > The previous one was less than 4 minutes, so it doesn't reveal anything > useful. > > This one is a little bit less than two hours. That's more useful, but still > pretty short. > > Here's the "heap after GC" graph from the larger file: > > https://www.dropbox.com/s/q9hs8fl0gfkfqi1/david.hastings.gc.graph.2019.12.png?dl=0 > > At around 14:15, the heap usage was rather high. It got up over 25GB. There > were some very long GCs right at that time, which probably means they were > full GCs. And they didn't free up any significant amount of memory. So I'm > betting that sometimes you actually *do* need a big chunk of that 60GB of > heap. You might try reducing it to 31g instead of 60000m. Java's memory > usage is a lot more efficient if the max heap size is less than 32 GB. > > I can't give you any information about what happened at that time which > required so much heap. You could see if you have logfiles that cover that > timeframe. > > Thanks, > Shawn
