Turns out the cause was multiple indexing jobs indexing into the index simultaneously, which one can imagine can cause jvm loads on certain replicas for sure. Once this was found and only one job ran at a time, things were back to normal.
Your comments seem right on no correlation to the stack trace! > On Apr 1, 2019, at 5:32 PM, Shawn Heisey <apa...@elyograg.org> wrote: > > 4/1/2019 5:40 PM, Aroop Ganguly wrote: >> Thanks Shawn, for the initial response. >> Digging into a bit, I was wondering if we’d care to read the inner most >> stack. >> From the inner most stack it seems to be telling us something about what >> trigger it ? >> Ofcourse, the system could have been overloaded as well, but is the >> exception telling us something or its of no use to consider this stack > > The stacktrace on OOME is rarely useful. The memory allocation where the > error is thrown probably has absolutely no connection to the part of the > program where major amounts of memory are being used. It could be ANY memory > allocation that actually causes the error. > > Thanks, > Shawn