On 4/23/2019 11:48 AM, Brian Ecker wrote:
I see. The other files I meant to attach were the GC log ( https://pastebin.com/raw/qeuQwsyd), the heap histogram ( https://pastebin.com/raw/aapKTKTU), and the screenshot from top ( http://oi64.tinypic.com/21r0bk.jpg).
I have no idea what to do with the histogram. I doubt it's all that useful anyway, as it wouldn't have any information about what parts of the system are using the most.
The GC log is not complete. It only covers 2 min 47 sec 674 ms of time. To get anything useful out of a GC log, it would probably need to cover hours of runtime.
But if you are experiencing OutOfMemoryError, then either you have run into something where a memory leak exists, or there's something about your index or your queries that needs more heap than you have allocated. Memory leaks are not super common in Solr, but they have happened.
Tuning GC will never help OOME problems. The screenshot looks like it matches the info below.
I'll work on getting the heap dump, but would it also be sufficient to use say a 5GB dump from when it's half full and then extrapolate to the contents of the heap when it's full? That way the dump would be a bit easier to work with.
That might be useful. The only way to know for sure is to take a look at it to see if the part of the code using lots of heap is detectable.
There are around 2,100,000 documents.
<snip>
The data takes around 9GB on disk.
Ordinarily, I would expect that level of data to not need a whole lot of heap. 10GB would be more than I would think necessary, but if your queries are big consumers of memory, I could be wrong. I ran indexes with 30 million documents taking up 50GB of disk space on an 8GB heap. I probably could have gone lower with no problems.
I have absolutely no idea what kind of requirements the spellcheck feature has. I've never used that beyond a few test queries. If the query information you sent is complete, I wouldn't expect the non-spellcheck parts to require a whole lot of heap. So perhaps spellcheck is the culprit here. Somebody else will need to comment on that.
Thanks, Shawn
