Re: off-heap OOM

2020-05-01 Thread Mikhail Khludnev
I don't know exactly, but couldn't it hit host-wide threads limit limitation? On Fri, May 1, 2020 at 11:02 AM Raji N wrote: > Thanks for your reply . Sure will take a look at the docker host log. But > even when we got "unable to create new native thread" error , the heap dump > taken within h

Re: off-heap OOM

2020-05-01 Thread Raji N
Thanks for your reply . Sure will take a look at the docker host log. But even when we got "unable to create new native thread" error , the heap dump taken within hour before (we have hourly heap generation) the OOM did not have more than 150 to 160 threads. So it doesn't look like it happens due

Re: off-heap OOM

2020-05-01 Thread Mikhail Khludnev
> java.lang.OutOfMemoryError: unable to create new native thread Usually mean code flaw, but there is a workaround to trigger heap GC. It happens when app creates threads instead of proper pooling, and no GC occurs, so java Thread objects hangs in heap in stopped state, but every of them holds a na

Re: off-heap OOM

2020-04-30 Thread Raji N
It used to occur every 3 days ,we reduced heap and it started occurring every 5 days . From the logs we can't get much. Some times we see "unable to create new native thread" in the logs and many times no exceptions . When it says "unable to create native thread" error , we got below exceptions a

Re: off-heap OOM

2020-04-30 Thread Mikhail Khludnev
Raji, how that "OOM for solr occur in every 5 days." exactly looks like? What is the error message? Where it's occurring exactly? On Thu, Apr 30, 2020 at 1:30 AM Raji N wrote: > Thanks so much Jan. Will try your suggestions , yes we are also running > solr inside docker. > > Thanks, > Raji > > O

Re: off-heap OOM

2020-04-29 Thread Raji N
Thanks so much Jan. Will try your suggestions , yes we are also running solr inside docker. Thanks, Raji On Wed, Apr 29, 2020 at 1:46 PM Jan Høydahl wrote: > I have seen the same, but only in Docker. > I think it does not relate to Solr’s off-heap usage for filters and other > data structures,

Re: off-heap OOM

2020-04-29 Thread Jan Høydahl
I have seen the same, but only in Docker. I think it does not relate to Solr’s off-heap usage for filters and other data structures, but rather how Docker treats memory-mapped files as virtual memory. As you know, when using MMapDirectoryFactory, you actually let Linux handle the loading and unlo

Re: off-heap OOM

2020-04-29 Thread matthew sporleder
What does the message look like, exactly, from solr.log ? On Wed, Apr 29, 2020 at 1:27 PM Raji N wrote: > > Thank you for your reply. When OOM happens somehow it doesn't generate > dump file. So we have hourly heaps running to diagnose this issue. Heap is > around 700MB and threads around 150. B

Re: off-heap OOM

2020-04-29 Thread Raji N
Thank you for your reply. When OOM happens somehow it doesn't generate dump file. So we have hourly heaps running to diagnose this issue. Heap is around 700MB and threads around 150. But 29GB of native memory is used up, it is consumed by java.io.DirectBufferR (27GB major consumption) and java.io.

Re: off-heap OOM

2020-04-29 Thread Shawn Heisey
On 4/29/2020 2:07 AM, Raji N wrote: Has anyone encountered off-heap OOM. We are thinking of reducing heap further and increasing the hardcommit interval . Any other suggestions? . Please share your thoughts. It sounds like it's not heap memory that's running out. When the OutOfMemoryError is l