Raji, how that "OOM for solr occur in every 5 days." exactly looks like?
What is the error message? Where it's occurring exactly?

On Thu, Apr 30, 2020 at 1:30 AM Raji N <rajis...@gmail.com> wrote:

> Thanks so much Jan. Will try your suggestions , yes we are also running
> solr inside docker.
>
> Thanks,
> Raji
>
> On Wed, Apr 29, 2020 at 1:46 PM Jan Høydahl <jan....@cominvent.com> wrote:
>
> > I have seen the same, but only in Docker.
> > I think it does not relate to Solr’s off-heap usage for filters and other
> > data structures, but rather how Docker treats memory-mapped files as
> > virtual memory.
> > As you know, when using MMapDirectoryFactory, you actually let Linux
> > handle the loading and unloading of the index files, and Solr will access
> > them as if they were in a huge virtual memory pool. Naturally the index
> > files grow large, and there is something strange going on in the way
> Docker
> > handles this, leading to OOM, not for Java heap but for the process.
> >
> > I have no definitive answer, but so far my research has found a few
> > possible settings
> >
> > Set env.var MALLOC_ARENA_MAX=2
> > Try to limit -XX:MaxDirectMemorySize
> > Lower mem swappiness in Docker (--memory-swappiness 0)
> > More generic insight into java mem allocation in Docker:
> > https://dzone.com/articles/native-memory-allocation-in-examples
> >
> > Have not yet found a silver bullet, so very interested in this thread.
> >
> > Jan
> >
> > > 29. apr. 2020 kl. 19:26 skrev Raji N <rajis...@gmail.com>:
> > >
> > > Thank you for your reply.  When OOM happens somehow it doesn't generate
> > > dump file. So we have hourly heaps running to diagnose this issue. Heap
> > is
> > > around 700MB and threads around 150. But 29GB of native memory is used
> > up,
> > > it is consumed by java.io.DirectBufferR (27GB major consumption) and
> > > java.io.DirectByteBuffer  objects .
> > >
> > > We use solr 7.6.0 in solrcloud mode and OS is alpine . Java version
> > >
> > > java -version
> > >
> > > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
> > >
> > > java version "1.8.0_211"
> > >
> > > Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
> > >
> > > Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)
> > >
> > >
> > >
> > > Thanks much for taking a look at it.
> > >
> > > Raji
> > >
> > >
> > >
> > > On Wed, Apr 29, 2020 at 10:04 AM Shawn Heisey <apa...@elyograg.org>
> > wrote:
> > >
> > >> On 4/29/2020 2:07 AM, Raji N wrote:
> > >>> Has anyone encountered off-heap OOM. We are thinking of reducing heap
> > >>> further and increasing the hardcommit interval . Any other
> > suggestions? .
> > >>> Please share your thoughts.
> > >>
> > >> It sounds like it's not heap memory that's running out.
> > >>
> > >> When the OutOfMemoryError is logged, it will also contain a message
> > >> mentioning which resource ran out.
> > >>
> > >> A common message that might be logged with the OOME is "Unable to
> create
> > >> native thread".  This type of error, if that's what's happening,
> > >> actually has nothing at all to do with memory, OOME is just how Java
> > >> happens to report it.
> > >>
> > >> You will need to know exactly which resource is running out before we
> > >> can offer any assistance.
> > >>
> > >> If the OOME is logged, the message you're looking for will be in the
> > >> solr log, not the tiny special log that is created when Solr is killed
> > >> by an OOME.  What version of Solr are you running, and what OS is it
> > >> running on?
> > >>
> > >> Thanks,
> > >> Shawn
> > >>
> >
> >
>


-- 
Sincerely yours
Mikhail Khludnev

Reply via email to