I don't know exactly, but couldn't it hit host-wide threads limit
limitation?

On Fri, May 1, 2020 at 11:02 AM Raji N <rajis...@gmail.com> wrote:

> Thanks for your  reply . Sure will take a look at the docker host log.  But
> even when we got "unable to create new native thread" error , the heap dump
> taken within hour before (we have hourly heap generation) the OOM did not
> have more than 150 to 160 threads. So it doesn't look like it happens due
> to running out of threads. Rather suspecting it happens because there is no
> native memory?.
>
> Thanks,
> Raji
>
> On Fri, May 1, 2020 at 12:13 AM Mikhail Khludnev <m...@apache.org> wrote:
>
> > > java.lang.OutOfMemoryError: unable to create new native thread
> > Usually mean code flaw, but there is a workaround to trigger heap GC.
> > It happens when app creates threads instead of proper pooling, and no GC
> > occurs, so java Thread objects hangs in heap in stopped state, but every
> of
> > them holds a native thread handler; and system run out of native threads
> > sooner or later. So, in this case reducing heap size, frees native thread
> > and app is able to recycle them. But you are right, it's rather better to
> > disable it.
> > Also, check docker host log, there's a specific error message for java
> > under docker.
> >
> > On Fri, May 1, 2020 at 3:55 AM Raji N <rajis...@gmail.com> wrote:
> >
> > > It used to occur every 3 days ,we reduced heap and it started
> > > occurring every 5 days .  From the logs we can't get much. Some times
> we
> > > see "unable to create  new native thread" in the logs and many times no
> > > exceptions .
> > > When it says "unable to create native thread" error , we got below
> > > exceptions as we use cdcr. To eliminate cdcr from this issue , we
> > disabled
> > > CDCR also. But we still get OOM.
> > >
> > >  WARN  (cdcr-update-log-synchronizer-93-thread-1) [   ]
> > > o.a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception
> > >
> > > java.lang.OutOfMemoryError: unable to create new native thread
> > >
> > >                at java.lang.Thread.start0(Native Method) ~[?:1.8.0_211]
> > >
> > >                at java.lang.Thread.start(Thread.java:717)
> ~[?:1.8.0_211]
> > >
> > >                at
> > >
> > >
> >
> org.apache.http.impl.client.IdleConnectionEvictor.start(IdleConnectionEvictor.java:96)
> > > ~[httpclient-4.5.3.jar:4.5.3]
> > >
> > >                at
> > >
> > >
> >
> org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1219)
> > > ~[httpclient-4.5.3.jar:4.5.3]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:319)
> > > ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f
> > > - nknize - 2018-12-07 14:47:53]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:330)
> > > ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f
> > > - nknize - 2018-12-07 14:47:53]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:268)
> > > ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f
> > > - nknize - 2018-12-07 14:47:53]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:255)
> > > ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f
> > > - nknize - 2018-12-07 14:47:53]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:200)
> > > ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f
> > > - nknize - 2018-12-07 14:47:53]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:957)
> > > ~[solr-solrj-7.6.0.jar:7.6.0 719cde97f84640faa1e3525690d262946571245f
> > > - nknize - 2018-12-07 14:47:53]
> > >
> > >                at
> > >
> > >
> >
> org.apache.solr.handler.CdcrUpdateLogSynchronizer$UpdateLogSynchronisation.run(CdcrUpdateLogSynchronizer.java:139)
> > > [solr-core-7.6.0.jar:7.6.0-SNAPSHOT
> > > 34d82ed033cccd8120431b73e93554b85b24a278 - i843100 - 2019-09-30
> > > 14:02:46]
> > >
> > >                at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > [?:1.8.0_211]
> > >
> > >                at
> > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > [?:1.8.0_211]
> > >
> > >                at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > [?:1.8.0_211]
> > >
> > >                at
> > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > [?:1.8.0_211]
> > >
> > >                at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > [?:1.8.0_211]
> > >
> > >                at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > [?:1.8.0_211]
> > >
> > > Thanks,
> > > Raji
> > > On Thu, Apr 30, 2020 at 12:24 AM Mikhail Khludnev <m...@apache.org>
> > wrote:
> > >
> > > > Raji, how that "OOM for solr occur in every 5 days." exactly looks
> > like?
> > > > What is the error message? Where it's occurring exactly?
> > > >
> > > > On Thu, Apr 30, 2020 at 1:30 AM Raji N <rajis...@gmail.com> wrote:
> > > >
> > > > > Thanks so much Jan. Will try your suggestions , yes we are also
> > running
> > > > > solr inside docker.
> > > > >
> > > > > Thanks,
> > > > > Raji
> > > > >
> > > > > On Wed, Apr 29, 2020 at 1:46 PM Jan Høydahl <jan....@cominvent.com
> >
> > > > wrote:
> > > > >
> > > > > > I have seen the same, but only in Docker.
> > > > > > I think it does not relate to Solr’s off-heap usage for filters
> and
> > > > other
> > > > > > data structures, but rather how Docker treats memory-mapped files
> > as
> > > > > > virtual memory.
> > > > > > As you know, when using MMapDirectoryFactory, you actually let
> > Linux
> > > > > > handle the loading and unloading of the index files, and Solr
> will
> > > > access
> > > > > > them as if they were in a huge virtual memory pool. Naturally the
> > > index
> > > > > > files grow large, and there is something strange going on in the
> > way
> > > > > Docker
> > > > > > handles this, leading to OOM, not for Java heap but for the
> > process.
> > > > > >
> > > > > > I have no definitive answer, but so far my research has found a
> few
> > > > > > possible settings
> > > > > >
> > > > > > Set env.var MALLOC_ARENA_MAX=2
> > > > > > Try to limit -XX:MaxDirectMemorySize
> > > > > > Lower mem swappiness in Docker (--memory-swappiness 0)
> > > > > > More generic insight into java mem allocation in Docker:
> > > > > > https://dzone.com/articles/native-memory-allocation-in-examples
> > > > > >
> > > > > > Have not yet found a silver bullet, so very interested in this
> > > thread.
> > > > > >
> > > > > > Jan
> > > > > >
> > > > > > > 29. apr. 2020 kl. 19:26 skrev Raji N <rajis...@gmail.com>:
> > > > > > >
> > > > > > > Thank you for your reply.  When OOM happens somehow it doesn't
> > > > generate
> > > > > > > dump file. So we have hourly heaps running to diagnose this
> > issue.
> > > > Heap
> > > > > > is
> > > > > > > around 700MB and threads around 150. But 29GB of native memory
> is
> > > > used
> > > > > > up,
> > > > > > > it is consumed by java.io.DirectBufferR (27GB major
> consumption)
> > > and
> > > > > > > java.io.DirectByteBuffer  objects .
> > > > > > >
> > > > > > > We use solr 7.6.0 in solrcloud mode and OS is alpine . Java
> > version
> > > > > > >
> > > > > > > java -version
> > > > > > >
> > > > > > > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
> > > > > > >
> > > > > > > java version "1.8.0_211"
> > > > > > >
> > > > > > > Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
> > > > > > >
> > > > > > > Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed
> mode)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks much for taking a look at it.
> > > > > > >
> > > > > > > Raji
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Apr 29, 2020 at 10:04 AM Shawn Heisey <
> > apa...@elyograg.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> On 4/29/2020 2:07 AM, Raji N wrote:
> > > > > > >>> Has anyone encountered off-heap OOM. We are thinking of
> > reducing
> > > > heap
> > > > > > >>> further and increasing the hardcommit interval . Any other
> > > > > > suggestions? .
> > > > > > >>> Please share your thoughts.
> > > > > > >>
> > > > > > >> It sounds like it's not heap memory that's running out.
> > > > > > >>
> > > > > > >> When the OutOfMemoryError is logged, it will also contain a
> > > message
> > > > > > >> mentioning which resource ran out.
> > > > > > >>
> > > > > > >> A common message that might be logged with the OOME is "Unable
> > to
> > > > > create
> > > > > > >> native thread".  This type of error, if that's what's
> happening,
> > > > > > >> actually has nothing at all to do with memory, OOME is just
> how
> > > Java
> > > > > > >> happens to report it.
> > > > > > >>
> > > > > > >> You will need to know exactly which resource is running out
> > before
> > > > we
> > > > > > >> can offer any assistance.
> > > > > > >>
> > > > > > >> If the OOME is logged, the message you're looking for will be
> in
> > > the
> > > > > > >> solr log, not the tiny special log that is created when Solr
> is
> > > > killed
> > > > > > >> by an OOME.  What version of Solr are you running, and what OS
> > is
> > > it
> > > > > > >> running on?
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Shawn
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Sincerely yours
> > > > Mikhail Khludnev
> > > >
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> >
>


-- 
Sincerely yours
Mikhail Khludnev

Reply via email to