Thanks so much Jan. Will try your suggestions , yes we are also running solr inside docker.
Thanks, Raji On Wed, Apr 29, 2020 at 1:46 PM Jan Høydahl <jan....@cominvent.com> wrote: > I have seen the same, but only in Docker. > I think it does not relate to Solr’s off-heap usage for filters and other > data structures, but rather how Docker treats memory-mapped files as > virtual memory. > As you know, when using MMapDirectoryFactory, you actually let Linux > handle the loading and unloading of the index files, and Solr will access > them as if they were in a huge virtual memory pool. Naturally the index > files grow large, and there is something strange going on in the way Docker > handles this, leading to OOM, not for Java heap but for the process. > > I have no definitive answer, but so far my research has found a few > possible settings > > Set env.var MALLOC_ARENA_MAX=2 > Try to limit -XX:MaxDirectMemorySize > Lower mem swappiness in Docker (--memory-swappiness 0) > More generic insight into java mem allocation in Docker: > https://dzone.com/articles/native-memory-allocation-in-examples > > Have not yet found a silver bullet, so very interested in this thread. > > Jan > > > 29. apr. 2020 kl. 19:26 skrev Raji N <rajis...@gmail.com>: > > > > Thank you for your reply. When OOM happens somehow it doesn't generate > > dump file. So we have hourly heaps running to diagnose this issue. Heap > is > > around 700MB and threads around 150. But 29GB of native memory is used > up, > > it is consumed by java.io.DirectBufferR (27GB major consumption) and > > java.io.DirectByteBuffer objects . > > > > We use solr 7.6.0 in solrcloud mode and OS is alpine . Java version > > > > java -version > > > > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 > > > > java version "1.8.0_211" > > > > Java(TM) SE Runtime Environment (build 1.8.0_211-b12) > > > > Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode) > > > > > > > > Thanks much for taking a look at it. > > > > Raji > > > > > > > > On Wed, Apr 29, 2020 at 10:04 AM Shawn Heisey <apa...@elyograg.org> > wrote: > > > >> On 4/29/2020 2:07 AM, Raji N wrote: > >>> Has anyone encountered off-heap OOM. We are thinking of reducing heap > >>> further and increasing the hardcommit interval . Any other > suggestions? . > >>> Please share your thoughts. > >> > >> It sounds like it's not heap memory that's running out. > >> > >> When the OutOfMemoryError is logged, it will also contain a message > >> mentioning which resource ran out. > >> > >> A common message that might be logged with the OOME is "Unable to create > >> native thread". This type of error, if that's what's happening, > >> actually has nothing at all to do with memory, OOME is just how Java > >> happens to report it. > >> > >> You will need to know exactly which resource is running out before we > >> can offer any assistance. > >> > >> If the OOME is logged, the message you're looking for will be in the > >> solr log, not the tiny special log that is created when Solr is killed > >> by an OOME. What version of Solr are you running, and what OS is it > >> running on? > >> > >> Thanks, > >> Shawn > >> > >
