Raji, how that "OOM for solr occur in every 5 days." exactly looks like? What is the error message? Where it's occurring exactly?
On Thu, Apr 30, 2020 at 1:30 AM Raji N <rajis...@gmail.com> wrote: > Thanks so much Jan. Will try your suggestions , yes we are also running > solr inside docker. > > Thanks, > Raji > > On Wed, Apr 29, 2020 at 1:46 PM Jan Høydahl <jan....@cominvent.com> wrote: > > > I have seen the same, but only in Docker. > > I think it does not relate to Solr’s off-heap usage for filters and other > > data structures, but rather how Docker treats memory-mapped files as > > virtual memory. > > As you know, when using MMapDirectoryFactory, you actually let Linux > > handle the loading and unloading of the index files, and Solr will access > > them as if they were in a huge virtual memory pool. Naturally the index > > files grow large, and there is something strange going on in the way > Docker > > handles this, leading to OOM, not for Java heap but for the process. > > > > I have no definitive answer, but so far my research has found a few > > possible settings > > > > Set env.var MALLOC_ARENA_MAX=2 > > Try to limit -XX:MaxDirectMemorySize > > Lower mem swappiness in Docker (--memory-swappiness 0) > > More generic insight into java mem allocation in Docker: > > https://dzone.com/articles/native-memory-allocation-in-examples > > > > Have not yet found a silver bullet, so very interested in this thread. > > > > Jan > > > > > 29. apr. 2020 kl. 19:26 skrev Raji N <rajis...@gmail.com>: > > > > > > Thank you for your reply. When OOM happens somehow it doesn't generate > > > dump file. So we have hourly heaps running to diagnose this issue. Heap > > is > > > around 700MB and threads around 150. But 29GB of native memory is used > > up, > > > it is consumed by java.io.DirectBufferR (27GB major consumption) and > > > java.io.DirectByteBuffer objects . > > > > > > We use solr 7.6.0 in solrcloud mode and OS is alpine . Java version > > > > > > java -version > > > > > > Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8 > > > > > > java version "1.8.0_211" > > > > > > Java(TM) SE Runtime Environment (build 1.8.0_211-b12) > > > > > > Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode) > > > > > > > > > > > > Thanks much for taking a look at it. > > > > > > Raji > > > > > > > > > > > > On Wed, Apr 29, 2020 at 10:04 AM Shawn Heisey <apa...@elyograg.org> > > wrote: > > > > > >> On 4/29/2020 2:07 AM, Raji N wrote: > > >>> Has anyone encountered off-heap OOM. We are thinking of reducing heap > > >>> further and increasing the hardcommit interval . Any other > > suggestions? . > > >>> Please share your thoughts. > > >> > > >> It sounds like it's not heap memory that's running out. > > >> > > >> When the OutOfMemoryError is logged, it will also contain a message > > >> mentioning which resource ran out. > > >> > > >> A common message that might be logged with the OOME is "Unable to > create > > >> native thread". This type of error, if that's what's happening, > > >> actually has nothing at all to do with memory, OOME is just how Java > > >> happens to report it. > > >> > > >> You will need to know exactly which resource is running out before we > > >> can offer any assistance. > > >> > > >> If the OOME is logged, the message you're looking for will be in the > > >> solr log, not the tiny special log that is created when Solr is killed > > >> by an OOME. What version of Solr are you running, and what OS is it > > >> running on? > > >> > > >> Thanks, > > >> Shawn > > >> > > > > > -- Sincerely yours Mikhail Khludnev
