On 12/14/2016 5:55 AM, moscovig wrote:
> We have solr 6.2.1.
> One of the collection is causing lots of updates.
> We see the next logs:
>
> /INFO org.apache.solr.core.SolrDeletionPolicy :
> SolrDeletionPolicy.onCommit: commits: num=2
>
> commit{dir=/opt/solr-6.2.1/server/solr/collection_shard1_replica2/data/index,segFN=segments_qbmv,generation=1228135}
>
> commit{dir=/opt/solr-6.2.1/server/solr/collection_shard1_replica2/data/index,segFN=segments_qbmw,generation=1228136}/
Those do not look like any problem at all. The first one says INFO, the
others probably do too, but what's here doesn't include the severity.
> As a result we are running out of memory in the instances hosting the
> collection.
> The used memory is increased by 1 percent per day.
>
> The used memory is not part of the Solr's JVM, but part of the Slab memory
> (which I get to know now :) )
Solr does not explicitly allocate memory outside of the JVM. Solr (via
Java) uses MMAP for access to index data, which relies on the OS using
memory for the disk cache, but this is normal OS function, and not
anything unusual. The OS can instantly re-allocate any of that memory
for use by programs that request it.
> when cat over /proc/meminfo we get:
> /
> Slab: 17906760 kB
> SReclaimable: 17841548 kB
> /
>
> and slabtop gives:
> 91635138 91635138 6% 0.19K 4363578 21 17454312K dentry
>
> ~17 GB for dentry.
>
> Is there any way to avoid this "memory leak"?
>
> echo 2 > /proc/sys/vm/drop_caches ; sync is cleaning the this "clean" cache
> but -
This sounds like the memory is being used for the OS disk cache -- which
is completely normal, and exactly how your spare memory SHOULD be used.
Solr has no control over this, and it's very likely that Java doesn't
either. This is *NOT* a memory leak. Your OS is working exactly how it
is supposed to work -- using otherwise unallocated memory to speed up
the system. If a program requests any of that memory, the OS will
instantly release whatever the program requests.
https://en.wikipedia.org/wiki/Page_cache
https://wiki.apache.org/solr/SolrPerformanceProblems
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> 2. What's the meaning of the SolrDeletionPolicy logs we get? do we commit
> lots of updates? deletes?
The deletion policy has to do with commit points. This is Lucene
functionality that Solr doesn't really use -- a typical deletion policy
will delete all but the most recent commit point. Solr has a lot of
logging by default that is not useful to the average person, but can
provide information vital to a developer who's familiar with Lucene.
Thanks,
Shawn
