On 10/29/2014 11:43 AM, Vijay Kokatnur wrote: > I am observing some weird behavior with how Solr is using memory. We are > running both Solr and zookeeper on the same node. We tested memory > settings on Solr Cloud Setup of 1 shard with 146GB index size, and 2 Shard > Solr setup with 44GB index size. Both are running on similar beefy > machines. > > After running the setup for 3-4 days, I see that a lot of memory is > inactive in all the nodes - > > 99052952 total memory > 98606256 used memory > 19143796 active memory > 75063504 inactive memory > > And inactive memory is never reclaimed by the OS. When total memory size > is reached, latency and disk IO shoots up. We observed this behavior in > both Solr Cloud setup with 1 shard and Solr setup with 2 shards.
Where are these numbers coming from? If they are coming from the operating system and not Java, then you have nothing to worry about. > For the Solr Cloud setup, we are running a cron job with following command > to clear out the inactive memory. It is working as expected. Even though > the index size of Cloud is 146GB, the used memory is always below 55GB. > Our response times are better and no errors/exceptions are thrown. (This > command causes issue in 2 Shard setup) > > echo 3 > /proc/sys/vm/drop_caches Don't do that. You're throwing away almost every performance advantage the operating system has to offer. If this changes the numbers so they look better to you, then I can almost guarantee you that you are not having any actual problem, and that dropping the caches like this is *hurting* performance, not helping it. It's completely normal for a correctly functioning system to report an extremely low amount of memory as free. The operating system is using the spare memory in your system as a filesystem cache, which makes everything run a lot faster. If a program needs more memory, the operating system will instantly give up some of its disk cache in order to satisfy the memory allocation. The "virtual memory" part of this blog post (which has direct relevance for Solr) hopefully can explain it better than I can. The entire blog post is worth reading. http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Thanks, Shawn
