Avishai: It sounds like you already understand mmap. Even so you might be interested in this excellent writeup of MMapDirectory and Lucene by Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best, Erick On Tue, Mar 18, 2014 at 7:23 AM, Avishai Ish-Shalom <avis...@fewbytes.com> wrote: > aha! mmap explains it. thank you. > > > On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey <s...@elyograg.org> wrote: > >> On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote: >> > My solr instances are configured with 10GB heap (Xmx) but linux shows >> > resident size of 16-20GB. even with thread stack and permgen taken into >> > account i'm still far off from these numbers. Could it be that jvm IO >> > buffers take so much space? does lucene use JNI/JNA memory allocations? >> >> Solr does not do anything off-heap. There is a project called >> heliosearch underway that aims to use off-heap memory extensively with >> Solr. >> >> There IS some mis-reporting of memory usage, though. See a screenshot >> that I just captured of top output, sorted by memory usage. The java >> process at the top of the list is Solr, running under the included Jetty: >> >> https://www.dropbox.com/s/03a3pp510mrtixo/solr-ram-usage-wrong.png >> >> I have a 6GB heap and 52GB of index data on this server. This makes the >> 62.2GB virtual memory size completely reasonable. The claimed resident >> memory size is 20GB, though. If you add that 20GB to the 49GB that is >> allocated to the OS disk cache and the 6GB that it says is free, that's >> 75GB. I've only got 64GB of RAM on the box, so something is being >> reported wrong. >> >> If I take my 20GB resident size and subtract the 14GB shared size, that >> is closer to reality, and it makes the numbers fit into the actual >> amount of RAM that's on the machine. I believe the misreporting is >> caused by the specific way that Java uses MMap when opening Lucene >> indexes. This information comes from what I remember about a >> conversation I witnessed in #lucene or #lucene-dev, not from my own >> exploration. I believe they said that the MMap methods which don't >> misreport memory usage would not do what Lucene requires. >> >> Thanks, >> Shawn >> >>
