aha! mmap explains it. thank you.
On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey <s...@elyograg.org> wrote: > On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote: > > My solr instances are configured with 10GB heap (Xmx) but linux shows > > resident size of 16-20GB. even with thread stack and permgen taken into > > account i'm still far off from these numbers. Could it be that jvm IO > > buffers take so much space? does lucene use JNI/JNA memory allocations? > > Solr does not do anything off-heap. There is a project called > heliosearch underway that aims to use off-heap memory extensively with > Solr. > > There IS some mis-reporting of memory usage, though. See a screenshot > that I just captured of top output, sorted by memory usage. The java > process at the top of the list is Solr, running under the included Jetty: > > https://www.dropbox.com/s/03a3pp510mrtixo/solr-ram-usage-wrong.png > > I have a 6GB heap and 52GB of index data on this server. This makes the > 62.2GB virtual memory size completely reasonable. The claimed resident > memory size is 20GB, though. If you add that 20GB to the 49GB that is > allocated to the OS disk cache and the 6GB that it says is free, that's > 75GB. I've only got 64GB of RAM on the box, so something is being > reported wrong. > > If I take my 20GB resident size and subtract the 14GB shared size, that > is closer to reality, and it makes the numbers fit into the actual > amount of RAM that's on the machine. I believe the misreporting is > caused by the specific way that Java uses MMap when opening Lucene > indexes. This information comes from what I remember about a > conversation I witnessed in #lucene or #lucene-dev, not from my own > exploration. I believe they said that the MMap methods which don't > misreport memory usage would not do what Lucene requires. > > Thanks, > Shawn > >
