thanks!
On Tue, Mar 18, 2014 at 4:37 PM, Erick Erickson <erickerick...@gmail.com>wrote: > Avishai: > > It sounds like you already understand mmap. Even so you might be > interested in this excellent writeup of MMapDirectory and Lucene by > Uwe: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html > > Best, > Erick > > On Tue, Mar 18, 2014 at 7:23 AM, Avishai Ish-Shalom > <avis...@fewbytes.com> wrote: > > aha! mmap explains it. thank you. > > > > > > On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey <s...@elyograg.org> wrote: > > > >> On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote: > >> > My solr instances are configured with 10GB heap (Xmx) but linux shows > >> > resident size of 16-20GB. even with thread stack and permgen taken > into > >> > account i'm still far off from these numbers. Could it be that jvm IO > >> > buffers take so much space? does lucene use JNI/JNA memory > allocations? > >> > >> Solr does not do anything off-heap. There is a project called > >> heliosearch underway that aims to use off-heap memory extensively with > >> Solr. > >> > >> There IS some mis-reporting of memory usage, though. See a screenshot > >> that I just captured of top output, sorted by memory usage. The java > >> process at the top of the list is Solr, running under the included > Jetty: > >> > >> https://www.dropbox.com/s/03a3pp510mrtixo/solr-ram-usage-wrong.png > >> > >> I have a 6GB heap and 52GB of index data on this server. This makes the > >> 62.2GB virtual memory size completely reasonable. The claimed resident > >> memory size is 20GB, though. If you add that 20GB to the 49GB that is > >> allocated to the OS disk cache and the 6GB that it says is free, that's > >> 75GB. I've only got 64GB of RAM on the box, so something is being > >> reported wrong. > >> > >> If I take my 20GB resident size and subtract the 14GB shared size, that > >> is closer to reality, and it makes the numbers fit into the actual > >> amount of RAM that's on the machine. I believe the misreporting is > >> caused by the specific way that Java uses MMap when opening Lucene > >> indexes. This information comes from what I remember about a > >> conversation I witnessed in #lucene or #lucene-dev, not from my own > >> exploration. I believe they said that the MMap methods which don't > >> misreport memory usage would not do what Lucene requires. > >> > >> Thanks, > >> Shawn > >> > >> >
