Re: Solr memory usage off-heap

2014-03-20 Thread Avishai Ish-Shalom
thanks! On Tue, Mar 18, 2014 at 4:37 PM, Erick Erickson wrote: > Avishai: > > It sounds like you already understand mmap. Even so you might be > interested in this excellent writeup of MMapDirectory and Lucene by > Uwe: > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html >

Re: Solr memory usage off-heap

2014-03-18 Thread Shawn Heisey
On 3/18/2014 8:37 AM, Erick Erickson wrote: > It sounds like you already understand mmap. Even so you might be > interested in this excellent writeup of MMapDirectory and Lucene by > Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html There is some actual bad memory report

Re: Solr memory usage off-heap

2014-03-18 Thread Erick Erickson
Avishai: It sounds like you already understand mmap. Even so you might be interested in this excellent writeup of MMapDirectory and Lucene by Uwe: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Best, Erick On Tue, Mar 18, 2014 at 7:23 AM, Avishai Ish-Shalom wrote: > aha

Re: Solr memory usage off-heap

2014-03-18 Thread Avishai Ish-Shalom
aha! mmap explains it. thank you. On Tue, Mar 18, 2014 at 3:11 PM, Shawn Heisey wrote: > On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote: > > My solr instances are configured with 10GB heap (Xmx) but linux shows > > resident size of 16-20GB. even with thread stack and permgen taken into > > acco

Re: Solr memory usage off-heap

2014-03-18 Thread Shawn Heisey
On 3/18/2014 5:30 AM, Avishai Ish-Shalom wrote: > My solr instances are configured with 10GB heap (Xmx) but linux shows > resident size of 16-20GB. even with thread stack and permgen taken into > account i'm still far off from these numbers. Could it be that jvm IO > buffers take so much space? doe

RE: Solr memory usage off-heap

2014-03-18 Thread Doug Turnbull
memory-and-storage-for-lucenesolr/ Sent from my Windows Phone From: Avishai Ish-Shalom Sent: 3/18/2014 7:31 AM To: solr-user@lucene.apache.org Subject: Solr memory usage off-heap Hi, My solr instances are configured with 10GB heap (Xmx) but linux shows resident size of 16-20GB. even with thread stack a

Solr memory usage off-heap

2014-03-18 Thread Avishai Ish-Shalom
Hi, My solr instances are configured with 10GB heap (Xmx) but linux shows resident size of 16-20GB. even with thread stack and permgen taken into account i'm still far off from these numbers. Could it be that jvm IO buffers take so much space? does lucene use JNI/JNA memory allocations?