If you happen to be looking at "top" or the like, you might be seeing virtual memory, see Uwe's excellent blog here: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best, Erick On Fri, Jan 1, 2016 at 11:46 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 12/31/2015 8:03 PM, Zheng Lin Edwin Yeo wrote: >> But the problem I'm facing now is that during optimizing, the memory usage >> of the server hit the maximum of 64GB, and I believe the optimization could >> not be completed fully as there is not enough memory, so when I check the >> index again, it says that it is not optimized. Before the optimization, the >> memory usage was less than 16GB, so the optimization actually uses up more >> than 48GB of memory. >> >> Is it normal for an index size of 200GB to use up so much memory during >> optimization? > > What *exactly* are you looking at that says Solr is using all your > memory? You must be extremely specific when answering this question. > This will determine whether we should be looking for a bug or not. > > It is completely normal for all modern operating systems to use all the > memory when the amount of data being handled is large. Some of the > memory will be allocated to programs like Java/Solr, and the operating > system will use everything else to cache data from I/O operations on the > disk. This is called the page cache. For Solr to perform well, the > page cache must be large enough to effectively cache your index data. > > https://en.wikipedia.org/wiki/Page_cache > > In another message thread, you indicated that your max heap was set to > 14GB. Java will only ever use that much memory for the program that is > being run, plus a relatively small amount so that Java itself can > operate. Any significantly large resident memory allocation beyond the > max heap would be an indication of a bug in Java, not a bug in Solr. > > With the index size at 200GB, I would hope to have at least 128GB of > memory in the server, but I would *want* 256GB. 64GB may not be enough > for good performance. > > Thanks, > Shawn >
