On 10/4/2012 3:22 PM, jame vaalet wrote:
so imagine i have merged the 150 Gb index into single segment, this would make a single segment of 150 GB in memory. When new docs are indexed it wouldn't alter this 150 Gb index unless i update or delete the older docs, right? will 150 Gb single segment have problem with memory swapping at OS level?
Supplement to my previous reply: the real memory mentioned in the last paragraph does not include the memory that the OS uses to cache disk access. If more memory is needed and all the free memory is being used by the disk cache, the OS will throw away part of the disk cache (a near-instantaneous operation that should never involve disk I/O) and give that memory to the application that requests it.
Here's a very good breakdown of how memory gets used with MMapDirectory in Solr. It's applicable to any program that uses memory mapping, not just Solr:
http://java.dzone.com/articles/use-lucene%E2%80%99s-mmapdirectory Thanks, Shawn
