The other memory is used by the OS as file buffers. All the important parts of the on-disk search index are buffered in memory. When the Solr process wants a block, it is already right there, no delays for disk access.
wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) On Feb 24, 2015, at 4:45 PM, Tang, Rebecca <rebecca.t...@ucsf.edu> wrote: > We gave the machine 180G mem to see if it improves performance. However, > after we increased the memory, Solr started using only 5% of the physical > memory. It has always used 90-something%. > > What could be causing solr to not grab all the physical memory (grabbing > so little of the physical memory)? > > Rebecca Tang > Applications Developer, UCSF CKM > Industry Documents Digital Libraries > E: rebecca.t...@ucsf.edu > > On 2/24/15 12:44 PM, "Shawn Heisey" <apa...@elyograg.org> wrote: > >> On 2/24/2015 1:09 PM, Tang, Rebecca wrote: >>> Our solr index used to perform OK on our beta production box (anywhere >>> between 0-3 seconds to complete any query), but today I noticed that the >>> performance is very bad (queries take between 12 15 seconds). >>> >>> I haven't updated the solr index configuration >>> (schema.xml/solrconfig.xml) lately. All that's changed is the data ‹ >>> every month, I rebuild the solr index from scratch and deploy it to the >>> box. We will eventually go to incremental builds. But for now, all >>> indexes are built from scratch. >>> >>> Here are the stats: >>> Solr index size 183G >>> Documents in index 14364201 >>> We just have single solr box >>> It has 100G memory >>> 500G Harddrive >>> 16 cpus >> >> The bottom line on this problem, and I'm sure it's not something you're >> going to want to hear: You don't have enough memory available to cache >> your index. I'd plan on at least 192GB of RAM for an index this size, >> and 256GB would be better. >> >> Depending on the exact index schema, the nature of your queries, and how >> large your Java heap for Solr is, 100GB of RAM could be enough for good >> performance on an index that size ... or it might be nowhere near >> enough. I would imagine that one of two things is true here, possibly >> both: 1) Your queries are very complex and involve accessing a very >> large percentage of the index data. 2) Your Java heap is enormous, >> leaving very little RAM for the OS to automatically cache the index. >> >> Adding more memory to the machine, if that's possible, might fix some of >> the problems. You can find a discussion of the problem here: >> >> http://wiki.apache.org/solr/SolrPerformanceProblems >> >> If you have any questions after reading that wiki article, feel free to >> ask them. >> >> Thanks, >> Shawn >> >
