Otis, We are not running master-slave configuration. We get very few searches(admin only) in a day so we didn't see the need of replication/snapshot. This problem is with one Solr instance managing 4 cores (each core 200 million records). Both indexing and searching is performed by the same Solr instance.
What are .tii files used for? I see this file under only one core. Still looking for what gets loaded in heap by Solr (during load time, indexing, and searching) and stays there. I see most of these are tenured objects and not getting released by GC - will post profile records tomorrow. Thanks, -vivek On Wed, May 13, 2009 at 6:34 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com> wrote: > > There is constant mixing of indexing concepts and searching concepts in this > thread. Are you having problems on the master (indexing) or on the slave > (searching)? > > > That .tii is only 20K and you said this is a large index? That doesn't smell > right... > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- >> From: vivek sar <vivex...@gmail.com> >> To: solr-user@lucene.apache.org >> Sent: Wednesday, May 13, 2009 5:12:00 PM >> Subject: Re: Solr memory requirements? >> >> Otis, >> >> In that case, I'm not sure why Solr is taking up so much memory as >> soon as we start it up. I checked for .tii file and there is only one, >> >> -rw-r--r-- 1 search staff 20306 May 11 21:47 >> ./20090510_1/data/index/_3au.tii >> >> I have all the cache disabled - so that shouldn't be a problem too. My >> ramBuffer size is only 64MB. >> >> I read note on sorting, >> http://wiki.apache.org/solr/SchemaDesign?highlight=(sort), and see >> something related to FieldCache. I don't see this as parameter defined >> in either solrconfig.xml or schema.xml. Could this be something that >> can load things in memory at startup? How can we disable it? >> >> I'm trying to find out if there is a way to tell how much memory Solr >> would consume and way to cap it. >> >> Thanks, >> -vivek >> >> >> >> >> On Wed, May 13, 2009 at 1:50 PM, Otis Gospodnetic >> wrote: >> > >> > Hi, >> > >> > Sorting is triggered by the sort parameter in the URL, not a >> > characteristic of >> a field. :) >> > >> > Otis >> > -- >> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> > >> > >> > >> > ----- Original Message ---- >> >> From: vivek sar >> >> To: solr-user@lucene.apache.org >> >> Sent: Wednesday, May 13, 2009 4:42:16 PM >> >> Subject: Re: Solr memory requirements? >> >> >> >> Thanks Otis. >> >> >> >> Our use case doesn't require any sorting or faceting. I'm wondering if >> >> I've configured anything wrong. >> >> >> >> I got total of 25 fields (15 are indexed and stored, other 10 are just >> >> stored). All my fields are basic data type - which I thought are not >> >> sorted. My id field is unique key. >> >> >> >> Is there any field here that might be getting sorted? >> >> >> >> >> >> required="true" omitNorms="true" compressed="false"/> >> >> >> >> >> >> compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> default="NOW/HOUR" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> compressed="false"/> >> >> >> >> compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> compressed="false"/> >> >> >> >> compressed="false"/> >> >> >> >> compressed="false"/> >> >> >> >> omitNorms="true" compressed="false"/> >> >> >> >> compressed="false"/> >> >> >> >> default="NOW/HOUR" omitNorms="true"/> >> >> >> >> >> >> >> >> >> >> omitNorms="true" multiValued="true"/> >> >> >> >> Thanks, >> >> -vivek >> >> >> >> On Wed, May 13, 2009 at 1:10 PM, Otis Gospodnetic >> >> wrote: >> >> > >> >> > Hi, >> >> > Some answers: >> >> > 1) .tii files in the Lucene index. When you sort, all distinct values >> >> > for >> the >> >> field(s) used for sorting. Similarly for facet fields. Solr caches. >> >> > 2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will >> consume >> >> during indexing. There is no need to commit every 50K docs unless you >> >> want >> to >> >> trigger snapshot creation. >> >> > 3) see 1) above >> >> > >> >> > 1.5 billion docs per instance where each doc is cca 1KB? I doubt that's >> going >> >> to fly. :) >> >> > >> >> > Otis >> >> > -- >> >> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >> > >> >> > >> >> > >> >> > ----- Original Message ---- >> >> >> From: vivek sar >> >> >> To: solr-user@lucene.apache.org >> >> >> Sent: Wednesday, May 13, 2009 3:04:46 PM >> >> >> Subject: Solr memory requirements? >> >> >> >> >> >> Hi, >> >> >> >> >> >> I'm pretty sure this has been asked before, but I couldn't find a >> >> >> complete answer in the forum archive. Here are my questions, >> >> >> >> >> >> 1) When solr starts up what does it loads up in the memory? Let's say >> >> >> I've 4 cores with each core 50G in size. When Solr comes up how much >> >> >> of it would be loaded in memory? >> >> >> >> >> >> 2) How much memory is required during index time? If I'm committing >> >> >> 50K records at a time (1 record = 1KB) using solrj, how much memory do >> >> >> I need to give to Solr. >> >> >> >> >> >> 3) Is there a minimum memory requirement by Solr to maintain a certain >> >> >> size index? Is there any benchmark on this? >> >> >> >> >> >> Here are some of my configuration from solrconfig.xml, >> >> >> >> >> >> 1) 64 >> >> >> 2) All the caches (under query tag) are commented out >> >> >> 3) Few others, >> >> >> a) true ==> >> >> >> would this require memory? >> >> >> b) 50 >> >> >> c) 200 >> >> >> d) >> >> >> e) false >> >> >> f) 2 >> >> >> >> >> >> The problem we are having is following, >> >> >> >> >> >> I've given Solr RAM of 6G. As the total index size (all cores >> >> >> combined) start growing the Solr memory consumption goes up. With 800 >> >> >> million documents, I see Solr already taking up all the memory at >> >> >> startup. After that the commits, searches everything become slow. We >> >> >> will be having distributed setup with multiple Solr instances (around >> >> >> 8) on four boxes, but our requirement is to have each Solr instance at >> >> >> least maintain around 1.5 billion documents. >> >> >> >> >> >> We are trying to see if we can somehow reduce the Solr memory >> >> >> footprint. If someone can provide a pointer on what parameters affect >> >> >> memory and what effects it has we can then decide whether we want that >> >> >> parameter or not. I'm not sure if there is any minimum Solr >> >> >> requirement for it to be able maintain large indexes. I've used Lucene >> >> >> before and that didn't require anything by default - it used up memory >> >> >> only during index and search times - not otherwise. >> >> >> >> >> >> Any help is very much appreciated. >> >> >> >> >> >> Thanks, >> >> >> -vivek >> >> > >> >> > >> > >> > > >
