Just an update on the memory issue - might be useful for others. I read the following,
http://wiki.apache.org/solr/SolrCaching?highlight=(SolrCaching) and looks like the first and new searcher listeners would populate the FieldCache. Commenting out these two listener entries seems to do the trick - at least the heap size is not growing as soon as Solr starts up. I ran some searches and they all came out fine. Index rate is also pretty good. Would there be any impact of disabling these listeners? Thanks, -vivek On Wed, May 13, 2009 at 2:12 PM, vivek sar <vivex...@gmail.com> wrote: > Otis, > > In that case, I'm not sure why Solr is taking up so much memory as > soon as we start it up. I checked for .tii file and there is only one, > > -rw-r--r-- 1 search staff 20306 May 11 21:47 > ./20090510_1/data/index/_3au.tii > > I have all the cache disabled - so that shouldn't be a problem too. My > ramBuffer size is only 64MB. > > I read note on sorting, > http://wiki.apache.org/solr/SchemaDesign?highlight=(sort), and see > something related to FieldCache. I don't see this as parameter defined > in either solrconfig.xml or schema.xml. Could this be something that > can load things in memory at startup? How can we disable it? > > I'm trying to find out if there is a way to tell how much memory Solr > would consume and way to cap it. > > Thanks, > -vivek > > > > > On Wed, May 13, 2009 at 1:50 PM, Otis Gospodnetic > <otis_gospodne...@yahoo.com> wrote: >> >> Hi, >> >> Sorting is triggered by the sort parameter in the URL, not a characteristic >> of a field. :) >> >> Otis >> -- >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >> >> >> ----- Original Message ---- >>> From: vivek sar <vivex...@gmail.com> >>> To: solr-user@lucene.apache.org >>> Sent: Wednesday, May 13, 2009 4:42:16 PM >>> Subject: Re: Solr memory requirements? >>> >>> Thanks Otis. >>> >>> Our use case doesn't require any sorting or faceting. I'm wondering if >>> I've configured anything wrong. >>> >>> I got total of 25 fields (15 are indexed and stored, other 10 are just >>> stored). All my fields are basic data type - which I thought are not >>> sorted. My id field is unique key. >>> >>> Is there any field here that might be getting sorted? >>> >>> >>> required="true" omitNorms="true" compressed="false"/> >>> >>> >>> compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> default="NOW/HOUR" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> compressed="false"/> >>> >>> compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> compressed="false"/> >>> >>> compressed="false"/> >>> >>> compressed="false"/> >>> >>> omitNorms="true" compressed="false"/> >>> >>> compressed="false"/> >>> >>> default="NOW/HOUR" omitNorms="true"/> >>> >>> >>> >>> >>> omitNorms="true" multiValued="true"/> >>> >>> Thanks, >>> -vivek >>> >>> On Wed, May 13, 2009 at 1:10 PM, Otis Gospodnetic >>> wrote: >>> > >>> > Hi, >>> > Some answers: >>> > 1) .tii files in the Lucene index. When you sort, all distinct values >>> > for the >>> field(s) used for sorting. Similarly for facet fields. Solr caches. >>> > 2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will >>> > consume >>> during indexing. There is no need to commit every 50K docs unless you want >>> to >>> trigger snapshot creation. >>> > 3) see 1) above >>> > >>> > 1.5 billion docs per instance where each doc is cca 1KB? I doubt that's >>> > going >>> to fly. :) >>> > >>> > Otis >>> > -- >>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >>> > >>> > >>> > >>> > ----- Original Message ---- >>> >> From: vivek sar >>> >> To: solr-user@lucene.apache.org >>> >> Sent: Wednesday, May 13, 2009 3:04:46 PM >>> >> Subject: Solr memory requirements? >>> >> >>> >> Hi, >>> >> >>> >> I'm pretty sure this has been asked before, but I couldn't find a >>> >> complete answer in the forum archive. Here are my questions, >>> >> >>> >> 1) When solr starts up what does it loads up in the memory? Let's say >>> >> I've 4 cores with each core 50G in size. When Solr comes up how much >>> >> of it would be loaded in memory? >>> >> >>> >> 2) How much memory is required during index time? If I'm committing >>> >> 50K records at a time (1 record = 1KB) using solrj, how much memory do >>> >> I need to give to Solr. >>> >> >>> >> 3) Is there a minimum memory requirement by Solr to maintain a certain >>> >> size index? Is there any benchmark on this? >>> >> >>> >> Here are some of my configuration from solrconfig.xml, >>> >> >>> >> 1) 64 >>> >> 2) All the caches (under query tag) are commented out >>> >> 3) Few others, >>> >> a) true ==> >>> >> would this require memory? >>> >> b) 50 >>> >> c) 200 >>> >> d) >>> >> e) false >>> >> f) 2 >>> >> >>> >> The problem we are having is following, >>> >> >>> >> I've given Solr RAM of 6G. As the total index size (all cores >>> >> combined) start growing the Solr memory consumption goes up. With 800 >>> >> million documents, I see Solr already taking up all the memory at >>> >> startup. After that the commits, searches everything become slow. We >>> >> will be having distributed setup with multiple Solr instances (around >>> >> 8) on four boxes, but our requirement is to have each Solr instance at >>> >> least maintain around 1.5 billion documents. >>> >> >>> >> We are trying to see if we can somehow reduce the Solr memory >>> >> footprint. If someone can provide a pointer on what parameters affect >>> >> memory and what effects it has we can then decide whether we want that >>> >> parameter or not. I'm not sure if there is any minimum Solr >>> >> requirement for it to be able maintain large indexes. I've used Lucene >>> >> before and that didn't require anything by default - it used up memory >>> >> only during index and search times - not otherwise. >>> >> >>> >> Any help is very much appreciated. >>> >> >>> >> Thanks, >>> >> -vivek >>> > >>> > >> >> >
