I think maxBufferedDocs has been deprecated in Solr 1.4 - it's recommended to use ramBufferSizeMB instead. My ramBufferSizeMB=64. This shouldn't be a problem I think.
There has to be something else that Solr is holding up in memory. Anyone else? Thanks, -vivek On Wed, May 13, 2009 at 4:01 PM, Jack Godwin <god...@gmail.com> wrote: > Have you checked the maxBufferedDocs? I had to drop mine down to 1000 with > 3 million docs. > Jack > > On Wed, May 13, 2009 at 6:53 PM, vivek sar <vivex...@gmail.com> wrote: > >> Disabling first/new searchers did help for the initial load time, but >> after 10-15 min the heap memory start climbing up again and reached >> max within 20 min. Now the GC is coming up all the time, which is >> slowing down the commit and search cycles. >> >> This is still puzzling what does Solr holds in the memory and doesn't >> release? >> >> I haven't been able to profile as the dump is too big. Would setting >> termIndexInterval help - not sure how can that be set using Solr. >> >> Some other query properties under solrconfig, >> >> <query> >> <maxBooleanClauses>1024</maxBooleanClauses> >> <enableLazyFieldLoading>true</enableLazyFieldLoading> >> <queryResultWindowSize>50</queryResultWindowSize> >> <queryResultMaxDocsCached>200</queryResultMaxDocsCached> >> <HashDocSet maxSize="3000" loadFactor="0.75"/> >> <useColdSearcher>false</useColdSearcher> >> <maxWarmingSearchers>2</maxWarmingSearchers> >> </query> >> >> Currently, I got 800 million documents and have specified 8G heap size. >> >> Any other suggestion on what can I do to control the Solr memory >> consumption? >> >> Thanks, >> -vivek >> >> On Wed, May 13, 2009 at 2:53 PM, vivek sar <vivex...@gmail.com> wrote: >> > Just an update on the memory issue - might be useful for others. I >> > read the following, >> > >> > http://wiki.apache.org/solr/SolrCaching?highlight=(SolrCaching) >> > >> > and looks like the first and new searcher listeners would populate the >> > FieldCache. Commenting out these two listener entries seems to do the >> > trick - at least the heap size is not growing as soon as Solr starts >> > up. >> > >> > I ran some searches and they all came out fine. Index rate is also >> > pretty good. Would there be any impact of disabling these listeners? >> > >> > Thanks, >> > -vivek >> > >> > On Wed, May 13, 2009 at 2:12 PM, vivek sar <vivex...@gmail.com> wrote: >> >> Otis, >> >> >> >> In that case, I'm not sure why Solr is taking up so much memory as >> >> soon as we start it up. I checked for .tii file and there is only one, >> >> >> >> -rw-r--r-- 1 search staff 20306 May 11 21:47 >> ./20090510_1/data/index/_3au.tii >> >> >> >> I have all the cache disabled - so that shouldn't be a problem too. My >> >> ramBuffer size is only 64MB. >> >> >> >> I read note on sorting, >> >> http://wiki.apache.org/solr/SchemaDesign?highlight=(sort), and see >> >> something related to FieldCache. I don't see this as parameter defined >> >> in either solrconfig.xml or schema.xml. Could this be something that >> >> can load things in memory at startup? How can we disable it? >> >> >> >> I'm trying to find out if there is a way to tell how much memory Solr >> >> would consume and way to cap it. >> >> >> >> Thanks, >> >> -vivek >> >> >> >> >> >> >> >> >> >> On Wed, May 13, 2009 at 1:50 PM, Otis Gospodnetic >> >> <otis_gospodne...@yahoo.com> wrote: >> >>> >> >>> Hi, >> >>> >> >>> Sorting is triggered by the sort parameter in the URL, not a >> characteristic of a field. :) >> >>> >> >>> Otis >> >>> -- >> >>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >>> >> >>> >> >>> >> >>> ----- Original Message ---- >> >>>> From: vivek sar <vivex...@gmail.com> >> >>>> To: solr-user@lucene.apache.org >> >>>> Sent: Wednesday, May 13, 2009 4:42:16 PM >> >>>> Subject: Re: Solr memory requirements? >> >>>> >> >>>> Thanks Otis. >> >>>> >> >>>> Our use case doesn't require any sorting or faceting. I'm wondering if >> >>>> I've configured anything wrong. >> >>>> >> >>>> I got total of 25 fields (15 are indexed and stored, other 10 are just >> >>>> stored). All my fields are basic data type - which I thought are not >> >>>> sorted. My id field is unique key. >> >>>> >> >>>> Is there any field here that might be getting sorted? >> >>>> >> >>>> >> >>>> required="true" omitNorms="true" compressed="false"/> >> >>>> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> default="NOW/HOUR" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> omitNorms="true" compressed="false"/> >> >>>> >> >>>> compressed="false"/> >> >>>> >> >>>> default="NOW/HOUR" omitNorms="true"/> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> omitNorms="true" multiValued="true"/> >> >>>> >> >>>> Thanks, >> >>>> -vivek >> >>>> >> >>>> On Wed, May 13, 2009 at 1:10 PM, Otis Gospodnetic >> >>>> wrote: >> >>>> > >> >>>> > Hi, >> >>>> > Some answers: >> >>>> > 1) .tii files in the Lucene index. When you sort, all distinct >> values for the >> >>>> field(s) used for sorting. Similarly for facet fields. Solr caches. >> >>>> > 2) ramBufferSizeMB dictates, more or less, how much Lucene/Solr will >> consume >> >>>> during indexing. There is no need to commit every 50K docs unless you >> want to >> >>>> trigger snapshot creation. >> >>>> > 3) see 1) above >> >>>> > >> >>>> > 1.5 billion docs per instance where each doc is cca 1KB? I doubt >> that's going >> >>>> to fly. :) >> >>>> > >> >>>> > Otis >> >>>> > -- >> >>>> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >>>> > >> >>>> > >> >>>> > >> >>>> > ----- Original Message ---- >> >>>> >> From: vivek sar >> >>>> >> To: solr-user@lucene.apache.org >> >>>> >> Sent: Wednesday, May 13, 2009 3:04:46 PM >> >>>> >> Subject: Solr memory requirements? >> >>>> >> >> >>>> >> Hi, >> >>>> >> >> >>>> >> I'm pretty sure this has been asked before, but I couldn't find a >> >>>> >> complete answer in the forum archive. Here are my questions, >> >>>> >> >> >>>> >> 1) When solr starts up what does it loads up in the memory? Let's >> say >> >>>> >> I've 4 cores with each core 50G in size. When Solr comes up how >> much >> >>>> >> of it would be loaded in memory? >> >>>> >> >> >>>> >> 2) How much memory is required during index time? If I'm committing >> >>>> >> 50K records at a time (1 record = 1KB) using solrj, how much memory >> do >> >>>> >> I need to give to Solr. >> >>>> >> >> >>>> >> 3) Is there a minimum memory requirement by Solr to maintain a >> certain >> >>>> >> size index? Is there any benchmark on this? >> >>>> >> >> >>>> >> Here are some of my configuration from solrconfig.xml, >> >>>> >> >> >>>> >> 1) 64 >> >>>> >> 2) All the caches (under query tag) are commented out >> >>>> >> 3) Few others, >> >>>> >> a) true ==> >> >>>> >> would this require memory? >> >>>> >> b) 50 >> >>>> >> c) 200 >> >>>> >> d) >> >>>> >> e) false >> >>>> >> f) 2 >> >>>> >> >> >>>> >> The problem we are having is following, >> >>>> >> >> >>>> >> I've given Solr RAM of 6G. As the total index size (all cores >> >>>> >> combined) start growing the Solr memory consumption goes up. With >> 800 >> >>>> >> million documents, I see Solr already taking up all the memory at >> >>>> >> startup. After that the commits, searches everything become slow. >> We >> >>>> >> will be having distributed setup with multiple Solr instances >> (around >> >>>> >> 8) on four boxes, but our requirement is to have each Solr instance >> at >> >>>> >> least maintain around 1.5 billion documents. >> >>>> >> >> >>>> >> We are trying to see if we can somehow reduce the Solr memory >> >>>> >> footprint. If someone can provide a pointer on what parameters >> affect >> >>>> >> memory and what effects it has we can then decide whether we want >> that >> >>>> >> parameter or not. I'm not sure if there is any minimum Solr >> >>>> >> requirement for it to be able maintain large indexes. I've used >> Lucene >> >>>> >> before and that didn't require anything by default - it used up >> memory >> >>>> >> only during index and search times - not otherwise. >> >>>> >> >> >>>> >> Any help is very much appreciated. >> >>>> >> >> >>>> >> Thanks, >> >>>> >> -vivek >> >>>> > >> >>>> > >> >>> >> >>> >> >> >> > >> >
