Jack, Sorry for the delayed response: Total memory allocated : 3GB Free Memory on startup of application server : 2.85GB (95%) Free Memory after first request by first user(1 request involves 3 queries) : 2.7GB (90%) Free Memory after a few requests by same user : 2.52GB (84%)
All values recorded above have been done after 2 force GCs were done to identify the free memory. The progression of memory usage looks quite high with the above numbers. As the number of searches widen, the speed of memory consumption decreases. But at some point it does hit OOM. - Rahul On Thu, May 3, 2012 at 8:37 PM, Jack Krupansky <j...@basetechnology.com>wrote: > Just for a baseline, how much memory is available in the JVM (using > jconsole or something similar) before you do your first query, and then > after your first query (that has these 50-70 facets), and then after a few > different queries (different facets.) Just to see how close you are to "the > edge" even before a volume of queries start coming in. > > > -- Jack Krupansky > > -----Original Message----- From: Rahul R > Sent: Thursday, May 03, 2012 1:28 AM > > To: solr-user@lucene.apache.org > Subject: Re: Lucene FieldCache - Out of memory exception > > Jack, > Yes, the queries work fine till I hit the OOM. The fields that start with > S_* are strings, F_* are floats, I_* are ints and so so. The dynamic field > definitions from schema.xml : > <dynamicField name="S_*" type="string" indexed="true" stored="true" > omitNorms="true"/> > <dynamicField name="I_*" type="sint" indexed="true" stored="true" > omitNorms="true"/> > <dynamicField name="F_*" type="sfloat" indexed="true" stored="true" > omitNorms="true"/> > <dynamicField name="D_*" type="date" indexed="true" stored="true" > omitNorms="true"/> > <dynamicField name="B_*" type="boolean" indexed="true" stored="true" > omitNorms="true"/> > > *Each FieldCache will be an array with maxdoc entries (your total number of > > documents - 1.4 million) times the size of the field value or whatever a > string reference is in your JVM* > > So if I understand correct - every field (dynamic or normal) will have its > own field cache. The size of the field cache for any field will be (maxDocs > * sizeOfField) ? If the field has only 100 unique values, will it occupy > (100 * sizeOfField) or will it still be (maxDocs * sizeOfField) ? > > *Roughly what is the typical or average length of one of your facet field > > values? And, on average, how many unique terms are there within a typical > faceted field?* > > Each field length may vary from 10 - 30 characters. Average of 20 maybe. > Number of unique terms within a faceted field will vary from 100 - 1000. > Average of 300. How will the number of unique terms affect performance ? > > *3 GB sounds like it might not be enough for such heavy use of faceting. It > > is probably not the 50-70 number, but the 440 or accumulated number across > many queries that pushes the memory usage up* > > I am using jdk1.5.0_14 - 32 bit. With 32 bit jdk, I think there is a > limitation that more RAM cannot be allocated. > > *When you hit OOM, what does the Solr admin stats display say for > FieldCache?* > > I don't have solr deployed as a separate web app. All solr jar files are > present in my webapp's WEB-INF\lib directory. I use EmbeddedSolrServer. So > is there a way I can get this information that the admin would show ? > > Thank you for your time. > > -Rahul > > > On Wed, May 2, 2012 at 5:19 PM, Jack Krupansky <j...@basetechnology.com>** > wrote: > > The FieldCache gets populated the first time a given field is referenced >> as a facet and then will stay around forever. So, as additional queries >> get >> executed with different facet fields, the number of FieldCache entries >> will >> grow. >> >> If I understand what you have said, theses faceted queries do work >> initially, but after awhile they stop working with OOM, correct? >> >> The size of a single FieldCache depends on the field type. Since you are >> using dynamic fields, it depends on your "dynamicField" types - which you >> have not told us about. From your query I see that your fields start with >> "S_" and "F_" - presumably you have dynamic field types "S_*" and "F_*"? >> Are they strings, integers, floats, or what? >> >> Each FieldCache will be an array with maxdoc entries (your total number of >> documents - 1.4 million) times the size of the field value or whatever a >> string reference is in your JVM. >> >> String fields will take more space than numeric fields for the FieldCache, >> since a separate table is maintained for the unique terms in that field. >> Roughly what is the typical or average length of one of your facet field >> values? And, on average, how many unique terms are there within a typical >> faceted field? >> >> If you can convert many of these faceted fields to simple integers the >> size should go down dramatically, but that depends on your application. >> >> 3 GB sounds like it might not be enough for such heavy use of faceting. It >> is probably not the 50-70 number, but the 440 or accumulated number across >> many queries that pushes the memory usage up. >> >> When you hit OOM, what does the Solr admin stats display say for >> FieldCache? >> >> -- Jack Krupansky >> >> -----Original Message----- From: Rahul R >> Sent: Wednesday, May 02, 2012 2:22 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Lucene FieldCache - Out of memory exception >> >> >> Here is one sample query that I picked up from the log file : >> >> q=*%3A*&fq=Category%3A%223__****107%22&fq=S_P1540477699%3A%** >> 22MICROCIRCUIT%2C+LINE+****TRANSCEIVERS%22&rows=0&facet=**** >> true&facet.mincount=1&facet.****limit=2&facet.field=S_** >> C1503120369&facet.field=S_****P1406389942&facet.field=S_** >> P1430116878&facet.field=S_****P1430116881&facet.field=S_** >> P1406453552&facet.field=S_****P1406451296&facet.field=S_** >> P1406452465&facet.field=S_****C2968809156&facet.field=S_** >> P1406389980&facet.field=S_****P1540477699&facet.field=S_** >> P1406389982&facet.field=S_****P1406389984&facet.field=S_** >> P1406451284&facet.field=S_****P1406389926&facet.field=S_** >> P1424886581&facet.field=S_****P2017662632&facet.field=F_** >> P1946367021&facet.field=S_****P1430116884&facet.field=S_** >> P2017662620&facet.field=F_****P1406451304&facet.field=F_** >> P1406451306&facet.field=F_****P1406451308&facet.field=S_** >> P1500901421&facet.field=S_****P1507138990&facet.field=I_** >> P1406452433&facet.field=I_****P1406453565&facet.field=I_** >> P1406452463&facet.field=I_****P1406453573&facet.field=I_** >> P1406451324&facet.field=I_****P1406451288&facet.field=S_** >> P1406451282&facet.field=S_****P1406452471&facet.field=S_****P14248866 >> 05&facet.field=S_P1946367015&****facet.field=S_P1424886598&** >> facet.field=S_P1946367018&****facet.field=S_P1406453556&** >> facet.field=S_P1406389932&****facet.field=S_P2017662623&** >> facet.field=S_P1406450978&****facet.field=F_P1406452455&** >> facet.field=S_P1406389972&****facet.field=S_P1406389974&** >> facet.field=S_P1406389986&****facet.field=F_P1946367027&** >> facet.field=F_P1406451294&****facet.field=F_P1406451286&** >> facet.field=F_P1406451328&****facet.field=S_P1424886593&** >> facet.field=S_P1406453567&****facet.field=S_P2017662629&** >> facet.field=S_P1406453571&****facet.field=F_P1946367030&** >> facet.field=S_P1406453569&****facet.field=S_P2017662626&** >> facet.field=S_P1406389978&****facet.field=F_P1946367024 >> >> >> My primary question here is, can Solr handle this kind of queries with so >> many facet fields. I have tried using both enum and fc for facet.method >> and >> there is no improvement with either. >> >> Appreciate any help on this. Thank you. >> >> - Rahul >> >> >> On Mon, Apr 30, 2012 at 2:53 PM, Rahul R <rahul.s...@gmail.com> wrote: >> >> Hello, >> >>> I am using solr 1.3 with jdk 1.5.0_14 and weblogic 10MP1 application >>> server on Solaris. I use embedded solr server. More details : >>> Number of docs in solr index : 1.4 million >>> Physical size of index : 640MB >>> Total number of fields in the index : 700 (99% of these are dynamic >>> fields) >>> Total number of fields enabled for faceting : 440 >>> Avg number of facet fields participating in a faceted query : 50-70 >>> Total RAM allocated to weblogic appserver : 3GB (max possible) >>> >>> In a multi user environment with 3 users using this application for a >>> period of around 40 minutes, the application runs out of memory. Analysis >>> of the heap dump shows that almost 85% of the memory is retained by the >>> FieldCache. Now I understand that the field cache is out of our control >>> but >>> would appreciate some suggestions on how to handle this issue. >>> >>> Some questions on this front : >>> - some mail threads on this forum seem to indicate that there could be >>> some connection between having dynamic fields and usage of FieldCache. Is >>> this true ? Most of the fields in my index are dynamic fields. >>> - as mentioned above, most of my faceted queries could have around 50-70 >>> facet fields (I would do SolrQuery.addFacetField() for around 50-70 >>> fields >>> per query). Could this be the source of the problem ? Is this too high >>> for >>> solr to support ? >>> - Initially, I had a facet.sort defined in solrconfig.xml. Since >>> FieldCache builds up on sorting, I even removed the facet.sort and tried, >>> but no respite. The behavior is same as before. >>> - The document id that I have for each document is quite big (around 50 >>> characters on average). Can this be a problem ? I reduced this to around >>> 15 >>> characters and tried but still there is no improvement. >>> - Can the size of the data be a problem ? But on this forum, I see many >>> users talking of more than 100 million documents in their index. I have >>> only 1.4 million with physical size of 640MB. The physical server on >>> which >>> this application is running, has sufficient RAM and CPU. >>> - What gets stored in the FieldCache ? Is it the entire document or just >>> the document Id ? >>> >>> >>> Any help is much appreciated. Thank you. >>> >>> regards >>> Rahul >>> >>> >>> >>> >>> >> >
