Solr - multivalue fields - please help
Hi everyone, I know that Solr cannot match 1 value in a multi-valued field with the corresponding value in another multi-valued field. However my data set appears to be in that form at the moment. With that in mind does anyone know of any good articles or discussions that have addressed this issue, specifically the alternatives that can be easily done/considered etc The data is of the following format: I have an unique Employee ID field, Employer (multi-value), FromDate (multi-value) amd ToDate (multi-value). For a given employee ID I am trying to return the relevent data. For example for a ID of "21345" and emplyer "IMB" return the work dates from and to. Or for same id and 2 work dates return the company of companies that the id was associated with etc Employee ID Employer FromDate ToDate 21345 IBM 01/01/04 01/01/06 MS 01/01/07 01/01/08 BT 01/01/09 Present Any suggestions/comments/ideas/articles much appreciated... Thanks, S. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-multivalue-fields-please-help-tp2720067p2720067.html Sent from the Solr - User mailing list archive at Nabble.com.
solr OOM Crash
Helllo, We are experiencing unexplained OOM crashes. We have already seen it a few times, over our different solr instances. The crash happens only at a single shard of the collection. Environment details: 1. Solr 4.3, running on tomcat. 2. 24 Shards. 3. Indexing rate of ~800 docs per minute. Solrconfig.xml: 1. Merge factor 4 2. Sofrcommit every 10 min 3. Hardcommit every 30 min Main findings: 1. Solr logs: No query failures prior to the OOM, but DOUBLE the amount of soft and hard commits in comparison to other shards. 2. Analyzing the dump (VisualVM): Class byte[] takes 4gb out of 5gb resourced to the JVM, mainly referenced by CompressingStoredFieldsReader GC root (which by looking at the code, we suspect they were created due to CompressingSortedFieldsWriter.merge). Sub findings: 1. GC logs: Showed 108 GC fails prior to the crash. 2. CPI: Overall usage seems fine, but the % of CPU time for the GC stays high 6 min before the OOM. 3. Memory: Half an hour before OOM the usage slowly rises, until it gets to 5.4gb. Has anyone encountered higher than normal commit rate that seem to increase merge rate and cause what I described?
Re: solr OOM Crash
Hello Sébastien, Can you give some information about your environment so I can make sure we are having the same problem you had? Also, did you find out what caused the GC to go crazy or what caused the increased commit rate? Thanks, Sandra On Thu, Dec 19, 2013 at 12:34 PM, Sébastien Michel < sebastien.mic...@atos.net> wrote: > Hi Sandra, > > I'm not sure if your problem is same as ours, but we encountered the same > issue on our Solr 4.2, the major memory usage was due to > CompressingStoredFieldsReader and GC became crazy. > In our context, we have some stored fields and for some documents the > content of the text field could be huge. > > We resolved our issue with the backport of this fix : > https://issues.apache.org/jira/browse/LUCENE-4995 > > You should also upgrade to Solr 4.4 or more > > Regards, > Sébastien > > > 2013/12/12 Sandra Scott > > > Helllo, > > > > We are experiencing unexplained OOM crashes. We have already seen it a > few > > times, over our different solr instances. The crash happens only at a > > single shard of the collection. > > > > Environment details: > > 1. Solr 4.3, running on tomcat. > > 2. 24 Shards. > > 3. Indexing rate of ~800 docs per minute. > > > > Solrconfig.xml: > > 1. Merge factor 4 > > 2. Sofrcommit every 10 min > > 3. Hardcommit every 30 min > > > > Main findings: > > 1. Solr logs: No query failures prior to the OOM, but DOUBLE the amount > of > > soft and hard commits in comparison to other shards. > > 2. Analyzing the dump (VisualVM): Class byte[] takes 4gb out of 5gb > > resourced to the JVM, mainly referenced by CompressingStoredFieldsReader > GC > > root (which by looking at the code, we suspect they were created due to > > CompressingSortedFieldsWriter.merge). > > > > Sub findings: > > 1. GC logs: Showed 108 GC fails prior to the crash. > > 2. CPI: Overall usage seems fine, but the % of CPU time for the GC stays > > high 6 min before the OOM. > > 3. Memory: Half an hour before OOM the usage slowly rises, until it gets > to > > 5.4gb. > > > > Has anyone encountered higher than normal commit rate that seem to > increase > > merge rate and cause what I described? > > >
