At and above 4GB we get those GC errors though! Should I switch to something
like this?
Recommended Options
To use i-cms in Java SE 6, use the following command line options:
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Caused by: java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1068)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:418)
at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:467)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
... 11 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
-----Original Message-----
From: Lance Norskog [mailto:[email protected]]
Sent: Tuesday, June 29, 2010 8:42 PM
To: [email protected]
Subject: Re: OOM on uninvert field request
Yes, it is better to use ints for ids than strings. Also, the Trie int
fields have a compressed format that may cut the storage needs even
more. 8m * 4 = 32mb, times "a few hundred", we'll say 300, is 900mb of
IDs. I don't know how these fields are stored, but if they are
separate objects we've blown up to several gigs (per-object overheads
are surprising).
4G is probably not enough for what you want. If you watch the total
memory with 'top' and hit it with different queries, you will get a
stronger sense of how much memory your use cases need.
On Tue, Jun 29, 2010 at 4:32 PM, Robert Petersen <[email protected]> wrote:
> Hello I am trying to find the right max and min settings for Java 1.6 on 20GB
> index with 8 million docs, running 1.6_018 JVM with solr 1.4, and am
> currently have java set to an even 4GB (export JAVA_OPTS="-Xmx4096m
> -Xms4096m") for both min and max which is doing pretty well but occasionally
> still getting the below OOM errors. We're running on dual quad core xeons
> with 16GB memory installed. I've been getting the below OOM exceptions still
> though.
>
> Is the memsize mentioned in the INFO for the uninvert in bytes? Ie is
> memSize=29604020 mean 29MB? We have a few hundred of these fields and they
> contain ints used as IDs, and so I guess could they eat all the memory to
> uninvert them all after we apply load and enough queries are performed. Does
> the field type matter, would int be better than string if these are lookup
> ids sparsely populated across the index? BTW these are used for faceting and
> filtering only.
>
> <dynamicField name="*_contentAttributeToken" type="string"
> indexed="true" multiValued="true" stored="true" required="false"/>
>
> Jun 29, 2010 3:54:50 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field
> {field=768_contentAttributeToken,memSize=29604014,tindexSize=50,time=1841,phase1=1824,nTerms=1,bigTerms=0,termInstances=18,uses=0}
> Jun 29, 2010 3:54:52 PM org.apache.solr.request.UnInvertedField uninvert
> INFO: UnInverted multi-valued field
> {field=749_contentAttributeToken,memSize=29604020,tindexSize=56,time=1847,phase1=1829,nTerms=143,bigTerms=0,termInstances=951,uses=0}
> Jun 29, 2010 3:54:59 PM org.apache.solr.common.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.solr.request.UnInvertedField.uninvert(UnInvertedField.java:191)
> at
> org.apache.solr.request.UnInvertedField.<init>(UnInvertedField.java:178)
> at
> org.apache.solr.request.UnInvertedField.getUnInvertedField(UnInvertedField.java:839)
> at
> org.apache.solr.request.SimpleFacets.getTermCounts(SimpleFacets.java:250)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:283)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:166)
>
--
Lance Norskog
[email protected]
