http://wiki.apache.org/solr/SolrPerformanceFactors#Schema_Design_Considerations
The number of indexed fields greatly increases the following: > > - Memory usage during indexing > > > - Segment merge time > > > - Optimization times > > > - Index size > > These impacts can be reduced by the use of omitNorms="true" http://lucene.472066.n3.nabble.com/What-is-omitNorms-td2987547.html 1. length normalization will not work on the specific field-- > Which means matching documents with shorter length will not be > preferred/boost over matching documents with greater length for the specific > field, at search time. > For my application, I actually prefer documents with greater length. > 2. Index time boosting will not be available on the field. > If, both the above cases are not required by you, then, you can set > "omitNorms=true" for the specific fields. > This has an added advantage, it will save you some(or a lot of) RAM also, > since, with "omitNorms=false" on total "N" fields in the index will require > RAM of size: > Total docs in index * 1 byte * N > I have a lot of fields: I count 31 without omitNorms values, which means > false by default. Gak! 11,000,000 * 1 * 31 = 31 x 10M = 310MB RAM all by itself. On Wed, Oct 26, 2011 at 1:01 PM, Fred Zimmerman <zimzaz....@gmail.com>wrote: > More on what's happening. It seems to be timing out during the commit. > > The new documents are small, but the existing index is large (11 million > records). > > INFO: Closing Searcher@4a7df6 main >> >> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} >> ... >> > > >> Oct 26, 2011 4:51:17 PM >> org.apache.solr.update.processor.LogUpdateProcessor finish >> *INFO: {commit=} 0 2453 >> **Oct 26, 2011 4:51:17 PM org.apache.solr.core.SolrCore execute >> **INFO: [] webapp=/solr path=/update >> params={waitSearcher=true&waitFlush=true&wt=javabin&commit=true&version=2} >> status=0 QTime=2453 >> *Oct 26, 2011 4:51:52 PM org.apache.solr.core.SolrCore execute >> INFO: [] webapp=/solr path=/select >> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=11576871 >> *status=0 >> QTime=35298* >> Oct 26, 2011 4:51:53 PM org.apache.solr.core.SolrCore execute >> INFO: [] webapp=/solr path=/select >> params={fl=id&wt=javabin&q=id:[*+TO+*]&rows=1&version=2} hits=11576871 >> status=0 QTime=1 >> *java.lang.OutOfMemoryError: Java heap space* >> Dumping heap to /home/bitnami/apache-solr-3.4.0/example/heaplog ... >> Heap dump file created [306866344 bytes in 32.376 secs] > > > > On Wed, Oct 26, 2011 at 11:09 AM, Fred Zimmerman <zimzaz....@gmail.com>wrote: > >> It's a small indexing job coming from nutch. >> >> 2011-10-26 15:07:29,039 WARN mapred.LocalJobRunner - job_local_0011 >> java.io.IOException: org.apache.solr.client.solrj.SolrServerException: >> Error executi$ >> at >> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$ >> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) >> at >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) >> Caused by: org.apache.solr.client.solrj.SolrServerException: Error >> executing query >> at >> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$ >> at >> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118) >> at >> org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getRec$ >> ... 3 more >> Caused by: org.apache.solr.common.SolrException: Java heap space >> java.lang.OutOfMem$ >> >> Java heap space java.lang.OutOfMemoryError: Java heap space at >> org.apache.lucene$ >> >> request: localhost/solr/select?q=id:[* TO *]&fl=id,boost,tstamp,$ >> at >> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$ >> at >> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHt$ >> at >> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.ja$ >> ... 5 more >> >> >
