Could Solr just load cores one at a time, waiting for loader events to finish? Or continuously stage 2 or three simultaneously?
On Sat, Oct 2, 2010 at 7:08 AM, Yonik Seeley <yo...@lucidimagination.com> wrote: > On Fri, Oct 1, 2010 at 5:42 PM, Renee Sun <renee_...@mcafee.com> wrote: >> Hi Yonik, >> >> I attached the solrconfig.xml to you in previous post, and we do have >> firstSearch and newSearch hook ups. >> >> I commented them out, all 130 cores loaded up in 1 minute, same as in solr >> 1.3. total memory took about 1GB. Whereas in 1.3, with hook ups, it took >> about 6.5GB for same amount of data. > > For other's reference: here is the warming query (it's the same for > newSearcher too): > > <listener event="firstSearcher" class="solr.QuerySenderListener"> > <arr name="queries"> > <lst> > <str name="q">type:message</str> > <str name="start">0</str> > <str name="rows">10</str> > <str name="sort">message_date desc</str> > </lst> > </arr> > </listener> > > The sort field message_date is what will be taking up the memory. > > Starting with Lucene 2.9 (which is used in Solr 1.4), searching and > sorting is per-segment. > This is generally beneficial, but in this case I believe it is causing > the extra memory usage because the same date value that would have > been shared across all documents in the fieldcache is now repeated in > each segment it is used in. > > One potential fix (that requires you to reindex) is to use the "date" > fieldType as defined in the new 1.4 schema: > <fieldType name="date" class="solr.TrieDateField" omitNorms="true" > precisionStep="0" positionIncrementGap="0"/> > > This will use 8 bytes per document in your index, rather than 4 bytes > per doc + an array of unique string-date values per index. > > Trunk (4.0-dev) is also much more efficient at storing string-based > fields in the FieldCache - but that will only help you if you're > comfortable with using development versions. > > -Yonik > http://lucenerevolution.org Lucene/Solr Conference, Boston Oct 7-8 > -- Lance Norskog goks...@gmail.com
