Good info to have. Thanks Erick. -----Original Message----- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Tuesday, April 07, 2009 10:51 AM To: solr-user@lucene.apache.org Subject: Re: Coming up with a model of memory usage
Your observations about date sorting are probably correct. The issue is that the sort caches in Lucene look at the unique terms. There are many more unique terms (nearly every one) in 2008-08-12T12:18:26.510 then when the field is split. You can reduce memory consumption when sorting even more by splitting into more fields, but that's up to you to decide whether or not it's worth the effort.... Best Erick On Tue, Apr 7, 2009 at 10:55 AM, Joe Pollard <joe.poll...@bazaarvoice.com>wrote: > It doesn't seem to matter whether fields are stored or not, but I've > found a rather striking difference in the memory requirements during > sorting. Sorting on a string field representing datetime like > '2008-08-12T12:18:26.510' is about twice as memory intense as sorting > first by '2008-08-12' and then by '121826'. > > Any other tips/guidance like this would be great! > > Thanks, > -Joe > > On Mon, 2009-04-06 at 15:43 -0500, Joe Pollard wrote: > > To combat our frequent OutOfMemory Exceptions, I'm attempting to come up > > with a model so that we can determine how much memory to give Solr based > > on how much data we have (as we expand to more data types eligible to be > > supported this becomes more important). > > > > Are there any published guidelines on how much memory a particular > > document takes up in memory, based on the data types, etc? > > > > I have several stored fields, numerous other non-stored fields, a > > largish copyTo field, and I am doing some sorting on indexed, non-stored > > fields. > > > > Any pointers would be appreciated! > > > > Thanks, > > -Joe > > > >
