Hmmm, I'm missing something here then. Sorting over 15 fields of type long shouldn't use much memory, even if all the values are unique. When you say "12-15 dynamic fields", are you talking about 12-15 fields per query out of XXX total fields? And is XXX large? At a guess, how many different fields do you think you're sorting over cumulative by the time you get your OOM? Note if you sort over the field "erick_time" in 10 different queries, I'm only counting that as 1 field. I guess another way of asking this is "how many dynamic fields are there total?".
If this is really a sorting issue, you should be able to force this to happen almost immediately by firing off enough sort queries at the server. It'll tell you a lot if you can't make this happen, even on a relatively small test machine. Best Erick On Tue, Jun 22, 2010 at 12:59 PM, Matteo Fiandesio < matteo.fiande...@gmail.com> wrote: > Hi Erick, > the index is quite small (1691145 docs) but sorting is massive and > often on unique timestamp fields. > > OOM occur after a range of time between three and four hours. > Depending as well if users browse a part of the application. > > We use solrj to make the queries so we did not use Readers objects > directly. > > Without sorting we don't see the problem > Regards, > Matteo > > On 22 June 2010 17:01, Erick Erickson <erickerick...@gmail.com> wrote: > > Hmmmm.. A couple of details I'm wondering about. How many > > documents are we talking about in your index? Do you get > > OOMs when you start fresh or does it take a while? > > > > You've done some good investigations, so it seems like there > > could well be something else going on here than just "the usual > > suspects" of sorting.... > > > > I'm wondering if you aren't really closing readers somehow. > > Are you updating your index frequently and re-opening readers often? > > If so, how? > > > > I'm assuming that if you do NOT sort on all these fields, you don't have > > the problem, is that true? > > > > Best > > Erick > > > > On Fri, Jun 18, 2010 at 10:52 AM, Matteo Fiandesio < > > matteo.fiande...@gmail.com> wrote: > > > >> Hello, > >> we are experiencing OOM exceptions in our single core solr instance > >> (on a (huge) amazon EC2 machine). > >> We investigated a lot in the mailing list and through jmap/jhat dump > >> analyzing and the problem resides in the lucene FieldCache that fills > >> the heap and blows up the server. > >> > >> Our index is quite small but we have a lot of sort queries on fields > >> that are dynamic,of type long representing timestamps and are not > >> present in all the documents. > >> Those queries apply sorting on 12-15 of those fields. > >> > >> We are using solr 1.4 in production and the dump shows a lot of > >> Integer/Character and Byte Array filled up with 0s. > >> With solr's trunk code things does not change. > >> > >> In the mailing list we saw a lot of messages related to this issues: > >> we tried truncating the dates to day precision,using missingSortLast = > >> true,changing the field type from slong to long,setting autowarming to > >> different values,disabling and enabling caches with different values > >> but we did not manage to solve the problem. > >> > >> We were thinking to implement an LRUFieldCache field type to manage > >> the FieldCache as an LRU and preventing but, before starting a new > >> development, we want to be sure that we are not doing anything wrong > >> in the solr configuration or in the index generation. > >> > >> Any help would be appreciated. > >> Regards, > >> Matteo > >> > > >
