Re: Help with sort on dynamic field and out of memory error

2012-11-28 Thread Toke Eskildsen
On Wed, 2012-11-28 at 03:25 +0100, Arun Rangarajan wrote: [Sorting on 14M docs, 250 fields] > From what I have read, I understand that restricting the number of distinct > values on sortable Solr fields will bring down the fieldCache space. The > values in these sortable fields can be any integer

Re: Help with sort on dynamic field and out of memory error

2012-11-27 Thread Arun Rangarajan
Erick, Thanks for your reply. So there is no easy way to get around this problem. We have a way to rework the schema by keeping a single sort field. The dynamic fields we have are like relevance_CLASSID. The current schema has a unique key NODEID and a multi-valued field CLASSID - the relevance s

Re: Help with sort on dynamic field and out of memory error

2012-11-27 Thread Erick Erickson
I sure don't see how this can work given the constraints. Just to hold the values, assuming that each doc holds a value in 150 fields, you have 150 * 4 * 14,000,000 or 8.4G of memory required, and you just don't have that much memory to play around with. Sharding seems silly for 14M docs, but that