Re: No or limited use of FieldCache

2013-09-12 Thread Per Steffensen
On 9/12/13 3:28 PM, Toke Eskildsen wrote: On Thu, 2013-09-12 at 14:48 +0200, Per Steffensen wrote: Actually some months back I made PoC of a FieldCache that could expand beyond the heap. Basically imagine a FieldCache with room for "unlimited" data-arrays, that just behind the scenes goes to mem

Re: No or limited use of FieldCache

2013-09-12 Thread Toke Eskildsen
On Thu, 2013-09-12 at 14:48 +0200, Per Steffensen wrote: > Actually some months back I made PoC of a FieldCache that could expand > beyond the heap. Basically imagine a FieldCache with room for > "unlimited" data-arrays, that just behind the scenes goes to > memory-mapped files when there is no

Re: No or limited use of FieldCache

2013-09-12 Thread Per Steffensen
Yes, thanks. Actually some months back I made PoC of a FieldCache that could expand beyond the heap. Basically imagine a FieldCache with room for "unlimited" data-arrays, that just behind the scenes goes to memory-mapped files when there is no more room on heap. Never finished it, and it migh

Re: No or limited use of FieldCache

2013-09-12 Thread Erick Erickson
Per: One thing I'll be curious about. From my reading of DocValues, it uses little or no heap. But it _will_ use memory from the OS if I followed Simon's slides correctly. So I wonder if you'll hit swapping issues... Which are better than OOMs, certainly... Thanks, Erick On Thu, Sep 12, 2013 at

Re: No or limited use of FieldCache

2013-09-11 Thread Per Steffensen
Thanks, guys. Now I know a little more about DocValues and realize that they will do the job wrt FieldCache. Regards, Per Steffensen On 9/12/13 3:11 AM, Otis Gospodnetic wrote: Per, check zee Wiki, there is a page describing docvalues. We used them successfully in a solr for analytics scenari

Re: No or limited use of FieldCache

2013-09-11 Thread Otis Gospodnetic
Per, check zee Wiki, there is a page describing docvalues. We used them successfully in a solr for analytics scenario. Otis Solr & ElasticSearch Support http://sematext.com/ On Sep 11, 2013 9:15 AM, "Michael Sokolov" wrote: > On 09/11/2013 08:40 AM, Per Steffensen wrote: > >> The reason I menti

Re: No or limited use of FieldCache

2013-09-11 Thread Michael Sokolov
On 09/11/2013 08:40 AM, Per Steffensen wrote: The reason I mention sort is that we in my project, half a year ago, have dealt with the FieldCache->OOM-problem when doing sort-requests. We basically just reject sort-requests unless they hit below X documents - in case they do we just find them w

Re: No or limited use of FieldCache

2013-09-11 Thread Per Steffensen
The reason I mention sort is that we in my project, half a year ago, have dealt with the FieldCache->OOM-problem when doing sort-requests. We basically just reject sort-requests unless they hit below X documents - in case they do we just find them without sorting and sort them ourselves afterwa

Re: No or limited use of FieldCache

2013-09-11 Thread Erick Erickson
I don't know any more than Michael, but I'd _love_ some reports from the field. There are some restriction on DocValues though, I believe one of them is that they don't really work on analyzed data FWIW, Erick On Wed, Sep 11, 2013 at 7:00 AM, Michael Sokolov < msoko...@safaribooksonline.com

Re: No or limited use of FieldCache

2013-09-11 Thread Michael Sokolov
On 9/11/13 3:11 AM, Per Steffensen wrote: Hi We have a SolrCloud setup handling huge amounts of data. When we do group, facet or sort searches Solr will use its FieldCache, and add data in it for every single document we have. For us it is not realistic that this will ever fit in memory and w