Re: Out of memory on some faceting queries

2013-04-08 Thread Dotan Cohen
On Wed, Apr 3, 2013 at 8:47 PM, Shawn Heisey wrote: > On 4/2/2013 3:09 AM, Dotan Cohen wrote: >> I notice that this only occurs on queries that run facets. I start >> Solr with the following command: >> sudo nohup java -XX:NewRatio=1 -XX:+UseParNewGC >> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRema

Re: Out of memory on some faceting queries

2013-04-03 Thread Shawn Heisey
On 4/2/2013 3:09 AM, Dotan Cohen wrote: > I notice that this only occurs on queries that run facets. I start > Solr with the following command: > sudo nohup java -XX:NewRatio=1 -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled > -Dsolr.solr.home=/mnt/SolrFiles100/solr -jar >

Re: Out of memory on some faceting queries

2013-04-03 Thread Dotan Cohen
On Wed, Apr 3, 2013 at 10:11 AM, Toke Eskildsen wrote: >> However, once per day I would like to facet on the text field, >> which is a free-text field usually around 1 KiB (about 100 words), in >> order to determine what the top keywords / topics are. That query >> would take up to 200 seconds to

Re: Out of memory on some faceting queries

2013-04-03 Thread Dotan Cohen
On Tue, Apr 2, 2013 at 6:26 PM, Andre Bois-Crettez wrote: > warmupTime is available on the admin page for each type of cache (in > milliseconds) : > http://solr-box:8983/solr/#/core1/plugins/cache > > Or if you are only interested in the total : > http://solr-box:8983/solr/core1/admin/mbeans?stats

Re: Out of memory on some faceting queries

2013-04-03 Thread Toke Eskildsen
On Tue, 2013-04-02 at 17:08 +0200, Dotan Cohen wrote: > Most of the time I facet on one field that has about twenty unique > values. They are likely to be disk cached so warming those for 9M documents should only take a few seconds. > However, once per day I would like to facet on the text field,

Re: Out of memory on some faceting queries

2013-04-02 Thread Andre Bois-Crettez
On 04/02/2013 05:04 PM, Dotan Cohen wrote: How might I time the warming? I've been googling warming since your earlier message but there does not seem to be any really good documentation on the subject. If there is anything that you feel I should be reading I would appreciate a link or a keyword

Re: Out of memory on some faceting queries

2013-04-02 Thread Dotan Cohen
> How often do you commit and how many unique values does your facet > fields have? > Most of the time I facet on one field that has about twenty unique values. However, once per day I would like to facet on the text field, which is a free-text field usually around 1 KiB (about 100 words), in orde

Re: Out of memory on some faceting queries

2013-04-02 Thread Dotan Cohen
On Tue, Apr 2, 2013 at 5:33 PM, Toke Eskildsen wrote: > Memory does not help you if you commit too frequently. If you commit > each X seconds and warming takes X+Y seconds, then you will run out of > memory at some point. > How might I time the warming? I've been googling warming since your earli

Re: Out of memory on some faceting queries

2013-04-02 Thread Dotan Cohen
On Tue, Apr 2, 2013 at 5:33 PM, Toke Eskildsen wrote: > On Tue, 2013-04-02 at 15:55 +0200, Dotan Cohen wrote: > > [Tokd: maxWarmingSearchers limit exceeded?] > >> Thank you Toke, this is exactly on my "list of things to learn about >> Solr". We do get the error mentioned and we cannot reduce the a

Re: Out of memory on some faceting queries

2013-04-02 Thread Toke Eskildsen
On Tue, 2013-04-02 at 15:55 +0200, Dotan Cohen wrote: [Tokd: maxWarmingSearchers limit exceeded?] > Thank you Toke, this is exactly on my "list of things to learn about > Solr". We do get the error mentioned and we cannot reduce the amount > of commits. Also, I do believe that we have the necessa

Re: Out of memory on some faceting queries

2013-04-02 Thread Dotan Cohen
On Tue, Apr 2, 2013 at 2:41 PM, Toke Eskildsen wrote: > 9M documents in a heavily updated index with faceting. Maybe you are > committing faster than the faceting can be prepared? > https://wiki.apache.org/solr/FAQ#What_does_.22exceeded_limit_of_maxWarmingSearchers.3DX.22_mean.3F > Thank you Toke

Re: Out of memory on some faceting queries

2013-04-02 Thread Toke Eskildsen
On Tue, 2013-04-02 at 12:16 +0200, Dotan Cohen wrote: > 8971763 documents, growing at a rate of about 500 per minute. We > actually expect that to be ~5 per minute once we get out of > testing. 9M documents in a heavily updated index with faceting. Maybe you are committing faster than the face

Re: Out of memory on some faceting queries

2013-04-02 Thread Dotan Cohen
On Tue, Apr 2, 2013 at 12:59 PM, Toke Eskildsen wrote: > How many documents does your index have, how many fields do you facet on > and approximately how many unique values does your facet fields have? > 8971763 documents, growing at a rate of about 500 per minute. We actually expect that to be

Re: Out of memory on some faceting queries

2013-04-02 Thread Toke Eskildsen
On Tue, 2013-04-02 at 11:09 +0200, Dotan Cohen wrote: > On some queries I get out of memory errors: > > {"error":{"msg":"java.lang.OutOfMemoryError: Java heap [...] > org.apache.lucene.index.DocTermOrds.uninvert(DocTermOrds.java:273)\n\tat > org.apache.solr.request.UnInvertedField.(UnInvertedField

Out of memory on some faceting queries

2013-04-02 Thread Dotan Cohen
On some queries I get out of memory errors: {"error":{"msg":"java.lang.OutOfMemoryError: Java heap space","trace":"java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space\n\tat org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:462)\n\tat org.apache.solr.s