Re: Document Cache

2016-03-19 Thread Emir Arnautovic
Hi, Your cache will be cleared on soft commits - every two minutes. It seems that it is either configured to be huge or you have big documents and retrieving all fields or dont have lazy field loading set to true. Can you please share your document cache config and heap settings. Thanks, Emir

Re: Document Cache

2016-03-19 Thread Emir Arnautovic
Problem starts with autowarmCount="5000" - that executes 5000 queries when new searcher is created and as queries are executed, document cache is filled. If you have large queryResultWindowSize and queries return big number of documents, that will eat up memory before new search is executed. It

Re: Document Cache

2016-03-19 Thread Rallavagu
comments in line... On 3/17/16 2:16 PM, Erick Erickson wrote: First, I want to make sure when you say "TTL", you're talking about documents being evicted from the documentCache and not the "Time To Live" option whereby documents are removed completely from the index. May be TTL was not the rig

Re: Document Cache

2016-03-19 Thread Emir Arnautovic
Running single query that returns all docs and all fields will actually load as many document as queryResultWindowSize is. What you need to do is run multiple queries that will return different documents. In case your id is numeric, you can run something like id:[1 TO 100] and then id:[100 TO 20

Re: Document Cache

2016-03-19 Thread Rallavagu
On 3/18/16 9:27 AM, Emir Arnautovic wrote: Running single query that returns all docs and all fields will actually load as many document as queryResultWindowSize is. What you need to do is run multiple queries that will return different documents. In case your id is numeric, you can run somethi

Re: Document Cache

2016-03-19 Thread Rallavagu
On 3/18/16 8:56 AM, Emir Arnautovic wrote: Problem starts with autowarmCount="5000" - that executes 5000 queries when new searcher is created and as queries are executed, document cache is filled. If you have large queryResultWindowSize and queries return big number of documents, that will eat

Re: Document Cache

2016-03-19 Thread Rallavagu
Thanks for the recommendations Shawn. Those are the lines I am thinking as well. I am reviewing application also. Going with the note on cache invalidation for every two minutes due to soft commit, wonder how would it go OOM in simply two minutes or is it likely that a thread is holding the se

Re: Document Cache

2016-03-19 Thread Erick Erickson
First, I want to make sure when you say "TTL", you're talking about documents being evicted from the documentCache and not the "Time To Live" option whereby documents are removed completely from the index. The time varies with the number of new documents fetched. This is an LRU cache whose size is

Re: Document Cache

2016-03-18 Thread Shawn Heisey
On 3/18/2016 8:22 AM, Rallavagu wrote: > So, each soft commit would create a new searcher that would invalidate > the old cache? > > Here is the configuration for Document Cache > > initialSize="10" autowarmCount="0"/> > > true In an earlier message, you indicated you're running into OOM. I

Re: Document Cache

2016-03-18 Thread Rallavagu
So, each soft commit would create a new searcher that would invalidate the old cache? Here is the configuration for Document Cache autowarmCount="0"/> true Thanks On 3/18/16 12:45 AM, Emir Arnautovic wrote: Hi, Your cache will be cleared on soft commits - every two minutes. It seems that i

Re: document cache

2012-05-15 Thread Erick Erickson
Yes. In fact, all the caches get flushed on every commit/replication cycle. Some of the caches get autowarmed when a new searcher is opened, which happens...you guessed it...every time a commit/replication happens. Best Erick On Tue, May 15, 2012 at 1:32 AM, shinkanze wrote: >  hi , > > I want