Re: Re: solr 4.2.1 index gets slower over time

2014-04-02 Thread elisabeth benoit
This sounds interesting, I'll check this out. Thanks! Elisabeth 2014-04-02 8:54 GMT+02:00 Dmitry Kan : > Thanks, Markus, that is useful. > I'm guessing the higher the weight, the longer the op takes? > > > On Tue, Apr 1, 2014 at 10:39 PM, Markus Jelsma > wrote: > > > You may want to increase re

Re: Re: solr 4.2.1 index gets slower over time

2014-04-01 Thread Dmitry Kan
Thanks, Markus, that is useful. I'm guessing the higher the weight, the longer the op takes? On Tue, Apr 1, 2014 at 10:39 PM, Markus Jelsma wrote: > You may want to increase reclaimdeletesweight for tieredmergepolicy from 2 > to 3 or 4. By default it may keep too much deleted or updated docs in

Re: Re: solr 4.2.1 index gets slower over time

2014-04-01 Thread Markus Jelsma
You may want to increase reclaimdeletesweight for tieredmergepolicy from 2 to 3 or 4. By default it may keep too much deleted or updated docs in the index. This can increase index size by 50%!! Dmitry Kan schreef:Elisabeth, Yes, I believe you are right in that the deletes are part of the optim

Re: solr 4.2.1 index gets slower over time

2014-04-01 Thread Dmitry Kan
Elisabeth, Yes, I believe you are right in that the deletes are part of the optimize process. If you delete often, you may consider (if not already) the TieredMergePolicy, which is suited for this scenario. Check out this relevant discussion I had with Lucene committers: https://twitter.com/Dmitry

Re: solr 4.2.1 index gets slower over time

2014-04-01 Thread elisabeth benoit
Thanks a lot for your answers! Shawn. Our GC configuration has far less parameters defined, so we'll check this out. Dimitry, about the expungeDeletes option, we'll add that in the delete process. But from what I read, this is done in the optimize process (cf. http://lucene.472066.n3.nabble.com/D

Re: solr 4.2.1 index gets slower over time

2014-03-31 Thread Dmitry Kan
Hi, We have noticed something like this as well, but with older versions of solr, 3.4. In our setup we delete documents pretty often. Internally in Lucene, when a document is client requested to be deleted, it is not physically deleted, but only marked as "deleted". Our original optimization assum

Re: solr 4.2.1 index gets slower over time

2014-03-31 Thread Shawn Heisey
On 3/31/2014 9:03 AM, elisabeth benoit wrote: We use JVisualVM. The CPU usage is very high (90%), but the GC activity shows less than 0.01% average activity. Plus the heap usage stays low (below 4G while the max heap size is 16G). Do you have a different tool to suggest to check the GC? Do you t

Re: solr 4.2.1 index gets slower over time

2014-03-31 Thread elisabeth benoit
Hello, Thanks for your answer. We use JVisualVM. The CPU usage is very high (90%), but the GC activity shows less than 0.01% average activity. Plus the heap usage stays low (below 4G while the max heap size is 16G). Do you have a different tool to suggest to check the GC? Do you think there is s

Re: solr 4.2.1 index gets slower over time

2014-03-31 Thread Shawn Heisey
On 3/31/2014 6:57 AM, elisabeth benoit wrote: > We are currently using solr 4.2.1. Our index is updated on a daily basis. > After noticing solr query time has increased (two times the initial size) > without any change in index size or in solr configuration, we tried an > optimize on the index but

solr 4.2.1 index gets slower over time

2014-03-31 Thread elisabeth benoit
Hello, We are currently using solr 4.2.1. Our index is updated on a daily basis. After noticing solr query time has increased (two times the initial size) without any change in index size or in solr configuration, we tried an optimize on the index but it didn't fix our problem. We checked the garb