Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Shawn Heisey
On 7/12/2016 9:45 AM, Jason wrote: > I'm using optimize because it's a option for fast search. Our index > updates one or more weekly. If I don't use optimize, many index files > should be kept. Any performance issues in that case? And I'm wondering > relation between index file size and heap size.

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Otis Gospodnetic
Heap: start small and increase as necessary. Leave as much RAM for FS cache, don't give it to the JVM until it starts crying. SPM for Solr will help you see when Solr and JVM are starting to hurt. Otis > On Jul 12, 2016, at 11:45, Jason wrote: > > I'm using optimize because it's a option for

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Erick Erickson
It's more a matter of "is unoptimized fast enough"? If so, why bother? The background merging will keep segment counts relatively reasonable. If you're updating your index only once a week, it's reasonable to optimize. Anecdotal reports are of on the order of a 10% speedup _at best_. As Yonik sa

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Jason
I'm using optimize because it's a option for fast search. Our index updates one or more weekly. If I don't use optimize, many index files should be kept. Any performance issues in that case? And I'm wondering relation between index file size and heap size. In case of running as master server that

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Jason
Let me know the guide reference address which is mentioned reasonable index size is around 15G. -- View this message in context: http://lucene.472066.n3.nabble.com/High-cpu-and-gc-time-when-performing-optimization-tp4286704p4286790.html Sent from the Solr - User mailing list archive at Nabble.

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Yonik Seeley
Optimize is a very expensive operation. It involves reading the entire index and merging and rewriting at a single segment. If you find it too expensive, do it less often, or don't do it at all. It's an optional operation. -Yonik On Mon, Jul 11, 2016 at 10:19 PM, Jason wrote: > hi, all. > > I'

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Kent Mu
as I said before. we also come across the issue. and I just guess the possible reason. let's wait the expert to explain for us. on the other hand. I find that your index data is 68G, that is too large, I recommend you to use solrcloud, as the guide reference, the reasonable size is around 15G. now

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Jason
hi, Kent thanks your reply. I think that I need more explain to my server status. I'm using solr 4.2.1 and master-slave replication model. On master server many solr(tomcat) instances are running. (server has 64 cores, 128G ram.) Now 4 solr(tomcat) instances are running and are allocated 32, 16, 1

Re: High cpu and gc time when performing optimization.

2016-07-12 Thread Kent Mu
we also came across this issue. I think it is not caused by gc time, but the optimize action, though I did not read the source code, I think when optimize the index in master internally, it will produce the replicate log file, and the replicates synchronize the log file, just like the DB master and