On 11/7/2016 6:27 AM, Mugeesh Husain wrote:
> For the large amount of data set, going to implement many shard and
> many node. I am unaware of performance tuning in solr ? how people use
> or check solr performance ? Is there any open source tool or i should
> create my own for this and how ?
See
g...@gmail.com]
Sent: August-17-09 1:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Performance Tuning: segment_merge:index_update=5:1 (timing)
Fuad,
I'd recommend indexing in Hadoop, then copying the new indexes to Solr
slaves. This removes the need for Solr master servers. Of cour
Fuad,
I'd recommend indexing in Hadoop, then copying the new indexes to Solr
slaves. This removes the need for Solr master servers. Of course
you'd need a Hadoop cluster larger than the number of master servers
you have now. The merge indexes command (which can be taxing on the
servers because
-09 4:20 PM
To: solr-user@lucene.apache.org
Subject: Re: Performance Tuning: segment_merge:index_update=5:1 (timing)
BTW, what version of Solr are you on?
On Aug 13, 2009, at 1:43 PM, Fuad Efendi wrote:
> UPDATE:
>
> I have 100,000,000 new documents in 24 hours, including possible
> updat
BTW, what version of Solr are you on?
On Aug 13, 2009, at 1:43 PM, Fuad Efendi wrote:
UPDATE:
I have 100,000,000 new documents in 24 hours, including possible
updates OR
possibly adding same document several times. I have two segments now
(30Gb
total), and network is overloaded (I use web
UPDATE:
I have 100,000,000 new documents in 24 hours, including possible updates OR
possibly adding same document several times. I have two segments now (30Gb
total), and network is overloaded (I use web crawler to generate documents).
I never had more than 25,000,000 within a month before...
I r
gh
only "timestamp" field changes in existing "refreshed" document)
-Original Message-
From: Grant Ingersoll
Sent: August-11-09 9:52 PM
To: solr-user@lucene.apache.org
Subject: Re: Performance Tuning: segment_merge:index_update=5:1 (timing)
Is there a tim
Is there a time of day you could schedule merges? See
http://www.lucidimagination.com/search/document/bd53b0431f7eada5/concurrentmergescheduler_and_mergepolicy_question
Or, you might be able to implement a scheduler that only merges the
small segments, and then does the larger ones at slow ti
Forgot to add: committing only once a day
I tried mergeFactor=1000 and performance of index write was extremely good
(more than 50,000,000 updates during part of a day)
However, "commit" was taking 2 days or more and I simply killed process
(suspecting that it can break my harddrive); I had about
Never tried profiling;
3000-5000 docs per second if SOLR is not busy with segment merge;
During segment merge 99% CPU, no disk swap; I can't suspect I/O...
During document updates (small batches 100-1000 docs) only 5-15% CPU
-server 2048Gb option of JVM (which is JRockit) + 256M for RAM Buffer
Have you tried profiling? How often are you committing? Have you
looked at Garbage Collection or any of the usual suspects like that?
On Aug 11, 2009, at 4:49 PM, Fuad Efendi wrote:
In a heavily loaded Write-only Master SOLR, I have 5 minutes of RAM
Buffer
Flash / Segment Merge per 1 min
Thank you! Those are set to zero for me too (at the moment!) so I guess it's
good news.
-Original Message-
From: Mike Klaas [mailto:[EMAIL PROTECTED]
Sent: 11 January 2007 23:13
To: solr-user@lucene.apache.org
Subject: Re: Performance tuning
On 1/11/07, Stephanie Belton &l
On 1/11/07, Stephanie Belton <[EMAIL PROTECTED]> wrote:
Thanks for that. I am sorry this isn't really Solr-related but how can I
monitor the swapping if I can't rely on the output of the free command?
$ vmstat -S M 3
procs ---memory-- ---swap-- -io --system-- cpu
On 1/11/07 2:33 PM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote:
> On 1/11/07, Stephanie Belton <[EMAIL PROTECTED]> wrote:
>> The reason I am keeping a close eye on resource usage is that our traffic is
>> increasing by around 20% every month (currently over 400,000 page
>> impressions/day although n
On 1/11/07, Stephanie Belton <[EMAIL PROTECTED]> wrote:
The reason I am keeping a close eye on resource usage is that our traffic is
increasing by around 20% every month (currently over 400,000 page
impressions/day although not all of them are search queries!) and I want to
make sure we tackle an
rather keep load balancing as a last resort due to cost implications.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley
Sent: 11 January 2007 22:02
To: solr-user@lucene.apache.org
Subject: Re: Performance tuning
On 1/11/07, Stephanie Belton
On 1/11/07, Stephanie Belton <[EMAIL PROTECTED]> wrote:
Thanks for that. I am sorry this isn't really Solr-related but how can I
monitor the swapping if I can't rely on the output of the free command?
Do you think I could still achieve any significant improvements by going
through the performanc
al Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley
Sent: 11 January 2007 20:32
To: solr-user@lucene.apache.org
Subject: Re: Performance tuning
On 1/11/07, Stephanie Belton <[EMAIL PROTECTED]> wrote:
> This is the output of the free command:
>
>
age was peaking at 20% and
memory around 28% no swapping.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley
Sent: 11 January 2007 15:12
To: solr-user@lucene.apache.org
Subject: Re: Performance tuning
On 1/11/07, Stephanie Belton <[EMAIL PROTECT
Thanks for sending this link, I seem to have missed that on the wiki!
-Original Message-
From: Thorsten Scherler [mailto:[EMAIL PROTECTED]
Sent: 11 January 2007 15:06
To: solr-user@lucene.apache.org
Subject: Re: Performance tuning
On Thu, 2007-01-11 at 14:57 +, Stephanie Belton
it's been
running for 1 year and up until now the CPU usage was peaking at 20% and
memory around 28% no swapping.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley
Sent: 11 January 2007 15:12
To: solr-user@lucene.apache.org
Subject: Re: Per
On 1/11/07, Stephanie Belton <[EMAIL PROTECTED]> wrote:
Solr is now up and running on our production environment and working great.
However it is taking up a lot of extra CPU and memory (CPU usage has doubled
and memory is swapping). Is there any documentation on performance tuning?
There see
On Thu, 2007-01-11 at 14:57 +, Stephanie Belton wrote:
> Hello,
>
>
>
> Solr is now up and running on our production environment and working great.
> However it is taking up a lot of extra CPU and memory (CPU usage has doubled
> and memory is swapping). Is there any documentation on perfo
23 matches
Mail list logo
