Re: Uneven shard heap usage

2014-06-02 Thread Michael Sokolov
nd how much query or update activity is happening in that time? Is there any other activity showing up in the log? If you bring up only a single node in that problematic shard, do you still see the problem? -- Jack Krupansky -Original Message- From: Joe Gresock Sent: Saturday, May 3

Re: Uneven shard heap usage

2014-06-02 Thread Joe Gresock
t; >>>>>> JVM > >>>>> > >>>>>> parameters are the same? Grasping at straws here. > >>>>>>> > >>>>>>> More grasping at straws: I'm a little suspicious that you are using > >>>>>>&

Re: Uneven shard heap usage

2014-06-02 Thread Erick Erickson
gt;>>> More grasping at straws: I'm a little suspicious that you are using >>>>>>> routing. You say that the indexes are about the same size, but is it >>>>>>> >>>>>> is >>>>> >>>>>> possible that your routing is s

Re: Uneven shard heap usage

2014-06-02 Thread Michael Sokolov
in that problematic shard, do you still see the problem? -- Jack Krupansky -Original Message- From: Joe Gresock Sent: Saturday, May 31, 2014 9:34 AM To: solr-user@lucene.apache.org Subject: Uneven shard heap usage Hi folks, I'm trying to figure out why one shard of an evenly-distr

Re: Uneven shard heap usage

2014-06-02 Thread Joe Gresock
gt; On Sat, May 31, 2014 at 8:33 AM, Joe Gresock >>> wrote: >>> > > >>> > > > It has taken as little as 2 minutes to happen the last time we >>> tried. >>> > It >>> > > > basically happens upon high query load (peak us

Re: Uneven shard heap usage

2014-06-02 Thread Joe Gresock
pens upon high query load (peak user hours during the >> > day). >> > > > When we reduce functionality by disabling most searches, it >> > stabilizes. >> > > > So it really is only on high query load. Our ingest rate is fairly >> > low. >>

Re: Uneven shard heap usage

2014-06-01 Thread Joe Gresock
t; > > > > Joe > > > > > > > > > > > > On Sat, May 31, 2014 at 11:04 AM, Jack Krupansky < > > > j...@basetechnology.com> > > > > wrote: > > > > > > > > > When you restart, how long does it take it h

Re: Uneven shard heap usage

2014-05-31 Thread Otis Gospodnetic
n Sat, May 31, 2014 at 11:04 AM, Jack Krupansky < > > j...@basetechnology.com> > > > wrote: > > > > > > > When you restart, how long does it take it hit the problem? And how > > much > > > > query or update activity is happe

Re: Uneven shard heap usage

2014-05-31 Thread Michael Sokolov
up only a single node in that problematic shard, do you still see the problem? -- Jack Krupansky -Original Message- From: Joe Gresock Sent: Saturday, May 31, 2014 9:34 AM To: solr-user@lucene.apache.org Subject: Uneven shard heap usage Hi folks, I'm trying to figure out why one

Re: Uneven shard heap usage

2014-05-31 Thread Joe Gresock
the problem? And how > much > > > query or update activity is happening in that time? Is there any other > > > activity showing up in the log? > > > > > > If you bring up only a single node in that problematic shard, do you > > still > > > see

Re: Uneven shard heap usage

2014-05-31 Thread Erick Erickson
only a single node in that problematic shard, do you > still > > see the problem? > > > > -- Jack Krupansky > > > > -Original Message- From: Joe Gresock > > Sent: Saturday, May 31, 2014 9:34 AM > > To: solr-user@lucene.apache.org > > Subject: Uneven shar

Re: Uneven shard heap usage

2014-05-31 Thread Joe Gresock
n the log? > > If you bring up only a single node in that problematic shard, do you still > see the problem? > > -- Jack Krupansky > > -Original Message- From: Joe Gresock > Sent: Saturday, May 31, 2014 9:34 AM > To: solr-user@lucene.apache.org > Subject:

Re: Uneven shard heap usage

2014-05-31 Thread Jack Krupansky
-Original Message- From: Joe Gresock Sent: Saturday, May 31, 2014 9:34 AM To: solr-user@lucene.apache.org Subject: Uneven shard heap usage Hi folks, I'm trying to figure out why one shard of an evenly-distributed 3-shard cluster would suddenly start running out of heap space, after 9+ m

Uneven shard heap usage

2014-05-31 Thread Joe Gresock
Hi folks, I'm trying to figure out why one shard of an evenly-distributed 3-shard cluster would suddenly start running out of heap space, after 9+ months of stable performance. We're using the "!" delimiter in our ids to distribute the documents, and indeed the disk size of our shards are very si