We have Solr and ZK running in Docker containers. There is no more then
one Solr/ZK node per host but Solr and ZK node can run on the same host.
So Solr and ZK are spread out separately.
I have not seen this problem during normal processing just when we
recycle nodes or when we have nodes fail, which is pretty much always
caused by being out of memory, which again is unfortunately a bit
complex in Docker. When nodes come up they add quite a few tasks to the
overseer queue. I assume one task for every core. We have about 2000
cores on each node. If nodes come up too fast the queue might grow to a
few thousand entries. At maybe 10000 entries it usually reaches the
point of no return and Solr is just added more tasks then it is able to
process. So it's best to pull the plug at that point as you will not
have to play with jute.maxbuffer to get Solr up again.
We are using Solr 6.3. There is some improvements in 6.6:
https://issues.apache.org/jira/browse/SOLR-10524
https://issues.apache.org/jira/browse/SOLR-10619
On 22.08.2017 14:41, Jeff Courtade wrote:
Thanks very much.
I will followup when we try this.
Im curious in the env this is happening to you.... are the zookeeper
servers residing on solr nodes? Are the solr nodes underpowered ram and or
cpu?
Jeff Courtade
M: 240.507.6116
On Aug 22, 2017 8:30 AM, "Hendrik Haddorp" <hendrik.hadd...@gmx.net> wrote:
I'm always using a small Java program to delete the nodes directly. I
assume you can also delete the whole node but that is nothing I have tried
myself.
On 22.08.2017 14:27, Jeff Courtade wrote:
So ...
Using the zkCli.sh i have the jute.maxbuffer setup so I can list it now.
Can I
rmr /overseer/queue
Or do i need to delete individual entries?
Will
rmr /overseer/queue/*
work?
Jeff Courtade
M: 240.507.6116
On Aug 22, 2017 8:20 AM, "Hendrik Haddorp" <hendrik.hadd...@gmx.net>
wrote:
When Solr is stopped it did not cause a problem so far.
I cleared the queue also a few times while Solr was still running. That
also didn't result in a real problem but some replicas might not come up
again. In those case it helps to either restart the node with the
replicas
that are in state "down" or to remove the failed replica and then
recreate
it. But as said, clearing it when Solr is stopped worked fine so far.
On 22.08.2017 14:03, Jeff Courtade wrote:
How does the cluster react to the overseer q entries disapeering?
Jeff Courtade
M: 240.507.6116
On Aug 22, 2017 8:01 AM, "Hendrik Haddorp" <hendrik.hadd...@gmx.net>
wrote:
Hi Jeff,
we ran into that a few times already. We have lots of collections and
when
nodes get started too fast the overseer queue grows faster then Solr
can
process it. At some point Solr tries to redo things like leaders votes
and
adds new tasks to the list, which then gets longer and longer. Once it
is
too long you can not read out the data anymore but Solr is still adding
tasks. In case you already reached that point you have to start
ZooKeeper
and the ZooKeeper client with and increased "jute.maxbuffer" value. I
usually double it until I can read out the queue again. After that I
delete
all entries in the queue and then start the Solr nodes one by one, like
every 5 minutes.
regards,
Hendrik
On 22.08.2017 13:42, Jeff Courtade wrote:
Hi,
I have an issue with what seems to be a blocked up /overseer/queue
There are 700k + entries.
Solr cloud 6.x
You cannot addreplica or deletereplica the commands time out.
Full stop and start of solr and zookeeper does not clear it.
Is it safe to use the zookeeper supplied zkCli.sh to simple rmr the
/overseer/queue ?
Jeff Courtade
M: 240.507.6116
