Hi,
Sorry for the initial hurried up mail, here is some correction and further
explanation:
Problem I described previously was happening before we set zkClientTimeout
value so it was 30000 when it happened.
autoCommit maxTime value is 15000 and autoSoftCommit maxTime is 60000.
We recently removed maxDocs values from autoCommit settings and it seems
more stable so far and has better response time.
I can’t seem to find these values on Solr logs probably because logging
level is currently WARN but we left those as default so I think they’re set
as the values in solr.xml
<int name="distribUpdateSoTimeout">${distribUpdateSoTimeout:600000}</int>
<int name="distribUpdateConnTimeout">${distribUpdateConnTimeout:60000}</int>
We have 12 replicas using default routing. All commits and queries are
going to a single node because of the dummy client we use. Documents are
send in JSON format. I don’t have exact knowledge of document size, they
are mostly news article sized, though with lots of dynamic fields.
Sematext SPM currently shows “Added Docs Rate” as ~1.70k/sec for the server
that is receiving updates.
Once problem starts happening multiple replicas go down (not necessarily
the one receiving the update request from client) and cluster starts
returning errors to update requests.
We saw entries like following in Zookeeper logs that’s why we thought it
might be related to zkClientTimeout and value.
2017-04-03 09:13:03,040 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
client /192.168.30.32:36420 which had sessionid 0x25ad61c4507008c
2017-04-03 09:27:02,078 [myid:1] - WARN [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
0x15b14ba8a8e0026, likely client has closed socket
at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
2017-04-03 09:27:02,079 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for
client /192.168.30.32:35636 which had sessionid 0x15b14ba8a8e0026
2017-04-03 09:35:19,362 [myid:1] - INFO [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection
from /192.168.30.32:37970
*Salih Şen*
M: +90 533 131 17 07
E: sa...@dilisim.com
W: www.dilisim.com
Skype: slhsen
On 3 April 2017 at 18:01:15, Erick Erickson (erickerick...@gmail.com) wrote:
bq: We set Auto hardcommit time to 15sec and 10000 docs, and soft
commit to 60000 sec and 5000 seconds
Just a sanity check, the commit intervals are in milliseconds, your
units look mixed up above, I'm guessing it's just a typo though. I
usually don't use maxDocs because it's unpredictable. Say you're
indexing at a furious rate. If you are indexing at 5,000 docs a second
(and assuming the above was supposed to be soft committing every 60
seconds or 5,000 docs) you'll still be autocommitting every second.
While that could be related, it's not particularly germane to your
timeout. My guess is that you're getting these errors on the leader?
what do you have in solr.xml for:
distribUpdateConnTimeout and distribUpdateSoTimeout
Those are likely the timeouts that matter. And how big are your
documents? The scenario I'm thinking of is that the leader sends the
update to the replica and the timeout for the replica's response
exceeds the ones above.
BTW, it can be useful on startup to look at your solr.log. The
_actual_ values for all the timeouts are printed out, including any
sysvars you've used.
And how are you indexing? Mostly I'm wondering how fast you're sending
docs to each leader and how.
Best,
Erick
On Mon, Apr 3, 2017 at 6:52 AM, Salih Sen <sa...@dilisim.com> wrote:
> Hi,
>
> We have a three server set up with each server having 756G ram, 48 cores,
> 4SSDs (each having tree solr instances on them) and a dedicated
mechanical
> disk for zookeeper (3 zk instances total). Each Solr instances have 31G
of
> heap space allocated to them. In total we have 36 Solr Instances and 3
> Zookeeper instances (with 1G heapspace). Also servers 10Gig network
between
> them.
>
> We set Auto hardcommit time to 15sec and 10000 docs, and soft commit to
> 60000 sec and 5000 seconds in order to avoid soft committing too much and
> avoiding indexing bottlenecks. We also set DzkClientTimeout=90000.
>
> But it seems replicas still randomly go down while indexing. Do you have
any
> suggestions to prevent this situation?
>
> ERROR - 2017-04-03 12:24:02.503; [ ]
> org.apache.solr.cloud.OverseerCollectionMessageHandler; Error from shard:
> http://192.168.30.33:9132/solr
> org.apache.solr.client.solrj.SolrServerException: Timeout occured while
> waiting response from server at: http://192.168.30.33:9132/solr
> at
>
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:621)
> at
>
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
> at
>
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
> at
>
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:163)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
>
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> at
>
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
>
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
>
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> at
>
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
>
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> at
>
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
>
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
>
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> at
>
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
>
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
> ... 12 more
> ERROR - 2017-04-03 12:27:11.631; [c:doc s:shard3 r:core_node22
> x:doc_shard3_replica3]
>
org.apache.solr.update.StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient;
> error
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> at
>
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
>
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
>
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> at
>
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
>
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> at
>
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
>
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
>
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> at
>
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
>
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
> at
>
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
> at
>
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
>
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ERROR - 2017-04-03 12:27:11.633; [c:doc s:shard3 r:core_node22
> x:doc_shard3_replica3]
>
org.apache.solr.update.StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient;
> error
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> at
>
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
>
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
>
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> at
>
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
>
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> at
>
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
>
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
>
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> at
>
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
>
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
> at
>
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
> at
>
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
>
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> ERROR - 2017-04-03 12:27:11.645; [c:doc s:shard3 r:core_node22
> x:doc_shard3_replica3]
>
org.apache.solr.update.StreamingSolrClients$ErrorReportingConcurrentUpdateSolrClient;
> error
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> at
>
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
>
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
>
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at
>
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
>
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> at
>
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
>
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> at
>
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
>
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> at
>
org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:114)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
>
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> at
>
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at
>
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at
>
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:311)
> at
>
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:184)
> at
>
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
> at
>
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at
>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> W
>
>
>
>
> Salih Şen
> M: +90 533 131 17 07
> E: sa...@dilisim.com
> W: www.dilisim.com
> Skype: slhsen
