Hi Erick, Thanks for the response.
1. We have started from zero documents and are observing socketTimeout intermittently while indexing also. 2. We are not using schemaless as we are aware of the drawbacks and have a well defined schema. 3. We have tried async but it does not help. 4. We tried increasing timeout as mentioned in https://issues.apache.org/jira/browse/SOLR-13605 by using withConnectionTimeout() and withSocketTimeout() 5. Our System was pretty stable earlier but as we upgraded to 8.2 from 6.6 version this issue is making Solrcloud system unstable here is the detailed log which we are observing intermittently at time of indexing: ERROR (OverseerThreadFactory-9-thread-3-processing-n:prod-cloud2.net:8983_solr) [ ] o.a.s.c.a.c.OverseerCollectionMessageHandler Error from shard: http://prod-cloud2.net:8983/solr => org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: http://prodcloud2.net:8983/solr at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:667) org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: http://prod-cloud2.net:8983/solr at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:667) ~[?:?] at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262) ~[?:?] at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245) ~[?:?] at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1274) ~[?:?] at org.apache.solr.handler.component.HttpShardHandlerFactory$1.request(HttpShardHandlerFactory.java:176) ~[?:?] at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:199) ~[?:?] at org.apache.solr.handler.component.HttpShardHandler.dt_access$927(HttpShardHandler.java) ~[?:?] at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_131] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:1.8.0_131] at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:1.8.0_131] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) ~[metrics-core-4.0.5.jar:4.0.5] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[?:?] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.dt_access$304(ExecutorUtil.java) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:1.8.0_131] at java.lang.Thread.run(Unknown Source) [?:1.8.0_131] Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_131] at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:1.8.0_131] at java.net.SocketInputStream.read(Unknown Source) ~[?:1.8.0_131] at java.net.SocketInputStream.read(Unknown Source) ~[?:1.8.0_131] at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137) ~[?:?] at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153) ~[?:?] at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282) ~[?:?] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138) ~[?:?] at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56) ~[?:?] at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259) ~[?:?] at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163) ~[?:?] at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165) ~[?:?] at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273) ~[?:?] at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125) ~[?:?] at org.apache.solr.util.stats.InstrumentedHttpRequestExecutor.execute(InstrumentedHttpRequestExecutor.java:120) ~[?:?] at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[?:?] at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[?:?] at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[?:?] at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?] at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[?:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[?:?] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[?:?] at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:555) ~[?:?] ... 15 more Thanks for the help, any further suggestions to drill down to problem? Regards, kshitij On Wed, Feb 19, 2020 at 6:18 PM Erick Erickson <erickerick...@gmail.com> wrote: > Yogesh: > > Please do not hijack threads. The original poster requested information > about > socket timeouts. True “upgrade” was mentioned, but it was a completely > different issue. > > Kshitij: > > There’s not much information to go on here. It’s possible you were running > close to the timeout limit before and “something” changed just enough > to go over that limit. > > I’m a bit confused though, you talk about commands like reload while > indexing. > What _exactly_ are you trying to do? Details matter. > > One thing that did changes was “schemaless” became the default. This > causes reloads when Solr is indexing docs and comes across fields > for the first time. I personally don’t recommend “schemaless”, so what > happens if you turn that off? > > If you’re manually sending reloads, you might try doing them async. > > That said, you must start from a fresh index with _no_ documents in it > when you upgrade more than one major version. Did you start over? > > Best, > Erick > > > On Feb 19, 2020, at 3:58 AM, kshitij tyagi <kshitij.s...@gmail.com> > wrote: > > > > Hi, > > > > Any information on socket timeout issue when using collection apis? I am > > observing increased response time when using Collection APis in upgraded > > version > > > > On Wed, Feb 19, 2020 at 2:22 PM Jörn Franke <jornfra...@gmail.com> > wrote: > > > >> Yes you need to reindex. > >> Update solrconfig, schemas to leverage the later feature of the version > >> (some datatypes are now more optimal others are deprecated. > >> > >> Update Solrconfig.xml and schema to leverage the latest datatypes , > >> features etc.. > >> > >> Create new collection based on newest config. > >> Use your regular Index process to move documents to new collection. > >> > >> Check if new collection works and has expected performance. > >> > >> Delete old collection. > >> > >> Test before in a test environment and not in production! > >> > >>> Am 19.02.2020 um 09:46 schrieb Yogesh Chaudhari > >> <yogesh.chaudh...@harbingergroup.com.invalid>: > >>> > >>> Hi, > >>> > >>> Could you please share me the steps to upgrade SOlr? > >>> > >>> Now I am using Solr cloud 5.2.1 on production and wanted to upgrade to > >> SOlr7.7.2. I am doing this in 2 spteps SOlr 5.2.1 to SOlr 6.6.6 then > SOlr > >> 7.7.2. > >>> > >>> I have upgraded to Solr but getting issue for indexing of old > >> documents. I am badly stuck get get old document in migrated solr > version. > >>> > >>> Should I do the re-indexing? If yes can you please share the way to > >> re-indexing? > >>> > >>> Can you please provide your inputs on this? > >>> > >>> Thanks, > >>> > >>> Yogesh Chaudhari > >>> > >>> -----Original Message----- > >>> From: kshitij tyagi <kshitij.s...@gmail.com> > >>> Sent: Wednesday, February 19, 2020 12:52 PM > >>> To: solr-user@lucene.apache.org > >>> Subject: Solr Upgrade socketTimeout issue in 8.2 > >>> > >>> Hi, > >>> > >>> We have upgraded our solrCloud from version 6.6.0 to 8.2.0 > >>> > >>> At the time of indexing intermittently we are observing socketTimeout > >> exception when using Collection apis. example when we try reloading one > of > >> the collection using CloudSolrClient class. > >>> > >>> Is there any performance degradation in Solrcloud collection apis? > >>> > >>> logs: > >>> > >>> IOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught > >> end of stream exception > >>> > >>> EndOfStreamException: Unable to read additional data from client > >> sessionid 0x2663e756d775747, likely client has closed socket > >>> > >>> at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) > >>> > >>> at > >>> > >> > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203) > >>> > >>> at java.lang.Thread.run(Unknown Source) > >>> > >>> > >>> logs: > >>> > >>> > >>> Exception has occured in job switch: Timeout occurred while waiting > >> response from server at: > >> > http://secure-web.cisco.com/1w_RA10DqbruLQVC6aKUXuMgV4hC3T14viv2m2iUptQ2hyGjYLn0sSSy0Q_XNqcVxHym-e_mOyPc_AYM4zpIWWXdyRCpvXzL3mSeFK-DzhL_CqoNi2FwQUvhk2zb8OQKs1e11yBHqblc3Kyx0XlruLvb24BUj0lBBGmJVf5E9rrTaFQbFmCdNyccx1KCIpzf2MlyeqvvXVWKCW_YbqnLWGjcfqlAylbNqJTGuKf5rbBMdJ8pn14dbFlM0QDZjn6IORWVA8NqmdhC9VwD1rzpU6dVIpsph6qz_OcgoH61wlZALQ1Zj65XRFtXvuhqEWQeaabvKactprjz1o3pflKaxttbgxz1ItRxb4FjZkBgTC24uwalAmi_CyfeP7DECtIYATYf3AJFjCUfLV8_Rj2V5J0JeCTFDi7CWqKFUhiHXtpM8PvZt8kgMIRwfgPUKHPIJ/http%3A%2F%2Fprod-t-8.net%3A8983%2Fsolr > >>> > >>> > >>> Is anyone facing same type of issue? in Solrcloud? Any suggestions to > >> solve?? > >>> > >>> > >>> > >>> Regards, > >>> > >>> kshitij > >> > >
