Issue while executing in ssl mode for solrcloud "Error trying to proxy request for url:"
Hi Team,
I am facing an issue while selecting from solrcloud Node , can you let me
know the reason and resolution for the same
NOTE: I have create a custom authentication and authorization plugin , This
works fine when SSL is not enabled. This occurs when ssl is enabled along
with external zookeeper
I am using solr-6.2.1 , and have setup an external zookeeper, I have 2 solr
nodes instances at ports 8983 and 7574( same isse occurs qith single node
too)
When I execute the command-:
curl -E solr-ssl.pem:tcserver --cacert /home/was/root.pem -H
'API-KEY:trial:value' -X POST
'https://slvboxname.company.com:8983/solr/dev2/select?indent=on&q=:&wt=json'
Please find below error-:
Error 500
{metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.http.NoHttpResponseException},msg=Error
trying to proxy request for url:
http://10.211.21.160:8983/solr/dev2/select,trace=org.apache.solr.common.SolrException:
Error trying to proxy request for url:
http://10.901.21.900:8983/solr/dev2/select
at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:588)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:443)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.NoHttpResponseException: 10.901.21.900:8983
failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at
org.apache.http.impl.client.CloseableHttpClient.execute(
Re: CloudSolrClient$RouteException: Cannot talk to ZooKeeper - Updates are disabled.
Returning 500K rows is, as Shawn says, not Solr's sweet spot. My guess: All the work you're doing trying to return that many rows, particularly in SolrCloud mode is simply overloading your system to the point that the ZK connection times out. Don't do that. If you need that many rows, either Shawn's cursorMark option or use export/streaming aggregation are much better choices. Consider what happens on a sharded request: - the initial node sends a sub-request to a replica for each shard. - each replica returns it's candidate topN (doc ID and sort criteria) - the initial node sorts these lists (1M from each replica in your example) to get the true top N - the initial node requests the docs from each replica that made it into the true top N - each replica goes to disk, decompresses the doc and pulls out the fields - each replica sends its portion of the top N to the initial node - an enormous packet containing all 1M final docs is assembled and returned to the client. - this sucks up bandwidth and resources - that's bad enough, but especially if your ZK nodes are on the same box as your Solr nodes they're even more like to have a timeout issue. Best, Erick On Fri, Nov 18, 2016 at 8:45 PM, Shawn Heisey wrote: > On 11/18/2016 6:50 PM, Chetas Joshi wrote: >> The numFound is millions but I was also trying with rows= 1 Million. I will >> reduce it to 500K. >> >> I am sorry. It is state.json. I am using Solr 5.5.0 >> >> One of the things I am not able to understand is why my ingestion job is >> complaining about "Cannot talk to ZooKeeper - Updates are disabled." >> >> I have a spark streaming job that continuously ingests into Solr. My shards >> are always up and running. The moment I start a query on SolrCloud it starts >> running into this exception. However as you said ZK will only update the >> state of the cluster when the shards go down. Then why my job is trying to >> contact ZK when the cluster is up and why is the exception about updating ZK? > > SolrCloud and SolrJ (CloudSolrClient) both maintain constant connections > to all the zookeeper servers they are configured to use. If zookeeper > quorum is lost, SolrCloud will go read-only -- no updating is possible. > That is what is meant by "updates are disabled." > > Solr and Lucene are optimized for very low rowcounts, typically two or > three digits. Asking for hundreds of thousands of rows is problematic. > The cursorMark feature is designed for efficient queries when paging > deeply into results, but it assumes your rows value is relatively small, > and that you will be making many queries to get a large number of > results, each of which will be fast and won't overload the server. > > Since it appears you are having a performance issue, here's a few things > I have written on the topic: > > https://wiki.apache.org/solr/SolrPerformanceProblems > > Thanks, > Shawn >
