Re: solr hangs
Hi Eric,
I am using the restful api directly. In our application, system issues the http
request directly to Solr.
${solr.autoCommit.maxTime:15000}
1
true
Thanks
Hawk
> On 6 Aug 2017, at 11:10 AM, Erick Erickson wrote:
>
> How are you updating 50K docs? SolrJ? If so are you using
> CloudSolrClient? What are your commit settings? Details matter.
>
> Best,
> Erick
>
> On Sat, Aug 5, 2017 at 6:19 PM, hawk wrote:
>> Hi All,
>>
>> I have encountered one problem of Solr. In our environment, we setup 2 Solr
>> nodes, every hour we will update request to Solr to update the documents,
>> the total documents are around 50k. From time to time, the Solr hangs and
>> the client encounters the timeout issue.
>>
>> below is the exception in Solr log.
>>
>> 2017-08-06 07:28:03.682 ERROR (qtp401424608-31250) [c:taoke s:shard2
>> r:core_node2 x:taoke_shard2_replica2] o.a.s.s.HttpSolrCall
>> null:java.io.IOException: java.util.concurrent.TimeoutException: Idle
>> timeout expired: 5/5 ms
>>at
>> org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219)
>>at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:220)
>>at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:583)
>>at
>> org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:55)
>>
>>
>>
>> Thanks
>
Re: solr hangs
We found the problem is caused by the delete command. The request is used to
delete document by id.
url --> http://10.91.1.120:8900/solr/taoke/update?&commit=true&wt=json
body --> {"delete":["20ec36ade0ca4da3bcd78269e2300f6f"]}
When we send over 3000 requests, the Solr starts to give OOM exceptions.
Now have changed the logic to put all ids in the array, it seems Solr works
without any exception.
Not sure whether Solr internally optimized the delete.
Thanks
Hawk
> On 7 Aug 2017, at 9:20 AM, Erick Erickson wrote:
>
> You have several possibilities here:
> 1> you're hitting a massive GC pause that's timing out. You can turn
> on GC logging and analyze if that's the case.
> 2> your updates are getting backed up. At some point it's possible
> that the index writer blocks until merges are done IIUC.
>
> Does this ever happen if you throttle your updates? Does it go away if
> you batch your documents in batches of, say, 1,000
>
> Best,
> Erick
>
> On Sun, Aug 6, 2017 at 5:19 PM, hawk@139.com wrote:
>> Hi Eric,
>>
>> I am using the restful api directly. In our application, system issues the
>> http request directly to Solr.
>>
>>
>> ${solr.autoCommit.maxTime:15000}
>> 1
>> true
>>
>>
>>
>> Thanks
>> Hawk
>>
>>
>>
>>> On 6 Aug 2017, at 11:10 AM, Erick Erickson wrote:
>>>
>>> How are you updating 50K docs? SolrJ? If so are you using
>>> CloudSolrClient? What are your commit settings? Details matter.
>>>
>>> Best,
>>> Erick
>>>
>>> On Sat, Aug 5, 2017 at 6:19 PM, hawk wrote:
>>>> Hi All,
>>>>
>>>> I have encountered one problem of Solr. In our environment, we setup 2
>>>> Solr nodes, every hour we will update request to Solr to update the
>>>> documents, the total documents are around 50k. From time to time, the Solr
>>>> hangs and the client encounters the timeout issue.
>>>>
>>>> below is the exception in Solr log.
>>>>
>>>> 2017-08-06 07:28:03.682 ERROR (qtp401424608-31250) [c:taoke s:shard2
>>>> r:core_node2 x:taoke_shard2_replica2] o.a.s.s.HttpSolrCall
>>>> null:java.io.IOException: java.util.concurrent.TimeoutException: Idle
>>>> timeout expired: 5/5 ms
>>>> at
>>>> org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:219)
>>>> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:220)
>>>> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:583)
>>>> at
>>>> org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:55)
>>>>
>>>>
>>>>
>>>> Thanks
>>>
>>
>
Re: solr hangs
Below is the OOM exception.
2017-08-07 12:45:48.446 WARN (qtp33524623-4275) [c:taoke s:shard2 r:core_node4
x:taoke_shard2_replica1] o.e.j.u.t.QueuedThreadPool
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.startThreads(QueuedThreadPool.java:475)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.access$200(QueuedThreadPool.java:48)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:579)
at java.lang.Thread.run(Thread.java:745)
The timeout exception is thrown as well.
2017-08-07 13:20:20.336 ERROR (qtp33524623-2666) [c:taoke s:shard2 r:core_node4
x:taoke_shard2_replica1] o.a.s.h.RequestHandlerBase
org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Read timed out
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:973)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1912)
at
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:182)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:78)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2440)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:298)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
Thanks
Hawk
> On 7 Aug 2017, at 2:21 PM, Shawn Heisey wrote:
>
> On 8/6/2017 10:29 PM, hawk....@139.com wrote:
>> We found the problem is caused by the delete command. The request is used to
>> delete document by id.
>>
>> url --> http://10.91.1.120:8900/solr/taoke/update?&commit=true&wt=json
>>body --> {"delete":["20ec36ade0ca4da3bcd78269e2300f6f"]}
>>
>> When we send over 3000 requests, the Solr starts to give OOM exceptions.
>
> Do you have the full exc
Re: solr hangs
Hi Shawn, The software config of the server is: OS: CentOS 7 Solr Version: 6.5 The system sends delete requests to the Solr server, the requests are around 5k to 6k.The exception occurred from time to time, and we have to restart the node to continue the service. The max threads setting os the server is: max user processes (-u) 65535 Thanks Hawk > On 7 Aug 2017, at 10:01 PM, Shawn Heisey wrote: > > On 8/7/2017 12:35 AM, hawk....@139.com wrote: >> Below is the OOM exception. >> >> 2017-08-07 12:45:48.446 WARN (qtp33524623-4275) [c:taoke s:shard2 >> r:core_node4 x:taoke_shard2_replica1] o.e.j.u.t.QueuedThreadPool >> java.lang.OutOfMemoryError: unable to create new native thread > > The last line that I quoted above is the underlying problem here. > > Your OS has a limit on the number of processes/threads that each user is > allowed to run. Solr has exceeded that limit, and so the operating > system is preventing Solr from starting any more threads. The OS will > need to be reconfigured to allow the user to start more threads. > > Just like OOME problems that actually have to do with memory allocation, > program operation when a thread fails to start is *also* completely > unpredictable. > > What version of Solr are you running, what OS is it running on, and > exactly how did you start it? > > Thanks, > Shawn > >
