Thank you Joel. I gave each node in the cluster 24g of heap and it ran,
but then failed on the 50th iteration (was trying to do 1,000).
This time, I have the error on the node and the exception from the
client running the stream command. The node (Doris) has 3 errors that
occurred at the same time logged:
java.io.IOException: java.io.IOException:
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.read(TextLogitStream.java:498)
at
org.apache.solr.client.solrj.io.stream.PushBackStream.read(PushBackStream.java:87)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:109)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: java.lang.IndexOutOfBoundsException:
Index: 0, Size: 0
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.getShardUrls(TextLogitStream.java:365)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.read(TextLogitStream.java:457)
... 47 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at
org.apache.solr.client.solrj.io.stream.TextLogitStream.getShardUrls(TextLogitStream.java:357)
Then:
CloudSolrClient
Request to collection models failed due to (0) java.net.SocketException:
Socket closed,​ retry? 0
Then a warning:
org.apache.solr.client.solrj.SolrServerException:
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:976)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:922)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.uploadBatchToCollection(UpdateStream.java:299)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:124)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:173)
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:516)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
... 52 more
And finally:
CloudSolrClient
Request to collection models failed due to (0)
java.util.concurrent.RejectedExecutionException: Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,​
pool size = 0,​ active threads = 0,​ queued tasks =
0,​ completed tasks = 3],​ retry? 1
java.io.IOException: Unexpected error when adding documents to
collection models-
org.apache.solr.client.solrj.SolrServerException:java.util.concurrent.RejectedExecutionException:
Task
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$15/1356635348@74759d4e
rejected from
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1498be29[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at
org.apache.solr.client.solrj.io.stream.UpdateStream.uploadBatchToCollection(UpdateStream.java:304)
at
org.apache.solr.client.solrj.io.stream.UpdateStream.read(UpdateStream.java:124)
at
org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:68)
at
org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:627)
at
org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:87)
at
org.apache.solr.response.JSONWriter.writeIterator(JSONResponseWriter.java:523)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:180)
at
org.apache.solr.response.JSONWriter$2.put(JSONResponseWriter.java:559)
at
org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:84)
at
org.apache.solr.response.JSONWriter.writeMap(JSONResponseWriter.java:547)
at
org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:198)
at
org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:209)
at
org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:325)
at
org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:120)
at
org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:71)
at
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
at
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:806)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:535)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
The client has:
java.io.IOException: java.util.concurrent.ExecutionException:
java.io.IOException: params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522969708719,features(MODEL1033_1522969708719,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522969708719",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:405)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:275)
at
com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcessor.doWork(SolrModelBuilderProcessor.java:122)
at
com.ngc.intelenterprise.intelentutil.utils.Processor.run(Processor.java:140)
at
com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.process(IntelEntQueueProc.java:208)
at
org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:141)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:460)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:114)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:637)
at
org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:308)
at
org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:246)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1144)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1136)
at
org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1033)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522969708719,features(MODEL1033_1522969708719,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522969708719",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:399)
... 27 more
Caused by: java.io.IOException: params
expr=update(models,+batchSize%3D"50",train(MODEL1033_1522969708719,features(MODEL1033_1522969708719,q%3D"*:*",featureSet%3D"FSet_MODEL1033_1522969708719",field%3D"Text",outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at
org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:115)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:510)
at
org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
... 3 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://doris:9100/solr/MODEL1033_1522969708719_shard30_replica_n113
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:637)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at
org.apache.solr.client.solrj.io.stream.SolrStream.constructParser(SolrStream.java:269)
at
org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrStream.java:113)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
at
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at
org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:525)
... 12 more
The cluster was basically idle during this run, but that certainly
doesn't mean there was not a legitimate read timeout. Thanks for looking!
-Joe
On 4/5/2018 8:27 PM, Joel Bernstein wrote:
Hi Joe,
Currently you will eventually run into memory problems if the training sets
gets too large. Under the covers on each node it is creating a matrix with
a row for each document and a column for each feature. This can get large
quite quickly. By choosing fewer features you can make this matrix much
smaller.
Its fairly easy to make the train function work on a random sample of the
training set on each iteration rather then the entire training set, but
currently this is not how its implemented. Feel free to create a ticket
requesting this the sampling approach.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Apr 5, 2018 at 5:32 PM, Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
I tried to build a large model based on about 1.2 million documents. One
of the nodes ran out of memory and killed itself. Is this much data not
reasonable to use? The nodes have 16g of heap. Happy to increase it, but
not sure if this is possible?
Thank you!
-Joe
On 4/5/2018 10:24 AM, Joe Obernberger wrote:
Thank you Shawn - sorry so long to respond, been playing around with this
a good bit. It is an amazing capability. It looks like it could be
related to certain nodes in the cluster not responding quickly enough. In
one case, I got the concurrent.ExecutionException, but it looks like the
root cause was a SocketTimeoutException. I'm using HDFS for the index and
it gets hit pretty hard by other processes running, and I'm wondering if
that's causing this.
java.io.IOException: java.util.concurrent.ExecutionException:
java.io.IOException: params expr=update(models,+batchSize%
3D"50",train(MODEL1033_1522883727011,features(MODEL1033_
1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_
1522883727011",field%3D"Text",outcome%3D"out_i",
positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL10
33",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000")
)&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openS
treams(CloudSolrStream.java:405)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(
CloudSolrStream.java:275)
at com.ngc.bigdata.ie_solrmodelbuilder.SolrModelBuilderProcesso
r.doWork(SolrModelBuilderProcessor.java:114)
at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
Processor.java:140)
at com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.
process(IntelEntQueueProc.java:208)
at org.apache.camel.processor.DelegateSyncProcessor.process(Del
egateSyncProcessor.java:63)
at org.apache.camel.management.InstrumentationProcessor.process
(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
deliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(Ca
melInternalProcessor.java:190)
at org.apache.camel.processor.CamelInternalProcessor.process(Ca
melInternalProcessor.java:190)
at org.apache.camel.component.direct.DirectProducer.process(Dir
ectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcess
or.java:141)
at org.apache.camel.management.InstrumentationProcessor.process
(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
deliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(Ca
melInternalProcessor.java:190)
at org.apache.camel.processor.CamelInternalProcessor.process(Ca
melInternalProcessor.java:190)
at org.apache.camel.component.jms.EndpointMessageListener.onMes
sage(EndpointMessageListener.java:114)
at org.springframework.jms.listener.AbstractMessageListenerCont
ainer.doInvokeListener(AbstractMessageListenerContainer.java:699)
at org.springframework.jms.listener.AbstractMessageListenerCont
ainer.invokeListener(AbstractMessageListenerContainer.java:637)
at org.springframework.jms.listener.AbstractMessageListenerCont
ainer.doExecuteListener(AbstractMessageListenerContainer.java:605)
at org.springframework.jms.listener.AbstractPollingMessageListe
nerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.
java:308)
at org.springframework.jms.listener.AbstractPollingMessageListe
nerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.
java:246)
at org.springframework.jms.listener.DefaultMessageListenerConta
iner$AsyncMessageListenerInvoker.invokeListener(DefaultMessageLis
tenerContainer.java:1144)
at org.springframework.jms.listener.DefaultMessageListenerConta
iner$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessag
eListenerContainer.java:1136)
at org.springframework.jms.listener.DefaultMessageListenerConta
iner$AsyncMessageListenerInvoker.run(DefaultMessageListenerContai
ner.java:1033)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
Executor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
lExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
params expr=update(models,+batchSize%3D"50",train(MODEL1033_1522883
727011,features(MODEL1033_1522883727011,q%3D"*:*",
featureSet%3D"FSet_MODEL1033_1522883727011",field%3D"Text",
outcome%3D"out_i",positiveLabel%3D1,numTerms%3D1
000),q%3D"*:*",name%3D"MODEL1033",field%3D"Text",outcome%3D"
out_i",maxIterations%3D"1000"))&qt=/stream&explain=true&q=*:
*&fl=id&sort=id+asc&distrib=false
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openS
treams(CloudSolrStream.java:399)
... 27 more
Caused by: java.io.IOException: params expr=update(models,+batchSize%
3D"50",train(MODEL1033_1522883727011,features(MODEL1033_
1522883727011,q%3D"*:*",featureSet%3D"FSet_MODEL1033_
1522883727011",field%3D"Text",outcome%3D"out_i",
positiveLabel%3D1,numTerms%3D1000),q%3D"*:*",name%3D"MODEL10
33",field%3D"Text",outcome%3D"out_i",maxIterations%3D"1000")
)&qt=/stream&explain=true&q=*:*&fl=id&sort=id+asc&distrib=false
at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrS
tream.java:115)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream$Strea
mOpener.call(CloudSolrStream.java:510)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream$Strea
mOpener.call(CloudSolrStream.java:499)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
xecutor.lambda$execute$0(ExecutorUtil.java:188)
... 3 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://leda:9100/solr/MODEL1033_1522883727011_shard20_replica_n74
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
od(HttpSolrClient.java:637)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
pSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
pSolrClient.java:242)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.j
ava:1219)
at org.apache.solr.client.solrj.io.stream.SolrStream.constructP
arser(SolrStream.java:269)
at org.apache.solr.client.solrj.io.stream.SolrStream.open(SolrS
tream.java:113)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java
:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(Se
ssionInputBufferImpl.java:139)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(Se
ssionInputBufferImpl.java:155)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(Sess
ionInputBufferImpl.java:284)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
d(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHea
d(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(Abstract
MessageParser.java:261)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveRes
ponseHeader(DefaultBHttpClientConnection.java:165)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(C
PoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveRespon
se(HttpRequestExecutor.java:272)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpReq
uestExecutor.java:124)
at org.apache.http.impl.execchain.MainClientExec.execute(
MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(
ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.
java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(
RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
ernalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
eableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
eableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
od(HttpSolrClient.java:525)
-Joe
On 4/2/2018 7:09 PM, Shawn Heisey wrote:
On 4/2/2018 1:55 PM, Joe Obernberger wrote:
The training data was split across 20 shards - specifically created
with:
http://icarus.querymasters.com:9100/solr/admin/collections?
action=CREATE&name=MODEL1024_1522696624083&numShards=20&rep
licationFactor=2&maxShardsPerNode=5&collection.configName=TRAINING
Any ideas? The complete error is:
<snip>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing
/solr/MODEL1024_1522696624083_shard20_replica_n75/select. Reason:
<pre> Not Found</pre></p>
</body>
I'll warn you in advance that I know nothing at all about the learning
to rank functionality. I'm replying about the underlying error you're
getting, independent of what your query is trying to accomplish.
It's a 404 error, trying to access the URL mentioned above.
The error doesn't indicate exactly WHAT wasn't found. It could either
be the core named "MODEL1024_1522696624083_shard20_replica_n75" or the
"/select" handler on that core. That's something you need to figure
out. It could be that the core *does* exist, but for some reason, Solr
on that machine was unable to start it.
The solr.log file on the Solr instance that returned the error (which
seems to be on the machine named vesta, answering to port 9100) may have
more detail for the error, or some additional error messages.
Normally SolrCloud is good at making sure that requests aren't sent to
resources that aren't working. So I'm not sure why this happened.
Are there other errors or warnings in the solr.log file, either on the
instance where you sent your request, or the instance that returned the
404 error?
Thanks,
Shawn
---
This email has been checked for viruses by AVG.
http://www.avg.com
