Also note there were no exceptions in the actual Solr log, only on the
SolrJ side.
Thanks,
Briggs
On Mon, Oct 8, 2012 at 10:45 AM, Briggs Thompson <
w.briggs.thomp...@gmail.com> wrote:
> I have also just ran into this a few times over the weekend in a newly
> deployed system. We are running Solr 4.0 Beta (not using SolrCloud) and it
> is hosted via AWS.
>
> I have a RabbitMQ consumer that reads updates from a queue and posts
> updates to Solr via SolrJ. There is quite a bit of error handling around
> the indexing request, and even if Solr is not live the consumer application
> successfully logs the exception and attempts to move along in the queue.
> There are two consumer applications running at once, and at times processes
> 400 requests per minute. The high volume times is not necessarily when this
> problem occurs, though.
>
> This exception is causing the entire application to hang - which is
> surprising considering all SolrJ logic is wrapped with try/catches. Has
> anyone found out more information regarding the possible keep alive bug?
> Any insight is much appreciated.
>
> Thanks,
> Briggs Thompson
>
>
> Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
> tryExecute
> INFO: I/O exception (java.net.SocketException) caught when processing
> request: Broken pipe
> Oct 8, 2012 7:25:48 AM org.apache.http.impl.client.DefaultRequestDirector
> tryExecute
> INFO: Retrying request
> Oct 8, 2012 7:25:48 AM com.<>.rabbitmq.worker.SolrWriter work
> SEVERE: {"id":4049703,"datetime":"2012-10-08 07:22:05"}
> IOException occured when talking to server at:
> <<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
> server>
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at:
> <<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
> server>
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:362)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
> at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69)
> at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:96)
> at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:79)
> at com.<>.solr.SolrIndexService.IndexCoupon(SolrIndexService.java:57)
> at com.<>.solr.SolrIndexService.Index(SolrIndexService.java:36)
> at com.<>.rabbitmq.worker.SolrWriter.work(SolrWriter.java:47)
> at com.<>.rabbitmq.job.Runner.run(Runner.java:84)
> at com.<>.rabbitmq.job.SolrConsumer.main(SolrConsumer.java:10)
> Caused by: org.apache.http.client.ClientProtocolException
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:909)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:306)
> ... 10 more
> Caused by: org.apache.http.client.NonRepeatableRequestException: Cannot
> retry request with a non-repeatable request entity. The cause lists the
> reason the original request failed.
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:686)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517)
> at
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> ... 13 more
> Caused by: java.net.SocketException: Broken pipe
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at
> org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:147)
> at
> org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:154)
> at
> org.apache.http.impl.conn.LoggingSessionOutputBuffer.flush(LoggingSessionOutputBuffer.java:95)
> at
> org.apache.http.impl.io.ChunkedOutputStream.flush(ChunkedOutputStream.java:178)
> at
> org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:72)
> at
> org.apache.http.entity.mime.HttpMultipart.doWriteTo(HttpMultipart.java:206)
> at
> org.apache.http.entity.mime.HttpMultipart.writeTo(HttpMultipart.java:224)
> at
> org.apache.http.entity.mime.MultipartEntity.writeTo(MultipartEntity.java:183)
> at
> org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
> at
> org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
> at
> org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
> at
> org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
> at
> org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227)
> at
> org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
> at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712)
> ... 15 more
> Oct 8, 2012 7:27:22 AM com.<>.rabbitmq.worker.SolrWriter work
> SEVERE: {"id":4049704,"datetime":"2012-10-08 07:22:05"}
> Server at
> <<http://ec2-50-18-73-42.us-west-1.compute.amazonaws.com:8983/solr/coupon>
> server> returned non ok status:500, message:Internal Server Error
> org.apache.solr.client.solrj.SolrServerException: Server at <server> returned
> non ok status:500, message:Internal Server Error
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:328)
> at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
> at
> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:69)
> at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:96)
> at org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:79)
> at com.<>.solr.SolrIndexService.IndexCoupon(SolrIndexService.java:57)
> at com.<>.solr.SolrIndexService.Index(SolrIndexService.java:36)
> at com.<>.rabbitmq.worker.SolrWriter.work(SolrWriter.java:47)
> at com.<>.rabbitmq.job.Runner.run(Runner.java:84)
> at com.<>.rabbitmq.job.SolrConsumer.main(SolrConsumer.java:10)
>
>
> On Fri, Oct 5, 2012 at 4:20 PM, Erick Erickson <erickerick...@gmail.com>wrote:
>
>> Well, using embedded Solr isn't necessarily indicated. I have a couple
>> of questions.
>> 1> you say 30 tps. Are you sending a single doc at a time or batching them
>> up? I.e. server.add(doclist) or server.add(doc)?
>>
>> 2> Http isn't actually an inefficient protocol, I think the whole idea of
>> using embedded Solr is a red herring if that's the concern.
>>
>> 3> bq: " call the VIP in front of the SOLR nodes to distribute the update
>> load"
>> What does that mean? Only one process should be writing to an index
>> at a time. So there should be only one address to which all the
>> updates go. The servlet container will stack up multiple incoming
>> requests.
>> So I totally don't understand what you're trying to accomplish here.
>>
>> 4> You haven't told us what version of Solr you're using, so the stack
>> trace
>> isn't all that helpful. We can pick random Solr versions and hope they
>> match
>> the line numbers, but that's not too efficient.....
>>
>> 5> Have you thought about just putting some fallback logic in your SolrJ
>> program
>> and having it sleep for a few seconds and retrying?
>>
>> Best
>> Erick
>>
>> On Fri, Oct 5, 2012 at 4:35 PM, Sushil jain <jain.ayushm...@gmail.com>
>> wrote:
>> > If you need to use solr in an embedded application, this is the
>> recommended
>> > approach. It allows you to work with the same interface whether or not
>> you
>> > have access to HTTP.
>> >
>> > And it is not thread safe.
>> >
>> > On Sat, Oct 6, 2012 at 1:58 AM, balaji.gandhi
>> > <balaji.gan...@apollogrp.edu>wrote:
>> >
>> >> Sushil, we are trying to call the VIP in front of the SOLR nodes to
>> >> distribute the update load.
>> >>
>> >> Also is EmbeddedSolrServer thread safe?
>> >>
>> >> Balaji Gandhi, Senior Software Developer, Horizontal Platform Services
>> >> Product Engineering │ Apollo Group, Inc.
>> >> 1225 W. Washington St. | AZ23 | Tempe, AZ 85281
>> >> Phone: 602.713.2417 | Email: balaji.gan...@apollogrp.edu<mailto:
>> >> balaji.gan...@apollogrp.edu>
>> >>
>> >> P Go Green. Don't Print. Moreover soft copies can be indexed by
>> algorithms.
>> >>
>> >> From: Sushil jain [via Lucene] [mailto:
>> >> ml-node+s472066n4012179...@n3.nabble.com]
>> >> Sent: Friday, October 05, 2012 1:23 PM
>> >> To: Balaji Gandhi
>> >> Subject: Re: SolrJ - IOException
>> >>
>> >> Yes, I'd recommend EmbeddedSolrServer, because it doesn't require any
>> web
>> >> server for read/write/update/delete operations.
>> >>
>> >> On Sat, Oct 6, 2012 at 1:48 AM, balaji.gandhi
>> >> <[hidden email]</user/SendEmail.jtp?type=node&node=4012179&i=0>>wrote:
>> >>
>> >> > Sushil,
>> >> >
>> >> > 30 TPS = 30 transactions (updates) per second.
>> >> >
>> >> > Is the recommendation to use EmbeddedSolrServer instead of
>> >> HttpSolrServer?
>> >> >
>> >> > Thanks,
>> >> > Balaji
>> >> >
>> >> > Balaji Gandhi, Senior Software Developer, Horizontal Platform
>> Services
>> >> > Product Engineering │ Apollo Group, Inc.
>> >> > 1225 W. Washington St. | AZ23 | Tempe, AZ 85281
>> >> > Phone: 602.713.2417 | Email: [hidden
>> >> email]</user/SendEmail.jtp?type=node&node=4012179&i=1><mailto:
>> >> > [hidden email]</user/SendEmail.jtp?type=node&node=4012179&i=2>>
>> >> >
>> >> > P Go Green. Don't Print. Moreover soft copies can be indexed by
>> >> algorithms.
>> >> >
>> >> > From: Sushil jain [via Lucene] [mailto:
>> >> > [hidden email]</user/SendEmail.jtp?type=node&node=4012179&i=3>]
>> >> > Sent: Friday, October 05, 2012 1:14 PM
>> >> > To: Balaji Gandhi
>> >> > Subject: Re: SolrJ - IOException
>> >> >
>> >> > Balaji,
>> >> >
>> >> > What is 30 TPS ?
>> >> >
>> >> > Toke,
>> >> >
>> >> > You should use EmbeddedSolrServer Instead.
>> >> >
>> >> >
>> >> >
>> >> > On Fri, Oct 5, 2012 at 11:42 PM, balaji.gandhi
>> >> > <[hidden
>> email]</user/SendEmail.jtp?type=node&node=4012172&i=0>>wrote:
>> >> >
>> >> > > Hi Toke,
>> >> > >
>> >> > > Were you able to find anything on this issue? We are running at 30
>> TPS
>> >> > and
>> >> > > using the default HttpSolrServer for the posts.
>> >> > >
>> >> > > [cid:image001.png@01CDA2EA.370A6ED0]
>> >> > >
>> >> > > Thanks,
>> >> > > Balaji
>> >> > >
>> >> > > Balaji Gandhi, Senior Software Developer, Horizontal Platform
>> Services
>> >> > > Product Engineering │ Apollo Group, Inc.
>> >> > > 1225 W. Washington St. | AZ23 | Tempe, AZ 85281
>> >> > > Phone: 602.713.2417 | Email: [hidden
>> >> > email]</user/SendEmail.jtp?type=node&node=4012172&i=1><mailto:
>> >> > > [hidden email]</user/SendEmail.jtp?type=node&node=4012172&i=2>>
>> >> > >
>> >> > > P Go Green. Don't Print. Moreover soft copies can be indexed by
>> >> > algorithms.
>> >> > >
>> >> > > From: Toke Eskildsen [via Lucene] [mailto:
>> >> > > [hidden email]</user/SendEmail.jtp?type=node&node=4012172&i=3>]
>> >> > > Sent: Tuesday, September 25, 2012 12:19 AM
>> >> > > To: Balaji Gandhi
>> >> > > Subject: Re: SolrJ - IOException
>> >> > >
>> >> > > On Tue, 2012-09-25 at 01:50 +0200, balaji.gandhi wrote:
>> >> > > > I am encountering this error randomly (under load) when posting
>> to
>> >> Solr
>> >> > > > using SolrJ.
>> >> > > >
>> >> > > > Has anyone encountered a similar error?
>> >> > > >
>> >> > > > org.apache.solr.client.solrj.SolrServerException: IOException
>> occured
>> >> > > when
>> >> > > > talking to server at: http://localhost:8080/solr/profile at
>> >> > > >
>> >> > >
>> >> >
>> >>
>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:414)
>> >> > > [...]
>> >> > >
>> >> > > This looks suspiciously like a potential bug in the HTTP keep-alive
>> >> flow
>> >> > > that we encountered some weeks ago. I am guessing that you are
>> issuing
>> >> > > more than 100 separate updates/second. Could you please provide the
>> >> full
>> >> > > stack trace?
>> >> > >
>> >> > >
>> >> > > ________________________________
>> >> > > If you reply to this email, your message will be added to the
>> >> discussion
>> >> > > below:
>> >> > >
>> >> >
>> >>
>> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4010082.html
>> >> > > To unsubscribe from SolrJ - IOException, click here<
>> >> > > >.
>> >> > > NAML<
>> >> > >
>> >> >
>> >>
>> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>> >> > > >
>> >> > >
>> >> > > ________________________________
>> >> > > This message is private and confidential. If you have received it
>> in
>> >> > > error, please notify the sender and remove it from your system.
>> >> > >
>> >> > >
>> >> > >
>> >> > > image001.png (37K) <
>> >> > >
>> http://lucene.472066.n3.nabble.com/attachment/4012130/0/image001.png>
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > View this message in context:
>> >> > >
>> >> >
>> >>
>> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012130.html
>> >> > > Sent from the Solr - User mailing list archive at Nabble.com.
>> >> >
>> >> > ________________________________
>> >> > If you reply to this email, your message will be added to the
>> discussion
>> >> > below:
>> >> >
>> >>
>> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012172.html
>> >> > To unsubscribe from SolrJ - IOException, click here<
>> >> > >.
>> >> > NAML<
>> >> >
>> >>
>> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>> >> > >
>> >> >
>> >> > ________________________________
>> >> > This message is private and confidential. If you have received it in
>> >> > error, please notify the sender and remove it from your system.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > View this message in context:
>> >> >
>> >>
>> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012176.html
>> >> > Sent from the Solr - User mailing list archive at Nabble.com.
>> >> >
>> >>
>> >> ________________________________
>> >> If you reply to this email, your message will be added to the
>> discussion
>> >> below:
>> >>
>> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012179.html
>> >> To unsubscribe from SolrJ - IOException, click here<
>> >>
>> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4010026&code=YmFsYWppLmdhbmRoaUBhcG9sbG9ncnAuZWR1fDQwMTAwMjZ8LTEwNzE2NTA1NDI=
>> >> >.
>> >> NAML<
>> >>
>> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
>> >> >
>> >>
>> >> ________________________________
>> >> This message is private and confidential. If you have received it in
>> >> error, please notify the sender and remove it from your system.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://lucene.472066.n3.nabble.com/SolrJ-IOException-tp4010026p4012182.html
>> >> Sent from the Solr - User mailing list archive at Nabble.com.
>> >>
>>
>
>
