I'm inserting 10K in a batch (using addBeans method). I read somewhere in the wiki that it's better to use the same instance of SolrServer for better performance. Would MultiThreadedConnectionManager help? How do I use it?
I also wanted to know how can use EmbeddedSolrServer - does my app needs to be running in the same jvm with Solr webapp? Thanks, -vivek 2009/4/9 Noble Paul നോബിള് नोब्ळ् <noble.p...@gmail.com>: > how many documents are you inserting ? > may be you can create multiple instances of CommonshttpSolrServer and > upload in parallel > > > On Thu, Apr 9, 2009 at 11:58 AM, vivek sar <vivex...@gmail.com> wrote: >> Thanks Shalin and Paul. >> >> I'm not using MultipartRequest. I do share the same SolrServer between >> two threads. I'm not using MultiThreadedHttpConnectionManager. I'm >> simply using CommonsHttpSolrServer to create the SolrServer. I've also >> tried StreamingUpdateSolrServer, which works much faster, but does >> throws "connection reset" exception once in a while. >> >> Do I need to use MultiThreadedHttpConnectionManager? I couldn't find >> anything on it on Wiki. >> >> I was also thinking of using EmbeddedSolrServer - in what case would I >> be able to use it? Does my application and the Solr web app need to >> run into the same JVM for this to work? How would I use the >> EmbeddedSolrServer? >> >> Thanks, >> -vivek >> >> >> On Wed, Apr 8, 2009 at 10:46 PM, Shalin Shekhar Mangar >> <shalinman...@gmail.com> wrote: >>> Vivek, do you share the same SolrServer instance between your two threads? >>> If so, are you using the MultiThreadedHttpConnectionManager when creating >>> the HttpClient instance? >>> >>> On Wed, Apr 8, 2009 at 10:13 PM, vivek sar <vivex...@gmail.com> wrote: >>> >>>> single thread everything works fine. Two threads are fine too for a >>>> while and all the sudden problem starts happening. >>>> >>>> I tried indexing using REST services as well (instead of Solrj), but >>>> with that too I get following error after a while, >>>> >>>> 2009-04-08 10:04:08,126 ERROR [indexerThreadPool-5] Indexer - >>>> indexData()-> Failed to index >>>> java.net.SocketException: Broken pipe >>>> at java.net.SocketOutputStream.socketWrite0(Native Method) >>>> at >>>> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) >>>> at java.net.SocketOutputStream.write(SocketOutputStream.java:136) >>>> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) >>>> at java.io.FilterOutputStream.write(FilterOutputStream.java:80) >>>> at >>>> org.apache.commons.httpclient.methods.StringRequestEntity.writeRequest(StringRequestEntity.java:145) >>>> at >>>> org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114) >>>> at >>>> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096) >>>> at >>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) >>>> at >>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) >>>> at >>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>>> at >>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>>> >>>> >>>> Note, I'm using "simple" lock type. I'd tried "single" type before >>>> that once caused index corruption so I switched to "simple". >>>> >>>> Thanks, >>>> -vivek >>>> >>>> 2009/4/8 Noble Paul നോബിള് नोब्ळ् <noble.p...@gmail.com>: >>>> > do you see the same problem when you use a single thread? >>>> > >>>> > what is the version of SolrJ that you use? >>>> > >>>> > >>>> > >>>> > On Wed, Apr 8, 2009 at 1:19 PM, vivek sar <vivex...@gmail.com> wrote: >>>> >> Hi, >>>> >> >>>> >> Any ideas on this issue? I ran into this again - once it starts >>>> >> happening it keeps happening. One of the thread keeps failing. Here >>>> >> are my SolrServer settings, >>>> >> >>>> >> int socketTO = 0; >>>> >> int connectionTO = 100; >>>> >> int maxConnectionPerHost = 10; >>>> >> int maxTotalConnection = 50; >>>> >> boolean followRedirects = false; >>>> >> boolean allowCompression = true; >>>> >> int maxRetries = 1; >>>> >> >>>> >> Note, I'm using two threads to simultaneously write to the same index. >>>> >> >>>> >> org.apache.solr.client.solrj.SolrServerException: >>>> >> org.apache.commons.httpclient.ProtocolException: Unbuffered entity >>>> >> enclosing request can not be repeated. >>>> >> at >>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:470) >>>> >> at >>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242) >>>> >> at >>>> org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259) >>>> >> at >>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48) >>>> >> at >>>> org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:57) >>>> >> >>>> >> Thanks, >>>> >> -vivek >>>> >> >>>> >> On Sat, Apr 4, 2009 at 1:07 AM, vivek sar <vivex...@gmail.com> wrote: >>>> >>> Hi, >>>> >>> >>>> >>> I'm sending 15K records at once using Solrj (server.addBeans(...)) >>>> >>> and have two threads writing to same index. One thread goes fine, but >>>> >>> the second thread always fails with, >>>> >>> >>>> >>> >>>> >>> org.apache.solr.client.solrj.SolrServerException: >>>> >>> org.apache.commons.httpclient.ProtocolException: Unbuffered entity >>>> >>> enclosing request can not be repeated. >>>> >>> at >>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:470) >>>> >>> at >>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:242) >>>> >>> at >>>> org.apache.solr.client.solrj.request.UpdateRequest.process(UpdateRequest.java:259) >>>> >>> at >>>> org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:48) >>>> >>> at >>>> org.apache.solr.client.solrj.SolrServer.addBeans(SolrServer.java:57) >>>> >>> at >>>> com.apple.afterchat.indexer.solr.handler.BeanIndexHandler.indexData(BeanIndexHandler.java:44) >>>> >>> at >>>> com.apple.afterchat.indexer.Indexer.indexData(Indexer.java:77) >>>> >>> at com.apple.afterchat.indexer.Indexer.run(Indexer.java:39) >>>> >>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) >>>> >>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>> >>> at java.lang.Thread.run(Thread.java:637) >>>> >>> Caused by: org.apache.commons.httpclient.ProtocolException: Unbuffered >>>> >>> entity enclosing request can not be repeated. >>>> >>> at >>>> org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:487) >>>> >>> at >>>> org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114) >>>> >>> at >>>> org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096) >>>> >>> at >>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) >>>> >>> at >>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) >>>> >>> at >>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) >>>> >>> at >>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) >>>> >>> at >>>> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:417) >>>> >>> >>>> >>> Does anyone know what could be the problem? >>>> >>> >>>> >>> Thanks, >>>> >>> -vivek >>>> >>> >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > --Noble Paul >>>> > >>>> >>> >>> >>> >>> -- >>> Regards, >>> Shalin Shekhar Mangar. >>> >> > > > > -- > --Noble Paul >
