How long is your timeout? Maybe it should be longer, since this is normal Solr behavior. --wunder
-----Original Message----- From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] Sent: Monday, October 05, 2009 9:45 AM To: solr-user@lucene.apache.org Subject: RE: Solr Timeouts Is there somewhere other than solrConfig.xml that the autoCommit feature is enabled? I've looked through that file and found autocommit to be commented out: <!-- Perform a <commit/> automatically under certain conditions: maxDocs - number of updates since last commit is greater than this maxTime - oldest uncommited update (in ms) is this long ago <autoCommit> <maxDocs>10000</maxDocs> <maxTime>1000</maxTime> </autoCommit> --> This is what one of my SOLR requests look like: http://titans:8080/solr/update/extract/?literal.versionId=684936&literal.fil ingDate=1997-12-04T00:00:00Z&literal.formTypeId=95&literal.companyId=3567904 &literal.sourceId=0&resource.name=684936.txt&commit=false -----Original Message----- From: Feak, Todd [mailto:todd.f...@smss.sony.com] Sent: Monday, October 05, 2009 12:40 PM To: solr-user@lucene.apache.org Subject: RE: Solr Timeouts Actually, ignore my other response. I believe you are committing, whether you know it or not. This is in your provided stack trace org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProces sor, SolrParams, boolean) org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQuery Request, SolrQueryResponse) I think Yonik gave you additional information for how to make it faster. -Todd -----Original Message----- From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] Sent: Monday, October 05, 2009 9:30 AM To: solr-user@lucene.apache.org Subject: RE: Solr Timeouts I'm not committing at all actually - I'm waiting for all 6 million to be done. -----Original Message----- From: Feak, Todd [mailto:todd.f...@smss.sony.com] Sent: Monday, October 05, 2009 12:10 PM To: solr-user@lucene.apache.org Subject: RE: Solr Timeouts How often are you committing? Every time you commit, Solr will close the old index and open the new one. If you are doing this in parallel from multiple jobs (4-5 you mention) then eventually the server gets behind and you start to pile up commit requests. Once this starts to happen, it will cascade out of control if the rate of commits isn't slowed. -Todd ________________________________ From: Giovanni Fernandez-Kincade [mailto:gfernandez-kinc...@capitaliq.com] Sent: Monday, October 05, 2009 9:04 AM To: solr-user@lucene.apache.org Subject: Solr Timeouts Hi, I'm attempting to index approximately 6 million HTML/Text files using SOLR 1.4/Tomcat6 on Windows Server 2003 x64. I'm running 64 bit Tomcat and JVM. I've fired up 4-5 different jobs that are making indexing requests using the ExtractionRequestHandler, and everything works well for about 30-40 minutes, after which all indexing requests start timing out. I profiled the server and found that all of the threads are getting blocked by this call to flush the Lucene index to disk (see below). This leads me to a few questions: 1. Is this normal? 2. Can I reduce the frequency with which this happens somehow? I've greatly increased the indexing options in SolrConfig.xml (attached here) to no avail. 3. During these flushes, resource utilization (CPU, I/O, Memory Consumption) is significantly down compared to when requests are being handled. Is there any way to make this index go faster? I have plenty of bandwidth on the machine. I appreciate any insight you can provide. We're currently using MS SQL 2005 as our full-text solution and are pretty much miserable. So far SOLR has been a great experience. Thanks, Gio. http-8080-Processor21 [RUNNABLE] CPU time: 9:51 java.io.RandomAccessFile.seek(long) org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.readInternal(by te[], int, int) org.apache.lucene.store.BufferedIndexInput.refill() org.apache.lucene.store.BufferedIndexInput.readByte() org.apache.lucene.store.IndexInput.readVInt() org.apache.lucene.index.TermBuffer.read(IndexInput, FieldInfos) org.apache.lucene.index.SegmentTermEnum.next() org.apache.lucene.index.SegmentTermEnum.scanTo(Term) org.apache.lucene.index.TermInfosReader.get(Term, boolean) org.apache.lucene.index.TermInfosReader.get(Term) org.apache.lucene.index.SegmentTermDocs.seek(Term) org.apache.lucene.index.DocumentsWriter.applyDeletes(IndexReader, int) org.apache.lucene.index.DocumentsWriter.applyDeletes(SegmentInfos) org.apache.lucene.index.IndexWriter.applyDeletes() org.apache.lucene.index.IndexWriter.doFlushInternal(boolean, boolean) org.apache.lucene.index.IndexWriter.doFlush(boolean, boolean) org.apache.lucene.index.IndexWriter.flush(boolean, boolean, boolean) org.apache.lucene.index.IndexWriter.closeInternal(boolean) org.apache.lucene.index.IndexWriter.close(boolean) org.apache.lucene.index.IndexWriter.close() org.apache.solr.update.SolrIndexWriter.close() org.apache.solr.update.DirectUpdateHandler2.closeWriter() org.apache.solr.update.DirectUpdateHandler2.commit(CommitUpdateCommand) org.apache.solr.update.processor.RunUpdateProcessor.processCommit(CommitUpda teCommand) org.apache.solr.handler.RequestHandlerUtils.handleCommit(UpdateRequestProces sor, SolrParams, boolean) org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(SolrQuery Request, SolrQueryResponse) org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest, SolrQueryResponse) org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest (SolrQueryRequest, SolrQueryResponse) org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest, SolrQueryResponse) org.apache.solr.servlet.SolrDispatchFilter.execute(HttpServletRequest, SolrRequestHandler, SolrQueryRequest, SolrQueryResponse) org.apache.solr.servlet.SolrDispatchFilter.doFilter(ServletRequest, ServletResponse, FilterChain) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequ est, ServletResponse) org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response) org.apache.catalina.core.StandardContextValve.invoke(Request, Response) org.apache.catalina.core.StandardHostValve.invoke(Request, Response) org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response) org.apache.catalina.core.StandardEngineValve.invoke(Request, Response) org.apache.catalina.connector.CoyoteAdapter.service(Request, Response) org.apache.coyote.http11.Http11Processor.process(InputStream, OutputStream) org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processC onnection(TcpConnection, Object[]) org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(Socket, TcpConnection, Object[]) org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(Object[]) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() java.lang.Thread.run()
