Optimizing indexing time is a very different question. I'm guessing your 3mins+ time you refer to is the commit time.
There are a whole host of things to take into account regarding indexing, like: number of segments, schema, how many fields, storing fields, omitting norms, caching, autowarming, search activity etc. - the list goes on... The trouble is, you can look at 100 different Solr installations with slow indexing, and find 200 different reasons why each is slow. The best place to start is to get a full understanding of precisely how your data is being stored in the index, starting with adding docs, going through your schema, Lucene segments, solrconfig.xml etc, looking at caches, commit triggers etc. - really getting to know how each step is affecting performance. Once you really have a handle on all the indexing steps, you'll be able to spot the bottlenecks that relate to your particular environment. An index of 4.5GB isn't that big (but the number of documents tends to have more of an effect than the physical size), so the bottleneck(s) should be findable once you trace through the indexing operations. On Thu, Aug 11, 2011 at 1:02 PM, Naveen Gupta <nkgiit...@gmail.com> wrote: > Yes this was happening because of JVM heap size > > But the real issue is that if our index size is growing (very high) > > then indexing time is taking very long (using streaming) > > earlier for indexing 15,000 docs at a time (commit after 15000 docs) , it > was taking 3 mins 20 secs time, > > after deleting the index data, it is taking 9 secs > > What would be approach to have better indexing performance as well as index > size should also at the same time. > > The index size was around 4.5 GB > > Thanks > Naveen > > On Thu, Aug 11, 2011 at 3:47 PM, Peter Sturge <peter.stu...@gmail.com>wrote: > >> Hi, >> >> When you get this exception with no other error or explananation in >> the logs, this is almost always because the JVM has run out of memory. >> Have you checked/profiled your mem usage/GC during the stream operation? >> >> >> >> On Thu, Aug 11, 2011 at 3:18 AM, Naveen Gupta <nkgiit...@gmail.com> wrote: >> > Hi, >> > >> > We are doing streaming update to solr for multiple user, >> > >> > We are getting >> > >> > >> > Aug 10, 2011 11:56:55 AM org.apache.solr.common.SolrException log >> > >> > SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain >> timed >> > out: NativeFSLock@/var/lib/solr/data/index/write.lock >> > at org.apache.lucene.store.Lock.obtain(Lock.java:84) >> > at >> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1097) >> > at >> > org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:83) >> > at >> > >> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:102) >> > at >> > >> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:174) >> > at >> > >> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:222) >> > at >> > >> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) >> > at >> > org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:147) >> > at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77) >> > at >> > >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:55) >> > at >> > >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >> > at >> > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> > at >> > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> > at >> > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >> > at >> > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >> > at >> > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >> > at >> > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) >> > at >> > >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) >> > at >> > >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) >> > at org.apache.tomcat.util.net.JIoEndpoint >> > >> > Aug 10, 2011 12:00:16 PM org.apache.solr.common.SolrException log >> > SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain >> timed >> > out: NativeFSLock@/var/lib/solr/data/index/write.lock >> > at org.apache.lucene.store.Lock.obtain(Lock.java:84) >> > at >> org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1097) >> > at >> > org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:83) >> > at >> > >> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:102) >> > at >> > >> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:174) >> > at >> > >> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:222) >> > at >> > >> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:61) >> > at >> > org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:147) >> > at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77) >> > at >> > >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:55) >> > at >> > >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356) >> > at >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) >> > at >> > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) >> > at >> > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> > at >> > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) >> > at >> > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) >> > at >> > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) >> > at >> > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) >> > at >> > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) >> > at >> > >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) >> > at >> > >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) >> > at >> > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) >> > at java.lang.Thread.run(Thread.java:662) >> > >> > Aug 10, 2011 12:59:56 PM org.apache.solr.update.SolrIndexWriter finalize >> > SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a >> bug >> > -- POSSIBLE RESOURCE LEAK!!! >> > >> > Kindly tell me where it is failing >> > >> > We have increased timelockout. But still it is giving the same problem >> > >> > Thanks >> > Naveen >> > >> >
