It's hard to troubleshoot blindly like this, but have you tried manually comparing the contents of the index dir on the master and on the slave(s)? If they are out of sync, have you tried forcing of replication to see if one of the subsequent replication attempts gets the dirs in sync? Do you have more than 1 slave and do they all start having this problem at the same time? Any errors in the logs for any of the scripts involved in replication in 1.3?
Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR ----- Original Message ---- > From: Maduranga Kannangara <mkannang...@infomedia.com.au> > To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> > Sent: Sun, November 8, 2009 10:30:44 PM > Subject: Segment file not found error - after replicating > > Hi guys, > > We use Solr 1.3 for indexing large amounts of data (50G avg) on Linux > environment and use the replication scripts to make replicas those live in > load > balancing slaves. > > The issue we face quite often (only in Linux servers) is that they tend to > not > been able to find the segment file (segment_x etc) after the replicating > completed. As this has become quite common, we started hitting a serious > issue. > > Below is a stack trace, if that helps and any help on this matter is greatly > appreciated. > > -------------------------------- > > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created /admin/ping: org.apache.solr.handler.PingRequestHandler > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created /debug/dump: org.apache.solr.handler.DumpRequestHandler > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created gap: org.apache.solr.highlight.GapFragmenter > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created regex: org.apache.solr.highlight.RegexFragmenter > Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load > INFO: created html: org.apache.solr.highlight.HtmlFormatter > Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init > SEVERE: Could not start SOLR. Check solr/home property > java.lang.RuntimeException: java.io.FileNotFoundException: > /solrinstances/solrhome01/data/index/segments_v (No such file or directory) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960) > at org.apache.solr.core.SolrCore.(SolrCore.java:470) > at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397) > at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4363) > at > org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099) > at > org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916) > at > org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536) > at > org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.FileNotFoundException: > /solrinstances/solrhome01/data/index/segments_v (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:212) > at > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:552) > at > org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:582) > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482) > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214) > at > org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94) > at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653) > at > org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111) > at org.apache.lucene.index.IndexReader.open(IndexReader.java:316) > at org.apache.lucene.index.IndexReader.open(IndexReader.java:237) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951) > ... 30 more > Nov 5, 2009 11:34:46 PM org.apache.solr.common.SolrException log > SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException: > /solrinstances/solrhome01/data/index/segments_v (No such file or directory) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960) > at org.apache.solr.core.SolrCore.(SolrCore.java:470) > at > org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119) > at > org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69) > at > org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275) > at > org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397) > at > org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:108) > at > org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4363) > at > org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099) > at > org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916) > at > org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536) > at > org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.FileNotFoundException: > /solrinstances/solrhome01/data/index/segments_v (No such file or directory) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:212) > at > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.(FSDirectory.java:552) > at > org.apache.lucene.store.FSDirectory$FSIndexInput.(FSDirectory.java:582) > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) > at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482) > at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214) > at > org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94) > at > org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653) > at > org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111) > at org.apache.lucene.index.IndexReader.open(IndexReader.java:316) > at org.apache.lucene.index.IndexReader.open(IndexReader.java:237) > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951) > ... 30 more > > Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init > INFO: SolrDispatchFilter.init() done > Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrServlet init > INFO: SolrServlet.init() > > -------------------------------- > > Steps to re-produce the error (However, for me did not work in my local box. > Also remote server is too far away to remote-debug!). > > - Post some new data to the master server (Usually about 1Gb worth text > files) > - Run the replicate script in slave Solr instance > - Try to login to admin in slave Solr instance > > And you should see above stack trace even in the Tomcat output. > > > Thanks in advance. > Madu
