Could you post the full output of the CheckIndex command on the restored snapshot? Also what happens if you delete the snapshot indexes and attempt to restore again? Does it get corrupted again or is it a one off scenario?
On Wed, Mar 2, 2016 at 3:44 PM, Janit Anjaria (Tech-IT) < anjaria.ja...@flipkart.com> wrote: > Hi, > > Varun, we actually ran the test for our restored data snapshot and it > threw an error saying "Broken segment". > > How is it possible that the same test gives success on the snapshot, but > not on the restored snapshot? Can you please throw some light on this, so > we can proceed and fix this issue. > > Regards, > Janit > > On Tue, Mar 1, 2016 at 12:05 PM, Varun Thacker <varunthacker1...@gmail.com > > wrote: > >> Hi Janit, >> >> Please ask these questions the solr-user mailing list. There is no point >> in double posting to the dev list as well. >> >> Looking at the stacktrace it looks like the index files are corrupted. >> On the backed up index can you run the CheckIndex >> command to see what it says. The command would look something like this >> - : >> java -cp SOLR_HOME/example/webapps/WEB-INF/lib/lucene-core-[version].jar >> -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex >> /path/to/backup/data/index >> >> If the CheckIndex command says its corrupted then there is something >> wrong with the backup. If the CheckIndex >> says the index is fine then there might be something wrong with the >> restore process in which case we'll need to dig deeper >> >> On Tue, Mar 1, 2016 at 11:21 AM, Janit Anjaria (Tech-IT) < >> anjaria.ja...@flipkart.com> wrote: >> >>> Hi, >>> >>> Please find below the console logs for the above issue we have been >>> facing: >>> >>> org.apache.solr.common.SolrException: Error opening new searcher >>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:824) >>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:665) >>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:746) >>> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:466) >>> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:457) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> Caused by: org.apache.solr.common.SolrException: Error opening new searcher >>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1667) >>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1778) >>> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:920) >>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:797) >>> ... 9 more >>> Caused by: org.apache.lucene.index.CorruptIndexException: codec footer >>> mismatch (file truncated?): actual footer=438025 vs expected >>> footer=-1071082520 >>> (resource=MMapIndexInput(path="/var/solr/data/fk-ekl-mmi-mapsearch_shard1_replica1/data/restore.snapshot.20160204065051901/_1tdyo.cfs")) >>> at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:416) >>> at >>> org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:401) >>> at >>> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:79) >>> at >>> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) >>> at >>> org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1016) >>> at >>> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1033) >>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938) >>> at >>> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:79) >>> at >>> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:66) >>> at >>> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:235) >>> at >>> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:109) >>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1636) >>> ... 12 more >>> >>> 3/1/2016, 11:19:44 AM ERROR null CoreContainer Error waiting for >>> SolrCore to be created >>> >>> java.util.concurrent.ExecutionException: >>> org.apache.solr.common.SolrException: Unable to create core >>> [fk-ekl-mmi-mapsearch_shard1_replica1] >>> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >>> at java.util.concurrent.FutureTask.get(FutureTask.java:188) >>> at org.apache.solr.core.CoreContainer$2.run(CoreContainer.java:495) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>> at >>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> Caused by: org.apache.solr.common.SolrException: Unable to create core >>> [fk-ekl-mmi-mapsearch_shard1_replica1] >>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:760) >>> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:466) >>> at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:457) >>> ... 5 more >>> Caused by: org.apache.solr.common.SolrException: Error opening new searcher >>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:824) >>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:665) >>> at org.apache.solr.core.CoreContainer.create(CoreContainer.java:746) >>> ... 7 more >>> Caused by: org.apache.solr.common.SolrException: Error opening new searcher >>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1667) >>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1778) >>> at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:920) >>> at org.apache.solr.core.SolrCore.<init>(SolrCore.java:797) >>> ... 9 more >>> Caused by: org.apache.lucene.index.CorruptIndexException: codec footer >>> mismatch (file truncated?): actual footer=438025 vs expected >>> footer=-1071082520 >>> (resource=MMapIndexInput(path="/var/solr/data/fk-ekl-mmi-mapsearch_shard1_replica1/data/restore.snapshot.20160204065051901/_1tdyo.cfs")) >>> at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:416) >>> at >>> org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:401) >>> at >>> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:79) >>> at >>> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:71) >>> at >>> org.apache.lucene.index.IndexWriter.readFieldInfos(IndexWriter.java:1016) >>> at >>> org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1033) >>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938) >>> at >>> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:79) >>> at >>> org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:66) >>> at >>> org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:235) >>> at >>> org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:109) >>> at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1636) >>> ... 12 more >>> >>> >>> Regards, >>> >>> Janit >>> >>> >>> On Tue, Mar 1, 2016 at 11:02 AM, Janit Anjaria (Tech-IT) < >>> anjaria.ja...@flipkart.com> wrote: >>> >>>> Hi, >>>> >>>> We have been facing the problem of corrupted solr index, after >>>> restoring our data. When we restore the data, the status shows 'success' >>>> after some time. But later, when we make some config changes and then try >>>> to stop the instance and restart it with the new configurations - the index >>>> is corrupted. >>>> >>>> Can someone throw light on the reason that might lead to this problem, >>>> or is there something we are missing in the restore process ? Some help >>>> would be greatly appreciated as this is a high priority task for us. >>>> >>>> Hope to receive a positive and prompt reply. >>>> >>>> Regards, >>>> Janit >>>> >>> >>> >> >> >> -- >> >> >> Regards, >> Varun Thacker >> > > -- Regards, Varun Thacker
