Interestingly that worked. I deleted the slave index and restarted. After the first replication I shut down the server, deleted the lock file and started it again. It seems to be behaving itself now even though a lock file seems to be recreated. Thanks a lot for the help. This still seems like a bug though?
I don't have any writers open on the slaves, in fact one slave is only doing replication right now (no reads) to try to isolate the problem. On Sat, Jan 22, 2011 at 7:34 PM, Alexander Kanarsky <kanarsky2...@gmail.com> wrote: > I see the file > > -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19 > lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock > > was created on Dec. 15. At the end of the replication, as far as I > remember, the SnapPuller tries to open the writer to ensure the old > files are deleted, and in > your case it cannot obtain a lock on the index folder on Dec 16, > 17,18. Can you reproduce the problem if you delete the lock file, > restart the slave > and try replication again? Do you have any other Writer(s) open for > this folder outside of this core? > > -Alexander > > On Sat, Jan 22, 2011 at 3:52 PM, feedly team <feedly...@gmail.com> wrote: >> The file system checked out, I also tried creating a slave on a >> different machine and could reproduce the issue. I logged SOLR-2329. >> >> On Sat, Dec 18, 2010 at 8:01 PM, Lance Norskog <goks...@gmail.com> wrote: >>> This could be a quirk of the native locking feature. What's the file >>> system? Can you fsck it? >>> >>> If this error keeps happening, please file this. It should not happen. >>> Add the text above and also your solrconfigs if you can. >>> >>> One thing you could try is to change from the native locking policy to >>> the simple locking policy - but only on the child. >>> >>> On Sat, Dec 18, 2010 at 4:44 PM, feedly team <feedly...@gmail.com> wrote: >>>> I have set up index replication (triggered on optimize). The problem I >>>> am having is the old index files are not being deleted on the slave. >>>> After each replication, I can see the old files still hanging around >>>> as well as the files that have just been pulled. This causes the data >>>> directory size to increase by the index size every replication until >>>> the disk fills up. >>>> >>>> Checking the logs, I see the following error: >>>> >>>> SEVERE: SnapPull failed >>>> org.apache.solr.common.SolrException: Index fetch failed : >>>> at >>>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329) >>>> at >>>> org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:265) >>>> at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >>>> at >>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) >>>> at >>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>> at java.lang.Thread.run(Thread.java:619) >>>> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock >>>> obtain timed out: >>>> NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock >>>> at org.apache.lucene.store.Lock.obtain(Lock.java:84) >>>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065) >>>> at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954) >>>> at >>>> org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192) >>>> at >>>> org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:99) >>>> at >>>> org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173) >>>> at >>>> org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdateHandler2.java:376) >>>> at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471) >>>> at >>>> org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319) >>>> ... 11 more >>>> >>>> lsof reveals that the file is still opened from the java process. >>>> >>>> I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup >>>> is pretty vanilla. The OS is linux, the indexes are on local >>>> directories, write permissions look ok, nothing unusual in the config >>>> (default deletion policy, etc.). Contents of the index data dir: >>>> >>>> master: >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii >>>> -rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx >>>> -rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii >>>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx >>>> -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii >>>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx >>>> -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq >>>> -rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm >>>> >>>> slave: >>>> -rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii >>>> -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx >>>> -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq >>>> -rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19 >>>> lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii >>>> -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx >>>> -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii >>>> -rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx >>>> -rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii >>>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx >>>> -rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii >>>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx >>>> -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm >>>> -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm >>>> -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx >>>> -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt >>>> -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis >>>> -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii >>>> -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx >>>> -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq >>>> -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1 >>>> -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm >>>> >>>> >>>> Any pointers on how to proceed? Thanks. >>>> >>> >>> >>> >>> -- >>> Lance Norskog >>> goks...@gmail.com >>> >> >
