I have set up index replication (triggered on optimize). The problem I
am having is the old index files are not being deleted on the slave.
After each replication, I can see the old files still hanging around
as well as the files that have just been pulled. This causes the data
directory size to increase by the index size every replication until
the disk fills up.
Checking the logs, I see the following error:
SEVERE: SnapPull failed
org.apache.solr.common.SolrException: Index fetch failed :
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:265)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock
obtain timed out:
NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler.java:99)
at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandler2.java:173)
at
org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdateHandler2.java:376)
at org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
... 11 more
lsof reveals that the file is still opened from the java process.
I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup
is pretty vanilla. The OS is linux, the indexes are on local
directories, write permissions look ok, nothing unusual in the config
(default deletion policy, etc.). Contents of the index data dir:
master:
-rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt
-rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii
-rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx
-rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji
-rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt
-rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx
-rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt
-rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx
-rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq
-rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm
slave:
-rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen
-rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt
-rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii
-rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx
-rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq
-rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj
-rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm
-rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19
lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock
-rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt
-rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii
-rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx
-rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe
-rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt
-rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii
-rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx
-rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq
-rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm
-rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv
-rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt
-rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx
-rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx
-rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt
-rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx
-rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm
-rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm
-rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx
-rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt
-rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis
-rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii
-rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx
-rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq
-rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1
-rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm
Any pointers on how to proceed? Thanks.
