I'm seeing this issue as well on 1.4.1 where all slaves are using simple as
the locking mechanism. For some unknown reason slaves either don't remove old
index.DATE directories or old index files in the index directory. Only the
second slave has the correct index size.
master
4.8G index
4.8G total
slave 1
9.7G index
4.0K index.20110103022003
4.0K index.20110103125106
4.0K replication.properties
9.7G total
slave 2
4.8G index
4.0K index.20110103115106
4.0K replication.properties
4.8G total
slave 3
4.9G index
9.7G index.20101230101714
4.0K index.properties
4.0K replication.properties
15G total
I've read and searched and read and tried and what not but i cannot find the
cause of the problem, nor do i know how to reproduce but it smells like it has
something to do with restarting the servlet container.
Anyone with clues?
On Sunday 19 December 2010 02:01:40 Lance Norskog wrote:
> This could be a quirk of the native locking feature. What's the file
> system? Can you fsck it?
>
> If this error keeps happening, please file this. It should not happen.
> Add the text above and also your solrconfigs if you can.
>
> One thing you could try is to change from the native locking policy to
> the simple locking policy - but only on the child.
>
> On Sat, Dec 18, 2010 at 4:44 PM, feedly team <feedly...@gmail.com> wrote:
> > I have set up index replication (triggered on optimize). The problem I
> > am having is the old index files are not being deleted on the slave.
> > After each replication, I can see the old files still hanging around
> > as well as the files that have just been pulled. This causes the data
> > directory size to increase by the index size every replication until
> > the disk fills up.
> >
> > Checking the logs, I see the following error:
> >
> > SEVERE: SnapPull failed
> > org.apache.solr.common.SolrException: Index fetch failed :
> > at
> > org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:329)
> > at
> > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.ja
> > va:265) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> > at
> > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:31
> > 7) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acc
> > ess$101(ScheduledThreadPoolExecutor.java:98) at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
> > Periodic(ScheduledThreadPoolExecutor.java:181) at
> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run
> > (ScheduledThreadPoolExecutor.java:205) at
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecuto
> > r.java:886) at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> > va:908) at java.lang.Thread.run(Thread.java:619)
> > Caused by: org.apache.lucene.store.LockObtainFailedException: Lock
> > obtain timed out:
> > NativeFSLock@/var/solrhome/data/index/lucene-cdaa80c0fefe1a7dfc7aab89298c
> > 614c-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:84)
> > at
> > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1065) at
> > org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:954) at
> > org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:192)
> > at
> > org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler
> > .java:99) at
> > org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHandl
> > er2.java:173) at
> > org.apache.solr.update.DirectUpdateHandler2.forceOpenWriter(DirectUpdate
> > Handler2.java:376) at
> > org.apache.solr.handler.SnapPuller.doCommit(SnapPuller.java:471) at
> > org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:319)
> > ... 11 more
> >
> > lsof reveals that the file is still opened from the java process.
> >
> > I am running 4.0 rev 993367 with patch SOLR-1316. Otherwise, the setup
> > is pretty vanilla. The OS is linux, the indexes are on local
> > directories, write permissions look ok, nothing unusual in the config
> > (default deletion policy, etc.). Contents of the index data dir:
> >
> > master:
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 14 01:06 _1lg.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 14 01:07 _1lg.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 14 01:07 _1lg.fdt
> > -rw-rw-r-- 1 feeddo feeddo 474M Dec 14 01:12 _1lg.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 14 01:12 _1lg.tii
> > -rw-rw-r-- 1 feeddo feeddo 144M Dec 14 01:12 _1lg.prx
> > -rw-rw-r-- 1 feeddo feeddo 277M Dec 14 01:12 _1lg.frq
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 14 01:12 segments_1ji
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 14 01:12 _1lg.nrm
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt
> > -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii
> > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx
> > -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt
> > -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii
> > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx
> > -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq
> > -rw-rw-r-- 1 feeddo feeddo 20 Dec 18 13:35 segments.gen
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm
> >
> > slave:
> > -rw-rw-r-- 1 feeddo feeddo 20 Dec 13 17:54 segments.gen
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 01:07 _1mk.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 01:08 _1mk.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 01:08 _1mk.fdt
> > -rw-rw-r-- 1 feeddo feeddo 476M Dec 15 01:18 _1mk.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 01:18 _1mk.tii
> > -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 01:18 _1mk.prx
> > -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 01:18 _1mk.frq
> > -rw-rw-r-- 1 feeddo feeddo 312 Dec 15 01:18 segments_1kj
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 01:18 _1mk.nrm
> > -rw-rw-r-- 1 feeddo feeddo 0 Dec 15 01:19
> > lucene-cdaa80c0fefe1a7dfc7aab89298c614c-write.lock
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 15 13:14 _1qu.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 15 13:16 _1qu.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 15 13:16 _1qu.fdt
> > -rw-rw-r-- 1 feeddo feeddo 477M Dec 15 13:28 _1qu.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 15 13:28 _1qu.tii
> > -rw-rw-r-- 1 feeddo feeddo 144M Dec 15 13:28 _1qu.prx
> > -rw-rw-r-- 1 feeddo feeddo 278M Dec 15 13:28 _1qu.frq
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 15 13:28 segments_1oe
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 15 13:28 _1qu.nrm
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 01:12 _222.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 01:15 _222.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 01:15 _222.fdt
> > -rw-rw-r-- 1 feeddo feeddo 481M Dec 17 01:36 _222.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 01:36 _222.tii
> > -rw-rw-r-- 1 feeddo feeddo 145M Dec 17 01:36 _222.prx
> > -rw-rw-r-- 1 feeddo feeddo 281M Dec 17 01:36 _222.frq
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 01:36 _222.nrm
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 01:36 segments_1xv
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 17 13:10 _233.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 17 13:13 _233.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 17 13:13 _233.fdt
> > -rw-rw-r-- 1 feeddo feeddo 482M Dec 17 13:31 _233.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 17 13:31 _233.tii
> > -rw-rw-r-- 1 feeddo feeddo 146M Dec 17 13:31 _233.prx
> > -rw-rw-r-- 1 feeddo feeddo 282M Dec 17 13:31 _233.frq
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 17 13:31 segments_1xx
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 17 13:31 _233.nrm
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 01:11 _24e.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 01:12 _24e.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 01:12 _24e.fdt
> > -rw-rw-r-- 1 feeddo feeddo 483M Dec 18 01:23 _24e.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 01:23 _24e.tii
> > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 01:23 _24e.prx
> > -rw-rw-r-- 1 feeddo feeddo 283M Dec 18 01:23 _24e.frq
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 01:24 segments_1xz
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 01:24 _24e.nrm
> > -rw-rw-r-- 1 feeddo feeddo 191 Dec 18 13:15 _25z.fnm
> > -rw-rw-r-- 1 feeddo feeddo 26M Dec 18 13:16 _25z.fdx
> > -rw-rw-r-- 1 feeddo feeddo 1.9G Dec 18 13:16 _25z.fdt
> > -rw-rw-r-- 1 feeddo feeddo 484M Dec 18 13:35 _25z.tis
> > -rw-rw-r-- 1 feeddo feeddo 15M Dec 18 13:35 _25z.tii
> > -rw-rw-r-- 1 feeddo feeddo 146M Dec 18 13:35 _25z.prx
> > -rw-rw-r-- 1 feeddo feeddo 284M Dec 18 13:35 _25z.frq
> > -rw-rw-r-- 1 feeddo feeddo 311 Dec 18 13:35 segments_1y1
> > -rw-rw-r-- 1 feeddo feeddo 23M Dec 18 13:35 _25z.nrm
> >
> >
> > Any pointers on how to proceed? Thanks.
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350
