Re: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : SingleInstanceLock: write.lock

[Lance Norskog]

Hello-

There were a few bugs in this area that are fixed in Solr 1.4. There are 
many other bugs which were also fixed. We suggest everyone upgrade to 1.4.

There are different locking managers, and you may be able to use a 
different one. Also, if this is over NFS that can cause further problems.

Lance

Bharat Jain wrote:
Hi,

We are using SOLR 1.3 and getting this error often. There is only one
instance that index the data. I did some analysis which I have put below and
scenario when this error can happen. Can you guys please validate the issue?
thanks a lot in advance.

SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed
out
: SingleInstanceLock: write.lock
         at org.apache.lucene.store.Lock.obtain(Lock.java:85)
         at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1140)
         at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938)
         at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:116)
         at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHand
ler.java:122)
         at
org.apache.solr.update.DirectUpdateHandler2.openWriter(DirectUpdateHa
ndler2.java:167)
         at
org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandle
r2.java:221)

I think this error can happen in the following scneario

1. Thread T1 enters the commit method and its actually an optimize request.
     a. T1 gets the iwCommit lock
     b. T1 enters the openWriter() method
     c. T1 does the (writer == null check) ->  time x1

2. Thread T2 enters the addDoc method
     a. T2 gets iwAccess lock
     b. T2 gets the mutex "this"
     c. T2 enters operWriter method
     d. T2 does the (writer == null check) ->  time x1

Now after the 1.c is done thread yields and 2.d gets execution and it also
sees writer as null and so now both threads will try to create the
indexwriter and it will fail. I have pasted the relevant portion of code
here.


   // iwCommit protects internal data and open/close of the IndexWriter and
   // is a mutex. Any use of the index writer should be protected by
iwAccess,
   // which admits multiple simultaneous acquisitions.  iwAccess is
   // mutually-exclusive with the iwCommit lock.
   protected final Lock iwAccess, iwCommit;



// must only be called when iwCommit lock held
   protected void openWriter() throws IOException {
     if (writer==null) {
       writer = createMainIndexWriter("DirectUpdateHandler2", false);
     }
   }

addDoc(...) {
...

iwAccess.lock();
     try {

       // We can't use iwCommit to protect internal data here, since it would
       // block other addDoc calls.  Hence, we synchronize to protect
internal
       // state.  This is safe as all other state-changing operations are
       // protected with iwCommit (which iwAccess excludes from this block).
       synchronized (this) {
         // adding document -- prep writer
         closeSearcher();
         openWriter();
         tracker.addedDocument();
       } // end synchronized block

       // this is the only unsynchronized code in the iwAccess block, which
       // should account for most of the time

}

commit() {
...

iwCommit.lock();
     try {
       log.info("start "+cmd);

       if (cmd.optimize) {
         closeSearcher();
         openWriter();
         writer.optimize(cmd.maxOptimizeSegments);
       }

       closeSearcher();
       closeWriter();

       callPostCommitCallbacks();
}

Thanks
Bharat Jain