Hmm, unless the ulimits are low, or the default mergeFactor was changed, or you have many indexes open in a single JVM, or you keep too many IndexReaders open, even in an NRT or frequent commit use case, you should not run out of file descriptors.
Frequent commit/reopen should be perfectly fine, as long as you close the old readers... Mike McCandless http://blog.mikemccandless.com On Mon, Apr 2, 2012 at 8:35 AM, Erick Erickson <erickerick...@gmail.com> wrote: > How often are you committing index updates? This kind of thing > can happen if you commit too often. Consider setting > commitWithin to something like, say, 5 minutes. Or doing the > equivalent with the autoCommit parameters in solrconfig.xml > > If that isn't relevant, you need to provide some more details > about what you're doing and how you're using Solr > > Best > Erick > > On Sun, Apr 1, 2012 at 10:47 PM, Gopal Patwa <gopalpa...@gmail.com> wrote: >> I am using Solr 4.0 nightly build with NRT and I often get this >> error during auto commit "Too many open files". I have search this forum >> and what I found it is related to OS ulimit setting, please see below my >> ulimit settings. I am not sure what ulimit setting I should have for open >> file? ulimit -n unlimited?. >> >> Even if I set to higher number, it will just delay the issue until it reach >> new open file limit. What I have seen that Solr has kept deleted index file >> open by java process, which causing issue for our application server jboss >> to shutdown gracefully due this open files by java process. >> >> I have seen recently this issue was resolved in lucene, is it TRUE? >> >> https://issues.apache.org/jira/browse/LUCENE-3855 >> >> >> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 >> - 15GB, with Single shard >> >> We update the index every 5 seconds, soft commit every 1 second and hard >> commit every 15 minutes >> >> Environment: Jboss 4.2, JDK 1.6 64 bit, CentOS , JVM Heap Size = 24GB* >> >> >> ulimit: >> >> core file size (blocks, -c) 0 >> >> data seg size (kbytes, -d) unlimited >> >> scheduling priority (-e) 0 >> >> file size (blocks, -f) unlimited >> >> pending signals (-i) 401408 >> >> max locked memory (kbytes, -l) 1024 >> >> max memory size (kbytes, -m) unlimited >> >> open files (-n) 4096 >> >> pipe size (512 bytes, -p) 8 >> >> POSIX message queues (bytes, -q) 819200 >> >> real-time priority (-r) 0 >> >> stack size (kbytes, -s) 10240 >> >> cpu time (seconds, -t) unlimited >> >> max user processes (-u) 401408 >> >> virtual memory (kbytes, -v) unlimited >> >> file locks (-x) unlimited >> >> >> ERROR:* >> >> *2012-04-01* *20:08:35*,*323* [] *priority=ERROR* *app_name=* >> *thread=pool-10-thread-1* *location=CommitTracker* *line=93* *auto* >> *commit* *error...:org.apache.solr.common.SolrException:* *Error* >> *opening* *new* *searcher* >> *at* >> *org.apache.solr.core.SolrCore.openNewSearcher*(*SolrCore.java:1138*) >> *at* *org.apache.solr.core.SolrCore.getSearcher*(*SolrCore.java:1251*) >> *at* >> *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:409*) >> *at* >> *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*) >> *at* >> *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*) >> *at* >> *java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*) >> *at* *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*) >> *at* >> *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301*(*ScheduledThreadPoolExecutor.java:98*) >> *at* >> *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run*(*ScheduledThreadPoolExecutor.java:206*) >> *at* >> *java.util.concurrent.ThreadPoolExecutor$Worker.runTask*(*ThreadPoolExecutor.java:886*) >> *at* >> *java.util.concurrent.ThreadPoolExecutor$Worker.run*(*ThreadPoolExecutor.java:908*) >> *at* *java.lang.Thread.run*(*Thread.java:662*)*Caused* *by:* >> *java.io.FileNotFoundException:* >> */opt/mci/data/srwp01mci001/inventory/index/_4q1y_0.tip* (*Too many >> open files*) >> *at* *java.io.RandomAccessFile.open*(*Native* *Method*) >> *at* *java.io.RandomAccessFile.*<*init*>(*RandomAccessFile.java:212*) >> *at* >> *org.apache.lucene.store.FSDirectory$FSIndexOutput.*<*init*>(*FSDirectory.java:449*) >> *at* >> *org.apache.lucene.store.FSDirectory.createOutput*(*FSDirectory.java:288*) >> *at* >> *org.apache.lucene.codecs.BlockTreeTermsWriter.*<*init*>(*BlockTreeTermsWriter.java:161*) >> *at* >> *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsConsumer*(*Lucene40PostingsFormat.java:66*) >> *at* >> *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField*(*PerFieldPostingsFormat.java:118*) >> *at* >> *org.apache.lucene.index.FreqProxTermsWriterPerField.flush*(*FreqProxTermsWriterPerField.java:322*) >> *at* >> *org.apache.lucene.index.FreqProxTermsWriter.flush*(*FreqProxTermsWriter.java:92*) >> *at* *org.apache.lucene.index.TermsHash.flush*(*TermsHash.java:117*) >> *at* >> *org.apache.lucene.index.DocInverter.flush*(*DocInverter.java:53*) >> *at* >> *org.apache.lucene.index.DocFieldProcessor.flush*(*DocFieldProcessor.java:81*) >> *at* >> *org.apache.lucene.index.DocumentsWriterPerThread.flush*(*DocumentsWriterPerThread.java:475*) >> *at* >> *org.apache.lucene.index.DocumentsWriter.doFlush*(*DocumentsWriter.java:422*) >> *at* >> *org.apache.lucene.index.DocumentsWriter.flushAllThreads*(*DocumentsWriter.java:553*) >> *at* >> *org.apache.lucene.index.IndexWriter.getReader*(*IndexWriter.java:354*) >> *at* >> *org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter*(*StandardDirectoryReader.java:258*) >> *at* >> *org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged*(*StandardDirectoryReader.java:243*) >> *at* >> *org.apache.lucene.index.DirectoryReader.openIfChanged*(*DirectoryReader.java:250*) >> *at* >> *org.apache.solr.core.SolrCore.openNewSearcher*(*SolrCore.java:1091*) >> >> >> Thanks >> >> Gopal
