We cannot avoid auto soft commit, since we need Lucene NRT feature. And I use StreamingUpdateSolrServer for adding/updating index.
On Thu, Apr 19, 2012 at 7:42 AM, Boon Low <boon....@brightsolid.com> wrote: > Hi, > > Also came across this error recently, while indexing with > 10 DIH > processes in parallel + default index setting. The JVM grinds to a halt and > throws this error. Checking the index of a core reveals thousands of files! > Tuning the default autocommit from 15000ms to 900000ms solved the problem > for us. (no 'autosoftcommit'). > > Boon > > ----- > Boon Low > Search UX and Engine Developer > brightsolid Online Publishing > > On 14 Apr 2012, at 17:40, Gopal Patwa wrote: > > > I checked it was "MMapDirectory.UNMAP_SUPPORTED=true" and below are my > > system data. Is their any existing test case to reproduce this issue? I > am > > trying understand how I can reproduce this issue with unit/integration > test > > > > I will try recent solr trunk build too, if it is some bug in solr or > > lucene keeping old searcher open then how to reproduce it? > > > > SYSTEM DATA > > =========== > > PROCESSOR: Intel(R) Xeon(R) CPU E5504 @ 2.00GHz > > SYSTEM ID: x86_64 > > CURRENT CPU SPEED: 1600.000 MHz > > CPUS: 8 processor(s) > > MEMORY: 49449296 kB > > DISTRIBUTION: CentOS release 5.3 (Final) > > KERNEL NAME: 2.6.18-128.el5 > > UPTIME: up 71 days > > LOAD AVERAGE: 1.42, 1.45, 1.53 > > JBOSS Version: Implementation-Version: 4.2.2.GA (build: > > SVNTag=JBoss_4_2_2_GA date=20 > > JAVA Version: java version "1.6.0_24" > > > > > > On Thu, Apr 12, 2012 at 3:07 AM, Michael McCandless < > > luc...@mikemccandless.com> wrote: > > > >> Your largest index has 66 segments (690 files) ... biggish but not > >> insane. With 64K maps you should be able to have ~47 searchers open > >> on each core. > >> > >> Enabling compound file format (not the opposite!) will mean fewer maps > >> ... ie should improve this situation. > >> > >> I don't understand why Solr defaults to compound file off... that > >> seems dangerous. > >> > >> Really we need a Solr dev here... to answer "how long is a stale > >> searcher kept open". Is it somehow possible 46 old searchers are > >> being left open...? > >> > >> I don't see any other reason why you'd run out of maps. Hmm, unless > >> MMapDirectory didn't think it could safely invoke unmap in your JVM. > >> Which exact JVM are you using? If you can print the > >> MMapDirectory.UNMAP_SUPPORTED constant, we'd know for sure. > >> > >> Yes, switching away from MMapDir will sidestep the "too many maps" > >> issue, however, 1) MMapDir has better perf than NIOFSDir, and 2) if > >> there really is a leak here (Solr not closing the old searchers or a > >> Lucene bug or something...) then you'll eventually run out of file > >> descriptors (ie, same problem, different manifestation). > >> > >> Mike McCandless > >> > >> http://blog.mikemccandless.com > >> > >> 2012/4/11 Gopal Patwa <gopalpa...@gmail.com>: > >>> > >>> I have not change the mergefactor, it was 10. Compound index file is > >> disable > >>> in my config but I read from below post, that some one had similar > issue > >> and > >>> it was resolved by switching from compound index file format to > >> non-compound > >>> index file. > >>> > >>> and some folks resolved by "changing lucene code to disable > >> MMapDirectory." > >>> Is this best practice to do, if so is this can be done in > configuration? > >>> > >>> > >> > http://lucene.472066.n3.nabble.com/MMapDirectory-failed-to-map-a-23G-compound-index-segment-td3317208.html > >>> > >>> I have index document of core1 = 5 million, core2=8million and > >>> core3=3million and all index are hosted in single Solr instance > >>> > >>> I am going to use Solr for our site StubHub.com, see attached "ls -l" > >> list > >>> of index files for all core > >>> > >>> SolrConfig.xml: > >>> > >>> > >>> <indexDefaults> > >>> <useCompoundFile>false</useCompoundFile> > >>> <mergeFactor>10</mergeFactor> > >>> <maxMergeDocs>2147483647</maxMergeDocs> > >>> <maxFieldLength>10000</maxFieldLength--> > >>> <ramBufferSizeMB>4096</ramBufferSizeMB> > >>> <maxThreadStates>10</maxThreadStates> > >>> <writeLockTimeout>1000</writeLockTimeout> > >>> <commitLockTimeout>10000</commitLockTimeout> > >>> <lockType>single</lockType> > >>> > >>> <mergePolicy > class="org.apache.lucene.index.TieredMergePolicy"> > >>> <double name="forceMergeDeletesPctAllowed">0.0</double> > >>> <double name="reclaimDeletesWeight">10.0</double> > >>> </mergePolicy> > >>> > >>> <deletionPolicy class="solr.SolrDeletionPolicy"> > >>> <str name="keepOptimizedOnly">false</str> > >>> <str name="maxCommitsToKeep">0</str> > >>> </deletionPolicy> > >>> > >>> </indexDefaults> > >>> > >>> > >>> <updateHandler class="solr.DirectUpdateHandler2"> > >>> <maxPendingDeletes>1000</maxPendingDeletes> > >>> <autoCommit> > >>> <maxTime>900000</maxTime> > >>> <openSearcher>false</openSearcher> > >>> </autoCommit> > >>> <autoSoftCommit> > >>> > >> <maxTime>${inventory.solr.softcommit.duration:1000}</maxTime> > >>> </autoSoftCommit> > >>> > >>> </updateHandler> > >>> > >>> > >>> Forwarded conversation > >>> Subject: Large Index and OutOfMemoryError: Map failed > >>> ------------------------ > >>> > >>> From: Gopal Patwa <gopalpa...@gmail.com> > >>> Date: Fri, Mar 30, 2012 at 10:26 PM > >>> To: solr-user@lucene.apache.org > >>> > >>> > >>> I need help!! > >>> > >>> > >>> > >>> > >>> > >>> I am using Solr 4.0 nightly build with NRT and I often get this error > >> during > >>> auto commit "java.lang.OutOfMemoryError: Map failed". I have search > this > >>> forum and what I found it is related to OS ulimit setting, please se > >> below > >>> my ulimit settings. I am not sure what ulimit setting I should have? > and > >> we > >>> also get "java.net.SocketException: Too many open files" NOT sure how > >> many > >>> open file we need to set? > >>> > >>> > >>> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 - > >> 15GB, > >>> with Single shard > >>> > >>> > >>> We update the index every 5 seconds, soft commit every 1 second and > hard > >>> commit every 15 minutes > >>> > >>> > >>> Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB > >>> > >>> > >>> ulimit: > >>> > >>> core file size (blocks, -c) 0 > >>> data seg size (kbytes, -d) unlimited > >>> scheduling priority (-e) 0 > >>> file size (blocks, -f) unlimited > >>> pending signals (-i) 401408 > >>> max locked memory (kbytes, -l) 1024 > >>> max memory size (kbytes, -m) unlimited > >>> open files (-n) 1024 > >>> pipe size (512 bytes, -p) 8 > >>> POSIX message queues (bytes, -q) 819200 > >>> real-time priority (-r) 0 > >>> stack size (kbytes, -s) 10240 > >>> cpu time (seconds, -t) unlimited > >>> max user processes (-u) 401408 > >>> virtual memory (kbytes, -v) unlimited > >>> file locks (-x) unlimited > >>> > >>> > >>> > >>> ERROR: > >>> > >>> > >>> > >>> > >>> > >>> 2012-03-29 15:14:08,560 [] priority=ERROR app_name= > >> thread=pool-3-thread-1 > >>> location=CommitTracker line=93 auto commit > error...:java.io.IOException: > >> Map > >>> failed > >>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) > >>> at > >>> > >> > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293) > >>> at > >> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221) > >>> at > >>> > >> > org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.<init>(Lucene40PostingsReader.java:58) > >>> at > >>> > >> > org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer(Lucene40PostingsFormat.java:80) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat(PerFieldPostingsFormat.java:189) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:280) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.<init>(PerFieldPostingsFormat.java:186) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:186) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:256) > >>> at > >>> > >> > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:108) > >>> at > >> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:51) > >>> at > >>> > >> > org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader(IndexWriter.java:494) > >>> at > >>> > >> > org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:214) > >>> at > >>> > >> > org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:2939) > >>> at > >>> > >> > org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:2930) > >>> at > >> org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2681) > >>> at > >>> > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2804) > >>> at > >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2786) > >>> at > >>> > >> > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:391) > >>> at > org.apache.solr.update.CommitTracker.run(CommitTracker.java:197) > >>> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >>> at > >>> > >>> ... > >>> > >>> [Message clipped] > >>> ---------- > >>> From: Michael McCandless <luc...@mikemccandless.com> > >>> Date: Sat, Mar 31, 2012 at 3:15 AM > >>> To: solr-user@lucene.apache.org > >>> > >>> > >>> It's the virtual memory limit that matters; yours says unlimited below > >>> (good!), but, are you certain that's really the limit your Solr > >>> process runs with? > >>> > >>> On Linux, there is also a per-process map count: > >>> > >>> cat /proc/sys/vm/max_map_count > >>> > >>> I think it typically defaults to 65,536 but you should check on your > >>> env. If a process tries to map more than this many regions, you'll > >>> hit that exception. > >>> > >>> I think you can: > >>> > >>> cat /proc/<pid>/maps | wc > >>> > >>> to see how many maps your Solr process currently has... if that is > >>> anywhere near the limit then it could be the cause. > >>> > >>> Mike McCandless > >>> > >>> http://blog.mikemccandless.com > >>> > >>> On Sat, Mar 31, 2012 at 1:26 AM, Gopal Patwa <gopalpa...@gmail.com> > >> wrote: > >>>> *I need help!!* > >>>> > >>>> * > >>>> * > >>>> > >>>> *I am using Solr 4.0 nightly build with NRT and I often get this error > >>>> during auto commit "**java.lang.OutOfMemoryError:* *Map* *failed". I > >>>> have search this forum and what I found it is related to OS ulimit > >>>> setting, please se below my ulimit settings. I am not sure what ulimit > >>>> setting I should have? and we also get "**java.net.SocketException:* > >>>> *Too* *many* *open* *files" NOT sure how many open file we need to > >>>> set?* > >>>> > >>>> > >>>> I have 3 core with index size : core1 - 70GB, Core2 - 50GB and Core3 - > >>>> 15GB, with Single shard > >>>> > >>>> * > >>>> * > >>>> > >>>> *We update the index every 5 seconds, soft commit every 1 second and > >>>> hard commit every 15 minutes* > >>>> > >>>> * > >>>> * > >>>> > >>>> *Environment: Jboss 4.2, JDK 1.6 , CentOS, JVM Heap Size = 24GB* > >>>> > >>>> * > >>>> * > >>>> > >>>> ulimit: > >>>> > >>>> core file size (blocks, -c) 0 > >>>> data seg size (kbytes, -d) unlimited > >>>> scheduling priority (-e) 0 > >>>> file size (blocks, -f) unlimited > >>>> pending signals (-i) 401408 > >>>> max locked memory (kbytes, -l) 1024 > >>>> max memory size (kbytes, -m) unlimited > >>>> open files (-n) 1024 > >>>> pipe size (512 bytes, -p) 8 > >>>> POSIX message queues (bytes, -q) 819200 > >>>> real-time priority (-r) 0 > >>>> stack size (kbytes, -s) 10240 > >>>> cpu time (seconds, -t) unlimited > >>>> max user processes (-u) 401408 > >>>> virtual memory (kbytes, -v) unlimited > >>>> file locks (-x) unlimited > >>>> > >>>> > >>>> * > >>>> * > >>>> > >>>> *ERROR:* > >>>> > >>>> * > >>>> * > >>>> > >>>> *2012-03-29* *15:14:08*,*560* [] *priority=ERROR* *app_name=* > >>>> *thread=pool-3-thread-1* *location=CommitTracker* *line=93* *auto* > >>>> *commit* *error...:java.io.IOException:* *Map* *failed* > >>>> *at* > *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:748*) > >>>> *at* > >>>> > >> > *org.apache.lucene.store.MMapDirectory$MMapIndexInput.*<*init*>(*MMapDirectory.java:293*) > >>>> *at* > >>>> > >> > *org.apache.lucene.store.MMapDirectory.openInput*(*MMapDirectory.java:221*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.lucene40.Lucene40PostingsReader.*<*init*>(*Lucene40PostingsReader.java:58*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.lucene40.Lucene40PostingsFormat.fieldsProducer*(*Lucene40PostingsFormat.java:80*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.visitOneFormat*(*PerFieldPostingsFormat.java:189*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.*<*init*>(*PerFieldPostingsFormat.java:280*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader$1.*<*init*>(*PerFieldPostingsFormat.java:186*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.*<*init*>(*PerFieldPostingsFormat.java:186*) > >>>> *at* > >>>> > >> > *org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer*(*PerFieldPostingsFormat.java:256*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.SegmentCoreReaders.*<*init*>(*SegmentCoreReaders.java:108*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.SegmentReader.*<*init*>(*SegmentReader.java:51*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.IndexWriter$ReadersAndLiveDocs.getReader*(*IndexWriter.java:494*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.BufferedDeletesStream.applyDeletes*(*BufferedDeletesStream.java:214*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.IndexWriter.applyAllDeletes*(*IndexWriter.java:2939*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.IndexWriter.maybeApplyDeletes*(*IndexWriter.java:2930*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.IndexWriter.prepareCommit*(*IndexWriter.java:2681*) > >>>> *at* > >>>> > >> > *org.apache.lucene.index.IndexWriter.commitInternal*(*IndexWriter.java:2804*) > >>>> *at* > >>>> *org.apache.lucene.index.IndexWriter.commit*(*IndexWriter.java:2786*) > >>>> *at* > >>>> > >> > *org.apache.solr.update.DirectUpdateHandler2.commit*(*DirectUpdateHandler2.java:391*) > >>>> *at* > >>>> *org.apache.solr.update.CommitTracker.run*(*CommitTracker.java:197*) > >>>> *at* > >>>> > >> > *java.util.concurrent.Executors$RunnableAdapter.call*(*Executors.java:441*) > >>>> *at* > >>>> *java.util.concurrent.FutureTask$Sync.innerRun*(*FutureTask.java:303*) > >>>> *at* > *java.util.concurrent.FutureTask.run*(*FutureTask.java:138*) > >>>> *at* > >>>> > >> > *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301*(*ScheduledThreadPoolExecutor.java:98*) > >>>> *at* > >>>> > >> > *java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run*(*ScheduledThreadPoolExecutor.java:206*) > >>>> *at* > >>>> > >> > *java.util.concurrent.ThreadPoolExecutor$Worker.runTask*(*ThreadPoolExecutor.java:886*) > >>>> *at* > >>>> > >> > *java.util.concurrent.ThreadPoolExecutor$Worker.run*(*ThreadPoolExecutor.java:908*) > >>>> *at* *java.lang.Thread.run*(*Thread.java:662*)*Caused* *by:* > >>>> *java.lang.OutOfMemoryError:* *Map* *failed* > >>>> *at* *sun.nio.ch.FileChannelImpl.map0*(*Native* *Method*) > >>>> *at* > *sun.nio.ch.FileChannelImpl.map*(*FileChannelImpl.java:745*) > >>>> *...* *28* *more* > >>>> > >>>> * > >>>> * > >>>> > >>>> * > >>>> * > >>>> > >>>> * > >>>> > >>>> > >>>> SolrConfig.xml: > >>>> > >>>> > >>>> <indexDefaults> > >>>> <useCompoundFile>false</useCompoundFile> > >>>> <mergeFactor>10</mergeFactor> > >>>> <maxMergeDocs>2147483647</maxMergeDocs> > >>>> <maxFieldLength>10000</maxFieldLength--> > >>>> <ramBufferSizeMB>4096</ramBufferSizeMB> > >>>> <maxThreadStates>10</maxThreadStates> > >>>> <writeLockTimeout>1000</writeLockTimeout> > >>>> <commitLockTimeout>10000</commitLockTimeout> > >>>> <lockType>single</lockType> > >>>> > >>>> <mergePolicy > >> class="org.apache.lucene.index.TieredMergePolicy"> > >>>> <double name="forceMergeDeletesPctAllowed">0.0</double> > >>>> <double name="reclaimDeletesWeight">10.0</double> > >>>> </mergePolicy> > >>>> > >>>> <deletionPolicy class="solr.SolrDeletionPolicy"> > >>>> <str name="keepOptimizedOnly">false</str> > >>>> <str name="maxCommitsToKeep">0</str> > >>>> </deletionPolicy> > >>>> > >>>> </indexDefaults> > >>>> > >>>> > >>>> <updateHandler class="solr.DirectUpdateHandler2"> > >>>> <maxPendingDeletes>1000</maxPendingDeletes> > >>>> <autoCommit> > >>>> <maxTime>900000</maxTime> > >>>> <openSearcher>false</openSearcher> > >>>> </autoCommit> > >>>> <autoSoftCommit> > >>>> > >>>> <maxTime>${inventory.solr.softcommit.duration:1000}</maxTime> > >>>> </autoSoftCommit> > >>>> > >>>> </updateHandler> > >>>> > >>>> > >>>> > >>>> Thanks > >>>> Gopal Patwa > >>>> * > >>> > >>> ---------- > >>> From: Gopal Patwa <gopalpa...@gmail.com> > >>> Date: Tue, Apr 10, 2012 at 8:35 PM > >>> To: solr-user@lucene.apache.org > >>> > >>> > >>> Michael, Thanks for response > >>> > >>> it was 65K as you mention the default value for "cat > >>> /proc/sys/vm/max_map_count" . How we determine what value this should > be? > >>> is it number of document during hard commit in my case it is 15 > >> minutes? or > >>> it is number of index file or number of documents we have in all > cores. > >>> > >>> I have raised the number to 140K but I still get when it reaches to > >> 140K, we > >>> have to restart jboss server to free up the map count, sometime OOM > error > >>> happen during "Error opening new searcher" > >>> > >>> is making this number to unlimited is only solution?'' > >>> > >>> > >>> Error log: > >>> > >>> location=CommitTracker line=93 auto commit > >>> error...:org.apache.solr.common.SolrException: Error opening new > searcher > >>> at > >> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138) > >>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251) > >>> at > >>> > >> > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409) > >>> at > org.apache.solr.update.CommitTracker.run(CommitTracker.java:197) > >>> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >>> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>> at > >>> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) > >>> at > >>> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) > >>> at > >>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>> at > >>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>> at java.lang.Thread.run(Thread.java:662) > >>> Caused by: java.io.IOException: Map failed > >>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) > >>> at > >>> > >> > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293) > >>> at > >> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:262) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.<init>(PerFieldPostingsFormat.java:316) > >>> at > >>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316) > >>> at org.apache.lucene.codecs.Codec.files(Codec.java:56) > >>> at org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423) > >>> at > >> org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215) > >>> at > >>> > >> > org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220) > >>> at > >>> > >> > org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497) > >>> at > >>> > >> > org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477) > >>> at > >>> > >> > org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201) > >>> at > >>> > >> > org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119) > >>> at > >>> > >> > org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148) > >>> at org.apache.lucene.index. > >>> > >>> ... > >>> > >>> [Message clipped] > >>> ---------- > >>> From: Michael McCandless <luc...@mikemccandless.com> > >>> Date: Wed, Apr 11, 2012 at 2:20 AM > >>> To: solr-user@lucene.apache.org > >>> > >>> > >>> Hi, > >>> > >>> 65K is already a very large number and should have been sufficient... > >>> > >>> However: have you increased the merge factor? Doing so increases the > >>> open files (maps) required. > >>> > >>> Have you disabled compound file format? (Hmmm: I think Solr does so > >>> by default... which is dangerous). Maybe try enabling compound file > >>> format? > >>> > >>> Can you "ls -l" your index dir and post the results? > >>> > >>> It's also possible Solr isn't closing the old searchers quickly enough > >>> ... I don't know the details on when Solr closes old searchers... > >>> > >>> Mike McCandless > >>> > >>> http://blog.mikemccandless.com > >>> > >>> > >>> > >>> On Tue, Apr 10, 2012 at 11:35 PM, Gopal Patwa <gopalpa...@gmail.com> > >> wrote: > >>>> Michael, Thanks for response > >>>> > >>>> it was 65K as you mention the default value for "cat > >>>> /proc/sys/vm/max_map_count" . How we determine what value this should > >> be? > >>>> is it number of document during hard commit in my case it is 15 > >> minutes? > >>>> or it is number of index file or number of documents we have in all > >>>> cores. > >>>> > >>>> I have raised the number to 140K but I still get when it reaches to > >> 140K, > >>>> we have to restart jboss server to free up the map count, sometime OOM > >>>> error happen during "*Error opening new searcher"* > >>>> > >>>> is making this number to unlimited is only solution?'' > >>>> > >>>> > >>>> Error log: > >>>> > >>>> *location=CommitTracker line=93 auto commit > >>>> error...:org.apache.solr.common.SolrException: Error opening new > >>>> searcher > >>>> at > >>>> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1138) > >>>> at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1251) > >>>> at > >>>> > >> > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:409) > >>>> at > >> org.apache.solr.update.CommitTracker.run(CommitTracker.java:197) > >>>> at > >>>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >>>> at > >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>>> at > >>>> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98) > >>>> at > >>>> > >> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206) > >>>> at > >>>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >>>> at > >>>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >>>> at java.lang.Thread.run(Thread.java:662)Caused by: > >>>> java.io.IOException: Map failed > >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) > >>>> at > >>>> > >> > org.apache.lucene.store.MMapDirectory$MMapIndexInput.<init>(MMapDirectory.java:293) > >>>> at > >>>> > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:221) > >>>> at > >>>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$VisitPerFieldFile.<init>(PerFieldPostingsFormat.java:262) > >>>> at > >>>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$1.<init>(PerFieldPostingsFormat.java:316) > >>>> at > >>>> > >> > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.files(PerFieldPostingsFormat.java:316) > >>>> at org.apache.lucene.codecs.Codec.files(Codec.java:56) > >>>> at > >> org.apache.lucene.index.SegmentInfo.files(SegmentInfo.java:423) > >>>> at > >>>> org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:215) > >>>> at > >>>> > >> > org.apache.lucene.index.IndexWriter.prepareFlushedSegment(IndexWriter.java:2220) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriter.publishFlushedSegment(DocumentsWriter.java:497) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriter.finishFlush(DocumentsWriter.java:477) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:201) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:119) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriterFlushQueue.tryPurge(DocumentsWriterFlushQueue.java:148) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:438) > >>>> at > >>>> > >> > org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:553) > >>>> at > >>>> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:354) > >>>> at > >>>> > >> > org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:258) > >>>> at > >>>> > >> > org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:243) > >>>> at > >>>> > >> > org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:250) > >>>> at > >>>> org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1091) > >>>> ... 11 moreCaused by: java.lang.OutOfMemoryError: Map failed > >>>> at sun.nio.ch.FileChannelImpl.map0(Native Method) > >>>> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745)* > >>>> > >>>> > >>>> > >>>> And one more issue we came across i.e > >>> > >>> > >> > > > > > > ______________________________________________________________________ > > This email has been scanned by the brightsolid Email Security System. > Powered by MessageLabs > > ______________________________________________________________________ > > > ______________________________________________________________________ > "brightsolid" is used in this email to collectively mean brightsolid > online innovation limited and its subsidiary companies brightsolid online > publishing limited and brightsolid online technology limited. > findmypast.co.uk is a brand of brightsolid online publishing limited. > brightsolid online innovation limited, Gateway House, Luna Place, Dundee > Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC274983. > brightsolid online publishing limited, The Glebe, 6 Chapel Place, > Rivington Street, London EC2A 3DQ. Registered in England No. 04369607. > brightsolid online technology limited, Gateway House, Luna Place, Dundee > Technology Park, Dundee DD2 1TP. Registered in Scotland No. SC161678. > > Email Disclaimer > > This message is confidential and may contain privileged information. You > should not disclose its contents to any other person. If you are not the > intended recipient, please notify the sender named above immediately. It is > expressly declared that this e-mail does not constitute nor form part of a > contract or unilateral obligation. Opinions, conclusions and other > information in this message that do not relate to the official business of > brightsolid shall be understood as neither given nor endorsed by it. > ______________________________________________________________________ > This email has been scanned by the brightsolid Email Security System. > Powered by MessageLabs > ______________________________________________________________________ >
