[
https://issues.apache.org/jira/browse/LUCENE-9927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand resolved LUCENE-9927.
----------------------------------
Resolution: Won't Fix
+1 Robert
> Configurable BKDWriter maximum heap size
> ----------------------------------------
>
> Key: LUCENE-9927
> URL: https://issues.apache.org/jira/browse/LUCENE-9927
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Andras Salamon
> Priority: Major
>
> With Lucene's default codec, when writing dimensional points, we only give
> {{BKDWriter}} 16 MB heap to use for sorting. If we reach this limit the codec
> will change to
> [OfflinePointWriter|https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/bkd/OfflinePointWriter.java].
> This is specially bad for Solr versions not containing SOLR-14242, because we
> get the following error:
> {noformat}
> Caused by: java.lang.IllegalStateException: this writer hit an unrecoverable
> error; cannot complete forceMerge
> at
> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1998)
> at
> org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1940)
> at
> org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:662)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:102)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:68)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1079)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1066)
> at
> org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:160)
> at
> org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:72)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
> ... 15 more
> Caused by: java.lang.UnsupportedOperationException
> at
> org.apache.solr.store.hdfs.HdfsDirectory.createTempOutput(HdfsDirectory.java:119)
> at
> org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:79)
> at
> org.apache.lucene.store.FilterDirectory.createTempOutput(FilterDirectory.java:79)
> at
> org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:51)
> at
> org.apache.lucene.store.TrackingDirectoryWrapper.createTempOutput(TrackingDirectoryWrapper.java:51)
> at
> org.apache.lucene.util.bkd.OfflinePointWriter.<init>(OfflinePointWriter.java:45)
> at
> org.apache.lucene.util.bkd.BKDWriter.initPointWriter(BKDWriter.java:223)
> at org.apache.lucene.util.bkd.BKDWriter.add(BKDWriter.java:241)
> at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter$1.visit(Lucene60PointsWriter.java:123)
> at
> org.apache.lucene.codecs.PointsWriter$1$1$1.visit(PointsWriter.java:118)
> at
> org.apache.lucene.util.bkd.BKDReader.visitCompressedDocValues(BKDReader.java:735)
> at
> org.apache.lucene.util.bkd.BKDReader.visitDocValuesWithCardinality(BKDReader.java:684)
> at
> org.apache.lucene.util.bkd.BKDReader.visitDocValues(BKDReader.java:590)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:790)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:813)
> at org.apache.lucene.util.bkd.BKDReader.intersect(BKDReader.java:506)
> at
> org.apache.lucene.codecs.PointsWriter$1$1.intersect(PointsWriter.java:106)
> at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:116)
> at
> org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:63)
> at
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:227)
> at
> org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:206)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:164)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4467)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4059)
> at
> org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:201)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662){noformat}
> It would be great to make this 16MB value configurable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
