[
https://issues.apache.org/jira/browse/LUCENE-9751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17281874#comment-17281874
]
Michael McCandless commented on LUCENE-9751:
--------------------------------------------
Thanks [~dweiss], clearly we have a bug here! We need better testing of
"large" indexing RAM buffers.
This is the assert that tripped I think:
{noformat}
public void init(ByteBlockPool pool, int startIndex, int endIndex) {
assert endIndex-startIndex >= 0; {noformat}
So I think most likely the {{endIndex}} overflowed int and became negative
elsewhere.
We do know that our "best effort" will fail to catch indexing a gigantic
document that pushes the indexing buffer over 2.1 GB. We default to 1945 MB
cutoff ({{IWC.setRAMPerThreadhardLimitMB}} can be used to change that), but if
that one gigantic document takes the RAM usage from e.g. 1944 MB up beyond 2048
MB then it can lead to exceptions like this.
> Assertion error (int overflow) in ByteSliceReader
> -------------------------------------------------
>
> Key: LUCENE-9751
> URL: https://issues.apache.org/jira/browse/LUCENE-9751
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 8.7
> Reporter: Dawid Weiss
> Priority: Major
>
> New computers come with insane amounts of ram and heaps can get pretty big.
> If you adjust per-thread buffers to larger values strange things start
> happening. This happened to us today:
> {code}
> Caused by: java.lang.AssertionError
> at
> org.apache.lucene.index.ByteSliceReader.init(ByteSliceReader.java:44)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.TermsHashPerField.initReader(TermsHashPerField.java:88)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.FreqProxFields$FreqProxPostingsEnum.reset(FreqProxFields.java:430)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.FreqProxFields$FreqProxTermsEnum.postings(FreqProxFields.java:247)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:127)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:907)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:318)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:264)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:480)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:394)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:440)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
> ~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
> atrisharma - 2020-10-29 19:35:28]
> ... 7 more
> {code}
> Likely an int overflow in TermsHashPerField:
> {code}
> reader.init(bytePool,
>
> postingsArray.byteStarts[termID]+stream*ByteBlockPool.FIRST_LEVEL_SIZE,
> streamAddressBuffer[offsetInAddressBuffer+stream]);
> {code}
> Don't know if this can be prevented somehow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
