Dawid Weiss created LUCENE-9751:
-----------------------------------
Summary: Assertion error (int overflow) in ByteSliceReader
Key: LUCENE-9751
URL: https://issues.apache.org/jira/browse/LUCENE-9751
Project: Lucene - Core
Issue Type: Bug
Affects Versions: 8.7
Reporter: Dawid Weiss
New computers come with insane amounts of ram and heaps can get pretty big. If
you adjust per-thread buffers to larger values strange things start happening.
This happened to us today:
{code}
Caused by: java.lang.AssertionError
at
org.apache.lucene.index.ByteSliceReader.init(ByteSliceReader.java:44)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.TermsHashPerField.initReader(TermsHashPerField.java:88)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.FreqProxFields$FreqProxPostingsEnum.reset(FreqProxFields.java:430)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.FreqProxFields$FreqProxTermsEnum.postings(FreqProxFields.java:247)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:127)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:907)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:318)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:170)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:120)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:264)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:480)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.DocumentsWriter.postUpdate(DocumentsWriter.java:394)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:440)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471)
~[lucene-core-8.7.0.jar:8.7.0 2dc63e901c60cda27ef3b744bc554f1481b3b067 -
atrisharma - 2020-10-29 19:35:28]
... 7 more
{code}
Likely an int overflow in TermsHashPerField:
{code}
reader.init(bytePool,
postingsArray.byteStarts[termID]+stream*ByteBlockPool.FIRST_LEVEL_SIZE,
streamAddressBuffer[offsetInAddressBuffer+stream]);
{code}
Don't know if this can be prevented somehow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
