itschrispeck opened a new pull request, #13050: URL: https://github.com/apache/pinot/pull/13050
https://github.com/apache/pinot/pull/12744 changed the Lucene index build process to commit the realtime Lucene index, and reuse it in the immutable segment. `.commit()` is called at the beginning of segment conversion, and the Lucene index directory is copied to another location. Since `IndexWriter` can merge segments in the background even without insertions/updates to the index, these merges non-deterministically can trigger a `FileNotFoundException`: ``` java.lang.RuntimeException: Caught exception while instantiating the LuceneTextIndexCreator for column: __mergedTextIndex at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.<init>(LuceneTextIndexCreator.java:149) <...> Caused by: java.lang.RuntimeException: Failed to copy the mutable lucene index: java.io.FileNotFoundException: File system element for parameter 'source' does not exist: '/.../dataDir/table_REALTIME/consumers/table__0__3170__20240430T2313Z/col.lucene.v9.index/_3r.si' at org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator.convertMutableSegment(LuceneTextIndexCreator.java:192) ... ``` By using `NoMergeScheduler`, the async merges will never occur. This patch was tested in our staging/prod env and the intermittent segment build failures disappeared. tag: `bugfix` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
