[
https://issues.apache.org/jira/browse/LUCENE-10088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412399#comment-17412399
]
Simon Willnauer commented on LUCENE-10088:
------------------------------------------
Hey [~jpountz] I think your analysis is right and makes sense to me.
Unfortunately this is somewhat expected and an existing limitation of the
throttle mechanics in IW. I also agree it's an edge case and very unlikely to
happen anywhere outside of a randomized test. Due to the fact that we are doing
a full flush we can't finish flushing the buffered segments that pile up
caused by randomly set buffered docs. I think the TooManyOpenFilesExc is only
one possible issue we could hit here, OOM would be another. I think we can
improve the protection IW offers here and either add a non-aborting exception
that rejects the document before it gets our of hand with too many buffered
segments which would be a awkward to the user or we improve our stalling option
which is currently solely based on [how big our flush
buffer|https://github.com/apache/lucene/blob/ee0695eda8bc45611cf0312ea3eb6a8819537537/lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushControl.java#L283]
is. I think we can certainly find a good default that would prevent 600+
blocked segments without it having any impact in the real world. If I am not
mistaken I think we can simply change this one line
[here|https://github.com/apache/lucene/blob/ee0695eda8bc45611cf0312ea3eb6a8819537537/lucene/core/src/java/org/apache/lucene/index/DocumentsWriterFlushControl.java#L294]
to add some upper-bound to make it work.
> Too many open files in
> TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
> --------------------------------------------------------------------------------------------------
>
> Key: LUCENE-10088
> URL: https://issues.apache.org/jira/browse/LUCENE-10088
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Priority: Major
>
> [This build
> failure|https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/386/]
> reproduces for me. I'll try to dig.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
