mikemccand commented on PR #14964: URL: https://github.com/apache/lucene/pull/14964#issuecomment-3159879637
> I wonder if we're addressing the original issue. Maybe I need to spend more time reading though it, but my understanding is that `MergeRateLimiter`, which we use here, is meant to control the amount of resources we put into the merging process, while the idea we were considering was to limit the size of the output of merges. Please correct me if I got that wrong. I like this simplicity of this approach, vs the other ideas on https://github.com/apache/lucene/issues/14148 ... but I do agree that having the MP and/or MS aware that they are operating in a resource-scarce environment might do better. E.g. TMP might delay picking merges if it's already spent its short-term IO budget, and be forced to allow more deletions to accumulate. But with this simplistic approach, we might emergently see similar behavior. Throttling a lerge merge that is trying to reclaim deletes effectively stretches out the lifetime of the segments being merged, allowing more deletions to accumulate that would normally, and delaying the appearance of the newly merged segment so that MP/MS cannot pick it for merging until itself is done being merged. We need to test this approach to see... @nipunbatra8 `RateLimiter` (as hairy as it is!!) might be overly simplistic for your usage here, especially if we pivot the approach to share a single rate limiter across all merge threads ... it does not implement the "[leaky bucket](https://en.wikipedia.org/wiki/Leaky_bucket)" model to smooth out bursty IO. This is another reason to not pivot to this new approach just yet ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
