vigyasharma commented on issue #13883: URL: https://github.com/apache/lucene/issues/13883#issuecomment-2982081708
Thanks @yaser-aj , happy to see progress on this project. You're on the right track with understanding the problem. We want CMS to be aware of merge demands across IndexWriters, so that it can allocate/throttle merge threads more optimally. __ > There has to be one MultiTenantConcurrentMergeScheduler object that organizes how all ConcurrentMergeScheduler objects operate and divide resources wisely across them. It should handle addition and deletion of ConcurrentMergeScheduler objects on the go, optimally without the need to restart all ConcurrentMergeScheduler objects every time the number of ConcurrentMergeScheduler objects changes. This sounds like a CMS "Manager" that manages multiple other merge schedulers. It might be more effective to change the merge scheduler itself to be multi-tenant. This `MultiTenantConcurrentMergeScheduler` will consider all the active merges across all index writers whenever merges need to be throttled (see `ConcurrentMergeScheduler#maybeStall`, `ConcurrentMergeScheduler#updateIOThrottle`, `ConcurrentMergeScheduler#updateMergeThreads` etc). Internally, it would maintain some mapping of index writers -> merges to cleanly handle close for a single index writer without affecting merges for other writers. This would give you more direct control over scheduling merges from across index writers. We could then maintain a singleton for this "multiTenantCMS". Index writers would acquire an instance of that singleton and register themselves with it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
