yaser-aj opened a new pull request, #14953:
URL: https://github.com/apache/lucene/pull/14953
A proposed solution to #13883.
Currently, each `ConcurrentMergeScheduler` manages its own threads
independently, assuming no other merge schedulers are running in the JVM. This
leads to potential over-allocation of system resources when many `IndexWriter`s
are active. This draft PR demonstrates a simple approach to multi-tenancy by
introducing a global manager that divides a global thread budget among all
registered CMS instances.
### Key Changes
- New Class: `MultiTenantCMSManager`
- Singleton manager that tracks all active `ConcurrentMergeScheduler`
instances.
- Divides a global thread budget (based on available CPU cores) equally
among all registered CMSs.
- Dynamically updates each CMS’s `maxThreadCount` and `maxMergeCount` as
CMSs are registered or unregistered.
- Includes testing hooks for registration and budget management.
- Integration with `ConcurrentMergeScheduler`
- CMS now registers itself with the manager upon construction and
unregisters on close.
- CMS thread/merge limits are automatically managed by the global
manager.
- Tests
- Added unit tests for registration, dynamic rebalancing, live
`IndexWriter` integration, and auto-unregistering behavior.
### Important Notes
- This is a demonstration/first step toward multi-tenant merge scheduling.
The logic can be extended to support more advanced scheduling and dynamic
resource allocation in the future.
- The manager currently uses a simple equal-share policy for thread
allocation (total thread count divided by merge scheduler count).
Will update this draft PR with more points + to-dos soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
