J-HowHuang opened a new pull request, #17424:
URL: https://github.com/apache/pinot/pull/17424
## Description
Currently Pinot server uses the default Helix thread pool for state
transition messages (appears as
`HelixTaskExecutor-message_handle_thread_<tid>`). The main concern of this is
that
1. Every transition regardless of the states uses the same pool. New
consuming segments (OFFLINE->ONLINE) could be blocked by operations on other
tables (e.g. reset, rebalance) because they flood the message queue.
2. It's not possible to manage the threads within the scope of Pinot. For
example, change max threads.
## Change
* Add a new class `ServerThreadPoolManager` that constructs the thread pools
for consuming transitions and other transitions
* Override `ExecutorService getExecutorService(String resourceName, String
fromState, String toState)` in `SegmentOnlineOfflineStateModelFactory` to
return the thread pools created by `ServerThreadPoolManager`
* Add new configs
* `pinot.server.helix.transition.thread.pool.size`, default to `40`
* `pinot.server.helix.transition.consuming.thread.pool.size`, default to
`10`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
