[
https://issues.apache.org/jira/browse/HBASE-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Kumar Kumawat updated HBASE-29141:
----------------------------------------
Description:
Regarding the handler and queues HBase have these config.
* {{hbase.regionserver.handler.count}} => Number of handlers, Default is 30.
* {{hbase.ipc.server.callqueue.handler.factor}} => Queue-to-handler ratio.
Default it is 0.1. Means 10 handler per queue.
** {{number of call queues}} = {{handlerCount}} *
{{callQueuesHandlersFactor(0.1)}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195])
* {{hbase.ipc.server.max.callqueue.length}} => Max callqueue Length. Default
is {{DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) * {color:#FF0000}*total
handlerCount*{color}}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71])
** {*}Issue{*}: I think there’s a small issue here. We are currently treating
this as the length per queue, but it's calculated using the {_}total handler
count{_}, not the number of handlers per queue.
* In addition we also have a hard limit of
({{{}DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT{}}}) 250
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228])
this is hard limit on *minimum* value of queue size. Queue size can not be
less then 250.
# I believe we may have unintentionally created a dependency between the
*total number of handlers* and the call queue length. Should we reconsider this
logic?
# Regarding the hard limit: should we be enforcing a _minimum_ value for the
queue size or a _maximum_ value for the queue size ?
# In codel queue type we are we have a config
{{hbase.ipc.server.callqueue.codel.lifo.threshold}} (default 0.8). Having 100
handlers, by default all the 10 queues will be of size 1000 and queue
processing will not change to lifo unless there are 800 calls in a single
queue.
was:
By default there is quadratic relation between total number of handler and
total number of calls in general queues.
Regarding the handler and queues HBase have these config. *
{{hbase.regionserver.handler.count}} => Number of handlers, Default is 30.
* {{hbase.ipc.server.callqueue.handler.factor}} => Queue-to-handler ratio.
Default it is 0.1. Means 10 handler per queue.
** {{number of call queues}} = {{handlerCount}} *
{{callQueuesHandlersFactor(0.1)}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195])
* {{hbase.ipc.server.max.callqueue.length}} => Max callqueue Length. Default
is {{DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) * total handlerCount}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71])
** {*}Issue{*}: I think there’s a small issue here. We are currently treating
this as the length per queue, but it's calculated using the {_}total handler
count{_}, not the number of handlers per queue.
* In addition we also have a hard limit of
({{{}DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT{}}}) 250
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228])
this is hard limit on *minimum* value of queue size. Queue size can not be
less then 250.
{{}}
{{}} # I believe we may have unintentionally created a dependency between the
*total number of handlers* and the call queue length. Should we reconsider this
logic?
# Regarding the hard limit: should we be enforcing a _minimum_ value for the
queue size or a _maximum_ value for the queue size ?
# In codel queue type we are we have a config
{{hbase.ipc.server.callqueue.codel.lifo.threshold}} (default 0.8). Having 100
handlers, by default all the 10 queues will be of size 1000 and queue
processing will not change to lifo unless there are 800 calls in a single
queue.
> Default Size of call queues is too high
> ---------------------------------------
>
> Key: HBASE-29141
> URL: https://issues.apache.org/jira/browse/HBASE-29141
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 3.0.0-beta-1, 2.5.11, 2.6.2
> Reporter: Umesh Kumar Kumawat
> Assignee: Umesh Kumar Kumawat
> Priority: Major
>
> Regarding the handler and queues HBase have these config.
> * {{hbase.regionserver.handler.count}} => Number of handlers, Default is 30.
> * {{hbase.ipc.server.callqueue.handler.factor}} => Queue-to-handler ratio.
> Default it is 0.1. Means 10 handler per queue.
> ** {{number of call queues}} = {{handlerCount}} *
> {{callQueuesHandlersFactor(0.1)}}
> ([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195])
> * {{hbase.ipc.server.max.callqueue.length}} => Max callqueue Length.
> Default is {{DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) *
> {color:#FF0000}*total handlerCount*{color}}}
> ([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71])
> ** {*}Issue{*}: I think there’s a small issue here. We are currently
> treating this as the length per queue, but it's calculated using the {_}total
> handler count{_}, not the number of handlers per queue.
> * In addition we also have a hard limit of
> ({{{}DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT{}}}) 250
> ([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228])
> this is hard limit on *minimum* value of queue size. Queue size can not be
> less then 250.
>
>
> # I believe we may have unintentionally created a dependency between the
> *total number of handlers* and the call queue length. Should we reconsider
> this logic?
> # Regarding the hard limit: should we be enforcing a _minimum_ value for the
> queue size or a _maximum_ value for the queue size ?
> # In codel queue type we are we have a config
> {{hbase.ipc.server.callqueue.codel.lifo.threshold}} (default 0.8). Having 100
> handlers, by default all the 10 queues will be of size 1000 and queue
> processing will not change to lifo unless there are 800 calls in a single
> queue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
