[
https://issues.apache.org/jira/browse/HBASE-29141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Kumar Kumawat updated HBASE-29141:
----------------------------------------
Description:
Regarding the handler and queues HBase have these config.
* {{hbase.regionserver.handler.count}} => Number of handlers, Default is 30.
* {{hbase.ipc.server.callqueue.handler.factor}} => Queue-to-handler ratio.
Default it is 0.1. Means 10 handler per queue.
** {{number of call queues}} = {{handlerCount}} *
{{callQueuesHandlersFactor(0.1)}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195])
* {{hbase.ipc.server.max.callqueue.length}} => Max callqueue Length. Default
is {{DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) * {color:#ff0000}*total
handlerCount*{color}}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71])
** {*}Issue{*}: I think there’s a small issue here. We are currently treating
this as the length per queue, but it's calculated using the {_}total handler
count{_}, not the number of handlers per queue.
* In addition we also have a hard limit of
({{{}DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT{}}}) 250
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228])
this is hard limit on *minimum* value of queue size. Queue size can not be
less then 250.
In codel queue type, we have a config
{{hbase.ipc.server.callqueue.codel.lifo.threshold}} (default 0.8). Having 100
handlers, by default all the 10 queues will be of size 1000 and queue
processing will not change to lifo unless there are 800 calls waiitng in a
single queue. Don't you think it is too much.
# I believe we may have unintentionally created a dependency between the
*total number of handlers* and the call queue length. Should we reconsider this
logic?
# Regarding the hard limit: should we be enforcing a _minimum_ value for the
queue size or a _maximum_ value for the queue size ?
was:
Regarding the handler and queues HBase have these config.
* {{hbase.regionserver.handler.count}} => Number of handlers, Default is 30.
* {{hbase.ipc.server.callqueue.handler.factor}} => Queue-to-handler ratio.
Default it is 0.1. Means 10 handler per queue.
** {{number of call queues}} = {{handlerCount}} *
{{callQueuesHandlersFactor(0.1)}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195])
* {{hbase.ipc.server.max.callqueue.length}} => Max callqueue Length. Default
is {{DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) * {color:#FF0000}*total
handlerCount*{color}}}
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71])
** {*}Issue{*}: I think there’s a small issue here. We are currently treating
this as the length per queue, but it's calculated using the {_}total handler
count{_}, not the number of handlers per queue.
* In addition we also have a hard limit of
({{{}DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT{}}}) 250
([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228])
this is hard limit on *minimum* value of queue size. Queue size can not be
less then 250.
# I believe we may have unintentionally created a dependency between the
*total number of handlers* and the call queue length. Should we reconsider this
logic?
# Regarding the hard limit: should we be enforcing a _minimum_ value for the
queue size or a _maximum_ value for the queue size ?
# In codel queue type we are we have a config
{{hbase.ipc.server.callqueue.codel.lifo.threshold}} (default 0.8). Having 100
handlers, by default all the 10 queues will be of size 1000 and queue
processing will not change to lifo unless there are 800 calls in a single
queue.
> Default Size of call queues is too high
> ---------------------------------------
>
> Key: HBASE-29141
> URL: https://issues.apache.org/jira/browse/HBASE-29141
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
> Affects Versions: 3.0.0-beta-1, 2.5.11, 2.6.2
> Reporter: Umesh Kumar Kumawat
> Assignee: Umesh Kumar Kumawat
> Priority: Major
>
> Regarding the handler and queues HBase have these config.
> * {{hbase.regionserver.handler.count}} => Number of handlers, Default is 30.
> * {{hbase.ipc.server.callqueue.handler.factor}} => Queue-to-handler ratio.
> Default it is 0.1. Means 10 handler per queue.
> ** {{number of call queues}} = {{handlerCount}} *
> {{callQueuesHandlersFactor(0.1)}}
> ([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L195])
> * {{hbase.ipc.server.max.callqueue.length}} => Max callqueue Length.
> Default is {{DEFAULT_MAX_CALLQUEUE_LENGTH_PER_HANDLER(10) *
> {color:#ff0000}*total handlerCount*{color}}}
> ([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java#L71])
> ** {*}Issue{*}: I think there’s a small issue here. We are currently
> treating this as the length per queue, but it's calculated using the {_}total
> handler count{_}, not the number of handlers per queue.
> * In addition we also have a hard limit of
> ({{{}DEFAULT_CALL_QUEUE_SIZE_HARD_LIMIT{}}}) 250
> ([code|https://github.com/apache/hbase/blob/777010361abb203b8b17673d84acf4f7f1d0283a/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java#L228])
> this is hard limit on *minimum* value of queue size. Queue size can not be
> less then 250.
>
> In codel queue type, we have a config
> {{hbase.ipc.server.callqueue.codel.lifo.threshold}} (default 0.8). Having 100
> handlers, by default all the 10 queues will be of size 1000 and queue
> processing will not change to lifo unless there are 800 calls waiitng in a
> single queue. Don't you think it is too much.
>
> # I believe we may have unintentionally created a dependency between the
> *total number of handlers* and the call queue length. Should we reconsider
> this logic?
> # Regarding the hard limit: should we be enforcing a _minimum_ value for the
> queue size or a _maximum_ value for the queue size ?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
