[
https://issues.apache.org/jira/browse/IMPALA-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-13164:
-----------------------------------
Fix Version/s: Impala 5.0.0
> Add metrics for RPC reactor and deserialization threads
> -------------------------------------------------------
>
> Key: IMPALA-13164
> URL: https://issues.apache.org/jira/browse/IMPALA-13164
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Distributed Exec
> Affects Versions: Impala 4.4.0
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Critical
> Fix For: Impala 5.0.0
>
>
> When Impala starts to build up a queue in an RPC service - usually
> DataStreamService - they become a bottleneck that slows down all exchanges
> going through that node. To identify what's happening we generally need to
> collect pstacks to identify why the queue is backed up.
> If all KrpcDataStreamMgr deserialization threads are in use, it can help to
> increase the number of threads available via
> {{{}datastream_service_num_deserialization_threads{}}}. This is the most
> common bottleneck on larger machines; DataStream packets can be large and
> take time to deserialize.
> Less frequently the number of reactor threads in RpcMgr sending requests and
> receiving responses (so handling the actual network I/O) might be full; this
> is much less common because {{num_reactor_threads}} defaults to the number of
> CPU cores.
> ControlService uses a single thread pool for network I/O and deserialization
> because the packets tend to be much smaller; this is controlled by
> {{{}control_service_num_svc_threads{}}}, which defaults to the number of CPU
> cores as well.
> It would help to have metrics for RPC reactor and deserialization threads to
> tell when they're fully loaded and building up a queue; this could identify
> when we need to increase {{datastream_service_num_deserialization_threads}}
> or {{{}num_reactor_threads{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
