[
https://issues.apache.org/jira/browse/HDFS-17959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Arshad updated HDFS-17959:
-----------------------------------
Summary: Hive query execution hangs due to HDFS Client race condition
(was: Hive query execution hangs due to HDFS Client race condition where RPC
threads block on a zombie connection's queue)
> Hive query execution hangs due to HDFS Client race condition
> ------------------------------------------------------------
>
> Key: HDFS-17959
> URL: https://issues.apache.org/jira/browse/HDFS-17959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 3.4.1
> Reporter: Mohammad Arshad
> Assignee: Mohammad Arshad
> Priority: Major
>
> Hive query execution was blocked on HiveServer2 due to a race condition in
> the HDFS client.
> *Observation:*
> I collected and analyzed a thread dump from HiveServer2 (attached to this
> JIRA). The analysis showed that multiple threads were adding items to the
> rpcRequestQueue, but no thread was polling from it. As a result, the threads
> adding items could not proceed and remained blocked on the
> SynchronousQueue.offer() method, which in turn blocked Hive query execution.
> *Thread Dump Analysis Details:*
> Multiple threads were stuck on queue 0x00007ef98c380800, but no IPC Parameter
> Sending Thread (RpcRequestSender) exists to poll this queue.
> The four sender threads present in the dump poll four different queues, all
> recently created (13–33 seconds old).
> The stuck queue 0x00007ef98c380800 belongs to an older Connection whose
> sender thread is dead (a zombie queue).
> The blocked threads are stuck in the following code path:
> {code:java}
> at
> java.util.concurrent.SynchronousQueue.offer([email protected]/SynchronousQueue.java:857)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1187)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
