[
https://issues.apache.org/jira/browse/HDFS-17959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Arshad updated HDFS-17959:
-----------------------------------
Description:
Hive query execution was blocked on HiveServer2 due to a race condition in the
HDFS client.
*Observation:*
I collected and analyzed a thread dump from HiveServer2 (attached to this
JIRA). The analysis showed that multiple threads were adding items to the
rpcRequestQueue, but no thread was polling from it. As a result, the threads
adding items could not proceed and remained blocked on the
SynchronousQueue.offer() method, which in turn blocked Hive query execution.
*Thread Dump Analysis Details:*
Multiple threads were stuck on queue 0x00007ef98c380800, but no IPC Parameter
Sending Thread (RpcRequestSender) exists to poll this queue.
The four sender threads present in the dump poll four different queues, all
recently created (13–33 seconds old).
The stuck queue 0x00007ef98c380800 belongs to an older Connection whose sender
thread is dead (a zombie queue).
The blocked threads are stuck in the following code path:
{code:java}
at
java.util.concurrent.SynchronousQueue.offer([email protected]/SynchronousQueue.java:857)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1187)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
{code}
Multiple thread dumps were collected at 5-minute intervals over a period of
roughly half an hour. In each dump, the same threads remained blocked on the
SynchronousQueue.offer() method, confirming that Hive query execution made no
progress throughout this period.
was:
Hive query execution was blocked on HiveServer2 due to a race condition in the
HDFS client.
*Observation:*
I collected and analyzed a thread dump from HiveServer2 (attached to this
JIRA). The analysis showed that multiple threads were adding items to the
rpcRequestQueue, but no thread was polling from it. As a result, the threads
adding items could not proceed and remained blocked on the
SynchronousQueue.offer() method, which in turn blocked Hive query execution.
*Thread Dump Analysis Details:*
Multiple threads were stuck on queue 0x00007ef98c380800, but no IPC Parameter
Sending Thread (RpcRequestSender) exists to poll this queue.
The four sender threads present in the dump poll four different queues, all
recently created (13–33 seconds old).
The stuck queue 0x00007ef98c380800 belongs to an older Connection whose sender
thread is dead (a zombie queue).
The blocked threads are stuck in the following code path:
{code:java}
at
java.util.concurrent.SynchronousQueue.offer([email protected]/SynchronousQueue.java:857)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1187)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
{code}
> Hive query execution hangs due to HDFS Client race condition
> ------------------------------------------------------------
>
> Key: HDFS-17959
> URL: https://issues.apache.org/jira/browse/HDFS-17959
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
> Affects Versions: 3.4.1
> Reporter: Mohammad Arshad
> Assignee: Mohammad Arshad
> Priority: Critical
> Attachments: host1_Hiver_Server2_Thread_Dump.txt
>
>
> Hive query execution was blocked on HiveServer2 due to a race condition in
> the HDFS client.
> *Observation:*
> I collected and analyzed a thread dump from HiveServer2 (attached to this
> JIRA). The analysis showed that multiple threads were adding items to the
> rpcRequestQueue, but no thread was polling from it. As a result, the threads
> adding items could not proceed and remained blocked on the
> SynchronousQueue.offer() method, which in turn blocked Hive query execution.
> *Thread Dump Analysis Details:*
> Multiple threads were stuck on queue 0x00007ef98c380800, but no IPC Parameter
> Sending Thread (RpcRequestSender) exists to poll this queue.
> The four sender threads present in the dump poll four different queues, all
> recently created (13–33 seconds old).
> The stuck queue 0x00007ef98c380800 belongs to an older Connection whose
> sender thread is dead (a zombie queue).
> The blocked threads are stuck in the following code path:
> {code:java}
> at
> java.util.concurrent.SynchronousQueue.offer([email protected]/SynchronousQueue.java:857)
> at
> org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1187)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> {code}
> Multiple thread dumps were collected at 5-minute intervals over a period of
> roughly half an hour. In each dump, the same threads remained blocked on the
> SynchronousQueue.offer() method, confirming that Hive query execution made no
> progress throughout this period.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
