This can happen if the underlying storage is wedged. I would check that
it is working properly.
Usually the only way to clear this state is either fix the stuck storage
or reboot the node.
-Paul Edmon-
On 10/24/2020 12:22 PM, Kimera Rodgers wrote:
I'm setting up slume on OpenHPC cluster with one master node and 5
compute nodes.
When I run test jobs, the jobs completely get stuck in the CG state.
Can someone help me hint on where I might have gone wrong?
[root@kla-ac-ohpc-01 critical]# srun -c 8 --pty bash -i
srun: error: slurm_receive_msgs: Socket timed out on send/recv operation
srun: error: Task launch for 37.0 failed on node c-node3: Socket timed
out on send/recv operation
srun: error: Application launch failed: Socket timed out on send/recv
operation
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
[root@kla-ac-ohpc-01 critical]# squeue
JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON)
36 normal bash test CG 0:53 2
c-node[1-2]
37 normal bash root CG 0:52 1 c-node3
Thank you.
Regards,
Rodgers
