[slurm-users] ReqNodeNotAvail, but none of nodes in partition are listed.

[Prentice Bisbal]

Dear Slurm Users,

On my cluster, I have several partitions, each with their own QOS, time 
limits, etc.

Several times today, I've received complaints from users that they 
submitted jobs to a partition with available nodes, but jobs are stuck 
in the PD state. I have spent the majority of my day investigating this, 
but haven't turned up anything meaningful. Both jobs show the 
"ReqNodeNotAvail" reason, but none of the nodes listed at not available 
are even in the partition these jobs are submitted to. Neither job has 
requested a specific node, either.

I have checked slurmctld.log on the server, and have not been able to 
find any clues. Any where else I should look? Any ideas what could be 
causing this?

--
Prentice