So hangups like this can occur due to the slurmdbd being busy with
requests. I've seen that happen when an ill timed massive sacct request
hits when slurmdbd is doing its roll up. In that case the slurmctld
hangs while slurmdbd is busy. Typically in this case restarting
mysql/slurmdbd seems to fix the issue.
Otherwise this can happen due to massive traffic to the slurmctld. You
can try using the defer option for the SchedulerParamters. That slows
down the scheduler so it can handle the additional load.
-Paul Edmon-
On 11/8/2017 3:11 PM, Sean Caron wrote:
Hi all,
I see SLURM 17.02.9 slurmctld hang or become unresponsive every few
days with the message in syslog:
server_thread_count over limit (256), waiting
I believe from the user perspective they see "Socket timed out on
send/recv operation". Slurmctld never seems to recover once it's in
this state and will not respond to /etc/init.d/slurm restart. Only
after an admin does a kill -9 and restarts slurmctld does it snap back.
I don't see anything else in the logs that looks like an error message
that would help diagnose what is going on, even with log level debug3
on the SLURM controller daemon.
I monitor CPU and memory utilization with "htop" on the machine
running the controller daemon and it doesn't seem like it's
overwhelmed by slurmctld load or anything like that.
Machine running the controller daemon feels reasonable for the task,
for the size of our cluster. It's a repurposed Dell PowerEdge R410
with 24 threads and 32 GB physical. Unless I'm way off?
I tried all kinds of SchedulerParameter tweaks on sched/backfill and
even set the scheduler back to sched/builtin and it's still happening.
Didn't seem to affect the frequency much, either.
Any thoughts what could be causing SLURM to spawn so many threads and
hang up?
Our cluster is medium-sized, we probably have a few thousand jobs in
the queue on average at any given time.
Monitoring with sdiag, the max cycle time of main scheduler never
cracks 2 seconds. This seems reasonable?
Thanks,
Sean
