We've gotten around the issue where we could not remove the runaway jobs. We
had to go the manual route of manipulating the db directly. We actually used a
great script that Loris Bennet wrote a while back. I haven't had to use it for
a long while - thanks again! :)
An item of interest for the developers ... there seems to be a limit that we
had exceeded that the "sacctmgr show runawayjobs" command or associated command
could handle. After fixing a good portion of the jobs (~32K left) , "sacctmgr
show runawayjobs" command did not display this error:
sacctmgr: error: slurmdbd: Sending message type 1488: 11: No error
sacctmgr: error: Failed to fix runaway job: Resource temporarily unavailable
And instead gave the dialog as normal "would you like to remove them ... (y/n)"
, etc"
The limit appears to be here:
common/slurm_persist_conn.c
...
#define MAX_MSG_SIZE (16*1024*1024)
...
I wonder if that should be increased? It's probably not normal to have 56K
runaway jobs, but still likely worthwhile addressing.
Anyhoo, it seems things are back to normal as far as we can tell. We will be
looking into providing faster storage for the db but does it seem reasonable
for slurm to crash under the circumstances that I mentioned?
Best,
Chris
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
On 4/24/18, 10:20 AM, "slurm-users on behalf of Christopher Benjamin Coffey"
<slurm-users-boun...@lists.schedmd.com on behalf of chris.cof...@nau.edu> wrote:
Hi, we have an issue currently where we have a bunch (56K) of runaway jobs,
but we cannot clear them:
sacctmgr show runaway|wc -l
sacctmgr: error: slurmdbd: Sending message type 1488: 11: No error
sacctmgr: error: Failed to fix runaway job: Resource temporarily unavailable
58588
Has anyone run into this? We've tried, restarting slurmdbd, slurmctl,
mysql, etc. but does not help.
Slurmdbd log shows the following when the "sacctmgr show runawayjobs"
command is run.
[2018-04-24T07:56:03.869] error: Invalid msg_size (31302621) from
connection 12(172.16.2.1) uid(3510)
[2018-04-24T07:56:03.872] error: Invalid msg_size (31302621) from
connection 7(172.16.2.1) uid(3510)
[2018-04-24T07:56:03.874] error: Invalid msg_size (31302621) from
connection 12(172.16.2.1) uid(3510)
[2018-04-24T07:56:03.875] error: Invalid msg_size (31302621) from
connection 7(172.16.2.1) uid(3510)
[2018-04-24T07:56:03.877] error: Invalid msg_size (31302621) from
connection 12(172.16.2.1) uid(3510)
Seems to indicate that possibly there are too many runaway jobs needing to
be cleared? I wonder if there is a way to select a fewer number for removal.
Don't see that option however.
This all started last week when slurm crashed due to being seriously
hammered by a user submitting 500K 2min jobs. Slurmdbd appeared to not be able
to handle all the transactions that slurmctl was sending it:
...
[2018-04-15T20:16:35.021] slurmdbd: agent queue size 100
[2018-04-16T11:54:29.312] slurmdbd: agent queue size 200
[2018-04-18T17:53:22.339] slurmdbd: agent queue size 19100
[2018-04-18T17:59:58.413] slurmdbd: agent queue size 64100
[2018-04-18T18:06:10.143] slurmdbd: agent queue size 104300
...
...
[2018-04-18T18:20:37.597] error: slurmdbd: agent queue filling (200214),
RESTART SLURMDBD NOW
...
...
error: slurmdbd: Sending fini msg: No error
...
So now at this point, lots and lots of our nodes are idle, but slurm is not
starting jobs.
[cbc@siris ~ ]$ sreport cluster utilization
--------------------------------------------------------------------------------
Cluster Utilization 2018-04-23T00:00:00 - 2018-04-23T23:59:59
Usage reported in CPU Minutes
--------------------------------------------------------------------------------
Cluster Allocated Down PLND Dow Idle Reserved Reported
--------- --------- -------- -------- -------- -------- ---------
monsoon 4216320 0 0 0 0 4216320
sreport shows the entire cluster fully utilized, yet this is not the case.
I see that there is a fix for runaway jobs in version 17.11.5:
-- sacctmgr - fix runaway jobs identification.
We upgraded to 17.11.5 this morning but still we cannot clear the runaway
jobs. I wonder if we'll need to manually remove them with some mysql foo. We
are investigating this now.
Hope maybe someone has run into this.
Thanks,
Chris
—
—
Christopher Coffey
High-Performance Computing
Northern Arizona University
928-523-1167
