Re: [slurm-users] slurmctld up and running but not really working

[Ole Holm Nielsen]

Hi Julien,

You could make a database dump of the current database so that you can 
load it on another server outside the cluster, while you reinitialize 
Slurm with a fresh database.

So the database thinks that you have 253 running jobs?  I guess that 
slurmctld is not working, otherwise you could do: squeue -t running

This command can report current jobs that have been orphaned on the local 
cluster and are now runaway:

sacctmgr show runawayjobs

Read the sacctmgr manual page.

I hope this helps.

/Ole

On 7/20/22 14:19, Julien Rey wrote:
I don't mind losing jobs information but I certainly don't want to clear 
the slurm database altogether.

The /var/lib/slurm-llnl/slurmctld/node_state and 
/var/lib/slurm-llnl/slurmctld/node_state.old files look effectively empty. 
I then entered the following command:

sacct | grep RUNNING

and found about 253 jobs.

Is there any elegant way to remove these jobs from the database ?

J.

Le 19/07/2022 à 20:29, Ole Holm Nielsen a écrit :
Hi Julien,

Apparently your slurmdbd is quite happy, but it seems that your 
slurmctld StateSaveLocation has been corrupted:

[2022-07-19T15:17:58.356] error: Node state file 
/var/lib/slurm-llnl/slurmctld/node_state too small
[2022-07-19T15:17:58.356] error: NOTE: Trying backup state save file. 
Information may be lost!
[2022-07-19T15:17:58.356] debug3: Version string in node_state header 
is PROTOCOL_VERSION
[2022-07-19T15:17:58.357] Recovered state of 71 nodes
[2022-07-19T15:17:58.357] error: Job state file 
/var/lib/slurm-llnl/slurmctld/job_state too small
[2022-07-19T15:17:58.357] error: NOTE: Trying backup state save file. 
Jobs may be lost!
[2022-07-19T15:17:58.357] error: Incomplete job state save file 

Did something bad happen to your storage of 
/var/lib/slurm-llnl/slurmctld/ ?  Could you possibly restore this folder 
from the last backup?

I don't know if it's possible to recover from a corrupted slurmctld 
StateSaveLocation, maybe some others have an experience?

Even if you could restore it, the Slurm database probably needs to be 
consistent with your slurmctld StateSaveLocation, and I don't know if 
this is feasible...

Could you initialize your slurm 17.02.11 and start it from scratch?

Regarding an upgrade from 17.02 or 17.11, you may find some useful notes 
in my Wiki page 
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

/Ole


On 19-07-2022 16:28, Julien Rey wrote:
I am currently facing an issue with an old install of slurm (17.02.11). 
However, I cannot upgrade this version because I had troubles with 
database migration in the past (when upgrading to 17.11) and this 
install is set to be replaced in the next coming monthes. For the time 
being I have to keep it running because some of our services still rely 
on it.

This issue occured after a power outage.

slurmctld is up and running, however, when I enter "sinfo", I end up 
with this message after a few minutes:

slurm_load_partitions: Unable to contact slurm controller (connect 
failure)

I set SlurmctldDebug=7 in slurm.conf and DebugLevel=7 in slurmdbd.conf, 
however I don't get much info about any specific error that would 
prevent the slurm controller from working in the logs.

Any help would be greatly appreciated.

/var/log/slurm-llnl/slurmctld.log:

[2022-07-19T15:17:58.342] debug3: Version in assoc_usage header is 7936
[2022-07-19T15:17:58.345] debug3: Version in qos_usage header is 7936
[2022-07-19T15:17:58.345] debug:  Reading slurm.conf file: 
/etc/slurm-llnl/slurm.conf
[2022-07-19T15:17:58.347] debug:  Ignoring obsolete SchedulerPort option.
[2022-07-19T15:17:58.347] debug3: layouts: layouts_init()...
[2022-07-19T15:17:58.347] layouts: no layout to initialize
[2022-07-19T15:17:58.347] debug3: Trying to load plugin 
/usr/local/lib/slurm/topology_none.so
[2022-07-19T15:17:58.347] topology NONE plugin loaded
[2022-07-19T15:17:58.347] debug3: Success.
[2022-07-19T15:17:58.348] debug:  No DownNodes
[2022-07-19T15:17:58.348] debug3: Version in last_conf_lite header is 7936
[2022-07-19T15:17:58.349] debug3: Trying to load plugin 
/usr/local/lib/slurm/jobcomp_none.so
[2022-07-19T15:17:58.349] debug3: Success.
[2022-07-19T15:17:58.349] debug3: Trying to load plugin 
/usr/local/lib/slurm/sched_backfill.so
[2022-07-19T15:17:58.349] sched: Backfill scheduler plugin loaded
[2022-07-19T15:17:58.349] debug3: Success.
[2022-07-19T15:17:58.350] debug3: Trying to load plugin 
/usr/local/lib/slurm/route_default.so
[2022-07-19T15:17:58.350] route default plugin loaded
[2022-07-19T15:17:58.350] debug3: Success.
[2022-07-19T15:17:58.355] layouts: loading entities/relations information
[2022-07-19T15:17:58.355] debug3: layouts: loading node node0
[2022-07-19T15:17:58.356] debug3: layouts: loading node node1
[2022-07-19T15:17:58.356] debug3: layouts: loading node node2
[2022-07-19T15:17:58.356] debug3: layouts: loading node node3
[2022-07-19T15:17:58.356] debug3: layouts: loading node node4
[2022-07-19T15:17:58.356] debug3: layouts: loading node node5
[2022-07-19T15:17:58.356] debug3: layouts: loading node node6
[2022-07-19T15:17:58.356] debug3: layouts: loading node node7
[2022-07-19T15:17:58.356] debug3: layouts: loading node node8
[2022-07-19T15:17:58.356] debug3: layouts: loading node node9
[2022-07-19T15:17:58.356] debug3: layouts: loading node node10
[2022-07-19T15:17:58.356] debug3: layouts: loading node node11
[2022-07-19T15:17:58.356] debug3: layouts: loading node node12
[2022-07-19T15:17:58.356] debug3: layouts: loading node node13
[2022-07-19T15:17:58.356] debug3: layouts: loading node node14
[2022-07-19T15:17:58.356] debug3: layouts: loading node node15
[2022-07-19T15:17:58.356] debug3: layouts: loading node node16
[2022-07-19T15:17:58.356] debug3: layouts: loading node node17
[2022-07-19T15:17:58.356] debug3: layouts: loading node node18
[2022-07-19T15:17:58.356] debug3: layouts: loading node node19
[2022-07-19T15:17:58.356] debug3: layouts: loading node node20
[2022-07-19T15:17:58.356] debug3: layouts: loading node node21
[2022-07-19T15:17:58.356] debug3: layouts: loading node node22
[2022-07-19T15:17:58.356] debug3: layouts: loading node node23
[2022-07-19T15:17:58.356] debug3: layouts: loading node node24
[2022-07-19T15:17:58.356] debug3: layouts: loading node node25
[2022-07-19T15:17:58.356] debug3: layouts: loading node node26
[2022-07-19T15:17:58.356] debug3: layouts: loading node node27
[2022-07-19T15:17:58.356] debug3: layouts: loading node node28
[2022-07-19T15:17:58.356] debug3: layouts: loading node node29
[2022-07-19T15:17:58.356] debug3: layouts: loading node node30
[2022-07-19T15:17:58.356] debug3: layouts: loading node node31
[2022-07-19T15:17:58.356] debug3: layouts: loading node node42
[2022-07-19T15:17:58.356] debug3: layouts: loading node node43
[2022-07-19T15:17:58.356] debug3: layouts: loading node node44
[2022-07-19T15:17:58.356] debug3: layouts: loading node node45
[2022-07-19T15:17:58.356] debug3: layouts: loading node node46
[2022-07-19T15:17:58.356] debug3: layouts: loading node node47
[2022-07-19T15:17:58.356] debug3: layouts: loading node node49
[2022-07-19T15:17:58.356] debug3: layouts: loading node node50
[2022-07-19T15:17:58.356] debug3: layouts: loading node node51
[2022-07-19T15:17:58.356] debug3: layouts: loading node node52
[2022-07-19T15:17:58.356] debug3: layouts: loading node node53
[2022-07-19T15:17:58.356] debug3: layouts: loading node node54
[2022-07-19T15:17:58.356] debug3: layouts: loading node node55
[2022-07-19T15:17:58.356] debug3: layouts: loading node node56
[2022-07-19T15:17:58.356] debug3: layouts: loading node node60
[2022-07-19T15:17:58.356] debug3: layouts: loading node node61
[2022-07-19T15:17:58.356] debug3: layouts: loading node node62
[2022-07-19T15:17:58.356] debug3: layouts: loading node node63
[2022-07-19T15:17:58.356] debug3: layouts: loading node node64
[2022-07-19T15:17:58.356] debug3: layouts: loading node node65
[2022-07-19T15:17:58.356] debug3: layouts: loading node node66
[2022-07-19T15:17:58.356] debug3: layouts: loading node node67
[2022-07-19T15:17:58.356] debug3: layouts: loading node node68
[2022-07-19T15:17:58.356] debug3: layouts: loading node node73
[2022-07-19T15:17:58.356] debug3: layouts: loading node node74
[2022-07-19T15:17:58.356] debug3: layouts: loading node node75
[2022-07-19T15:17:58.356] debug3: layouts: loading node node76
[2022-07-19T15:17:58.356] debug3: layouts: loading node node77
[2022-07-19T15:17:58.356] debug3: layouts: loading node node78
[2022-07-19T15:17:58.356] debug3: layouts: loading node node100
[2022-07-19T15:17:58.356] debug3: layouts: loading node node101
[2022-07-19T15:17:58.356] debug3: layouts: loading node node102
[2022-07-19T15:17:58.356] debug3: layouts: loading node node103
[2022-07-19T15:17:58.356] debug3: layouts: loading node node104
[2022-07-19T15:17:58.356] debug3: layouts: loading node node105
[2022-07-19T15:17:58.356] debug3: layouts: loading node node106
[2022-07-19T15:17:58.356] debug3: layouts: loading node node107
[2022-07-19T15:17:58.356] debug3: layouts: loading node node108
[2022-07-19T15:17:58.356] debug3: layouts: loading node node109
[2022-07-19T15:17:58.356] debug:  layouts: 71/71 nodes in hash table, rc=0
[2022-07-19T15:17:58.356] debug:  layouts: loading stage 1
[2022-07-19T15:17:58.356] debug:  layouts: loading stage 1.1 (restore 
state)
[2022-07-19T15:17:58.356] debug:  layouts: loading stage 2
[2022-07-19T15:17:58.356] debug:  layouts: loading stage 3
[2022-07-19T15:17:58.356] error: Node state file 
/var/lib/slurm-llnl/slurmctld/node_state too small
[2022-07-19T15:17:58.356] error: NOTE: Trying backup state save file. 
Information may be lost!
[2022-07-19T15:17:58.356] debug3: Version string in node_state header 
is PROTOCOL_VERSION
[2022-07-19T15:17:58.357] Recovered state of 71 nodes
[2022-07-19T15:17:58.357] error: Job state file 
/var/lib/slurm-llnl/slurmctld/job_state too small
[2022-07-19T15:17:58.357] error: NOTE: Trying backup state save file. 
Jobs may be lost!
[2022-07-19T15:17:58.357] error: Incomplete job state save file
[2022-07-19T15:17:58.357] Recovered information about 0 jobs
[2022-07-19T15:17:58.357] cons_res: select_p_node_init
[2022-07-19T15:17:58.357] cons_res: preparing for 7 partitions
[2022-07-19T15:17:58.357] debug:  Ports available for reservation 
10000-30000
[2022-07-19T15:17:58.359] debug2: init_requeue_policy: 
kill_invalid_depend is set to 0
[2022-07-19T15:17:58.359] debug:  Updating partition uid access list
[2022-07-19T15:17:58.359] debug3: Version string in resv_state header 
is PROTOCOL_VERSION
[2022-07-19T15:17:58.359] Recovered state of 0 reservations
[2022-07-19T15:17:58.359] State of 0 triggers recovered