Try:
sacctmgr list runawayjobs
Brian Andrus
On 12/20/2022 7:54 AM, Reed Dier wrote:
Hoping this is a fairly simple one.
This is a small internal cluster that we’ve been using for about 6
months now, and we’ve had some infrastructure instability in that
time, which I think may be the root culprit behind this weirdness, but
hopefully someone can point me in the direction to solve the issue.
I do a daily email of sreport to show how busy the cluster was, and
who were the top users.
Weirdly, I have a user that seems to be able to use the same exact
usage day after day after day, down to hundredth of a percent,
conspicuously even when they were on vacation and claimed that they
didn’t have job submissions in cron/etc.
So then, taking a spin of the scom tui
<https://lists.schedmd.com/pipermail/slurm-users/2022-December/009514.html>posted
this morning, I then filtered that user, and noticed that even though
I was only looking 2 days back at job history, I was seeing a job from
August.
Conspicuously, the job state is cancelled, but the job end time is 1y
from the start time, meaning its job end time is in 2023.
So something with the dbd is confused about this/these jobs that are
lingering and reporting cancelled but still “on the books” somehow
until next August.
╭──────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ Job ID : 290742 │
│ Job Name : $jobname │
│ User : $user │
│ Group : $user │
│ Job Account : $account │
│ Job Submission : 2022-08-08 08:44:52 -0400 EDT │
│ Job Start : 2022-08-08 08:46:53 -0400 EDT │
│ Job End : 2023-08-08 08:47:01 -0400 EDT │
│ Job Wait time : 2m1s │
│ Job Run time : 8760h0m8s │
│ Partition : $part │
│ Priority : 127282 │
│ QoS : $qos │
│ │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────╯
Steps count: 0
Filter: $user Items: 13
Job ID Job Name Part. QoS Account
User Nodes State
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
290714 $jobname $part $qos $acct
$user node32 CANCELLED
290716 $jobname $part $qos $acct
$user node24 CANCELLED
290736 $jobname $part $qos $acct
$user node00 CANCELLED
290742 $jobname $part $qos $acct
$user node01 CANCELLED
290770 $jobname $part $qos $acct
$user node02 CANCELLED
290777 $jobname $part $qos $acct
$user node03 CANCELLED
290793 $jobname $part $qos $acct
$user node04 CANCELLED
290797 $jobname $part $qos $acct
$user node05 CANCELLED
290799 $jobname $part $qos $acct
$user node06 CANCELLED
290801 $jobname $part $qos $acct
$user node07 CANCELLED
290814 $jobname $part $qos $acct
$user node08 CANCELLED
290817 $jobname $part $qos $acct
$user node09 CANCELLED
290819 $jobname $part $qos $acct
$user node10 CANCELLED
I’d love to figure out the proper way to either purge these jid’s from
the accounting database cleanly, or change the job end/run time to a
sane/correct value.
Slurm is v21.08.8-2, and ntp is a stratum 1 server, so time is in sync
everywhere, not that multiple servers would drift 1 year off like this.
Thanks for any help,
Reed
